[whatwg] Drag-and-drop feedback

[Ian Hickson]

On Mon, 22 Feb 2010, Ian Hickson wrote:
> On Thu, 4 Feb 2010, Ian Hickson wrote:
> > On Sat, 23 Jan 2010, Eduard Pascual wrote:
> > >
> > > Would it be possible to provide a list of "drag items" (to call them 
> > > somehow) instead of, or in addition to, the current info provided by 
> > > the DataTransfer object?
> > 
> > That's a pretty good idea. I think we should probably do this when we 
> > add more types to the DataTransfer object.
> 
> Some engineers at Google discussed this a bit and came up with the 
> following proposal:
> 
>    dataTransfer.items = DataTransferItems
> 
>     DataTransferItems.length
>                      .getItem(n) = DataTransferItem
>                      .add(stringData, type)
>                      .add(blobData)
>                      .add(fileData)
>                      .add(dataTransferItem)
>                      .clear()
> 
>     DataTransferItem.kind = 'string', 'file', 'blob', ...
>                     .type = MIME type
>                     .binary = boolean 
>                     .getTextData(function callback (data)) - throws if binary is true
>                     .getBlob() - returns File or Blob
> 
> When we add promises later, this can easily be extended to support that 
> as well (basically, just by adding a new add() method for the promise 
> case).

I've added a simple version of this (no Blob support, and no way to add a 
dataTransferItem, but otherwise more or less the same).


On Mon, 22 Feb 2010, Daniel Cheng wrote:
>
> How does DataTransferItems interact with the original DataTransfer object?
> I'm assuming changes in one should be reflected in the other.

Yes.


> If that's the case, what should happen if I do this:
> dataTransfer.items.add(fileData);
> dataTransfer.getData(mimeTypeForFile);

getData() doesn't expose the items with files, only those that are text. 
(It can't expose the files since those are asynchronous binary Blobs, not 
synchronous Unicode strings).


> How come there's no DataTransferItems.get(type) method?

It would probably have to be get(type, kind) and would have to be able to 
return a list, at which point you might as well just iterate over the 
actual .items list and check the type and kind for each one directly. It 
seems that a get() here wouldn't really save much. If you have any use 
cases you could show, I'd be happy to see what the code would look like 
though to see if it would save much. If it does, certainly it would make 
sense to add something like that.


> DataTransferItem provides richer metadata than is available through the 
> native drag-and-drop interface on most platforms. When dragging data 
> from a non-DOM application, how do you extrapolate the metadata to fill 
> in the type/binary fields?

I don't follow. Could you give an example of what you mean?


On Fri, 26 Feb 2010, Dmitry Titov wrote:
> 
> I understand 'binary' as indicator of whether or not the item can be 
> read as a text string. I'm not sure why item.kind == "string" is not the 
> same. If the intent is to also cover some files that can be read as 
> string and as Blob, then it might be buggy because the only way to 
> establish if the file can be converted into JS string is to actually 
> read the bytes and try to convert to Unicode. There can be invalid 
> character sequences or the encoding info may be missing.
>
> Perhaps we should remove 'binary' and assume that items that item.kind 
> == "string" can use getTextData(callback) to pull the string.

Yeah, I ended up going that way.


> On a separate note, I think items.add(dataTransferItem) is not useful 
> w/o a way to create a new DataTransferItem separately from the 
> DataTransferItems list.

Agreed. I ended up not adding it.


On Tue, 9 Mar 2010, Daniel Cheng wrote:
> 
> I think files have always been a special case and should continue to be 
> handled that way. I don't think there's any platform pasteboard that 
> supports multiple items of one non-file type, so it'd make the most 
> sense to make it a 1:1 mapping from types to data.

Currently the new DataTransferItems API doesn't enforce the limit of one 
"string" item per type. Should it?


> [in the context of supporting native data dragged to the web]
> Should there be some sort of standard list of mappings that all UAs 
> should support wherever possible? What happens if the UA doesn't have a 
> mapping for a type?

I'm happy to add a mapping list if someone can provide one for common 
platforms. I don't currently have one. The spec just says that the UA 
should do a best effort attempt.


On Mon, 29 Mar 2010, Daniel Cheng proposed:
> 
> DataTransfer mirrors DataTransferItems, but DataTransfer.getData() will 
> throw an error if the data is a blob.

Currently the types of files and of string data are different namespaces, 
you can have a text/plain file and a text/plain string. Should I disallow 
this? Is it not possible to end up in such a situation in native DND UIs?


> Maybe a second getData method that can provide conversion given a text 
> encoding would be useful?

the File/Blob API should support that, I don't think we should add 
something like this specifically for drag-and-drop.


> DataTransfer.addFile(fileData);

It's now DataTransfer.items.add(fileData);


> When interacting with non-DOM apps or pages, some platforms can't easily 
> convert arbitrary MIME types to native data transfer types for 
> copy/paste or DnD. For this reason, I think the spec should explicitly 
> list MIME types for which UAs should handle the conversion to native 
> data transfer types. A couple that come to mind: text/plain, 
> text/uri-list, text/rtf, application/rtf, text/html, text/xml, 
> image/png, and image/svg+xml. UAs can make a best-effort attempt to 
> convert the other types, but it won't be guaranteed that they will be 
> there for interaction with non-DOM applications.

I'm not sure what this means exactly. Could you elaborate?


On Mon, 28 Jun 2010, Daniel Cheng wrote:
>
> It's pretty common for there to be non-text data in a drag-and-drop
> operation or copy-and-paste operation. DataTransfer doesn't allow for that
> currently, since it only sets and returns DOMStrings.
> 
> It'd be nice if we could extend setData/getData to allow for Blobs. Some
> random thoughts:
> 1. Add a bool parameter to setData/getData. If false, treat the data as a
> DOMString; if true, treat the data as a Blob.
> 2. Add an encoding parameter to setData/getData. Encoding can be a string
> value naming a text encoding like UTF-8 or ISO-8859-1, or it can be the
> string value "binary". If encoding names a text encoding, the UA will
> transcode the requested data into/from a DOMString. Otherwise, if the
> encoding value is "binary", the UA will treat data as a Blob.
> 3. Create new DataTransfer functions instead of overloading them, e.g.
> setDataBlob, getDataBlob.
> 
> I'm not sure which one is the preferred approach. It seems like it'd be nice
> to have native support for whatever text encodings the browser understands,
> but it seems complicated and I'm not sure it's necessary. Thoughts?

The new DataTransferItems feature could let you do this using 
dataTransfer.items.add(blob), if we added that. Right now you can add just 
text strings and File objects. We could also one day support arbitrary JS 
objects (with the structured clone stuff) the same way, too.


> Also, if I wanted to go ahead and implement a prototype in WebKit, should I
> prefix it with a UA-specific string, e.g. webkitSetDataBlob?

Yes.


On Thu, 12 Aug 2010, Jason Gross wrote:
>
> The specification says that the dragenter event is "used to determine 
> whether or not the drop target is to accept the drop".  Do functions 
> bound to this event get any information about the object being dragged?
> In particular, is there a good way to have N drop targets, and have each 
> of them accept only certain draggables?  If not, it seems to me like a 
> good feature to have, especially as multi-touch applications/devices 
> become more prevalent.

I've added a dropzone="" attribute that lets you do this easily.


On Mon, 16 Aug 2010, Jason Gross wrote:
>
> Is it possible to get more specificity than just the type of the object 
> being dragged?  For example, if I have red images and blue images, and a 
> red target and a blue target, and I want to be able to drop red images 
> only on the red target, and blue images only on the blue target, is 
> there a good way to do this, other than globally keep track of which 
> thing is being dragged?

You can give the images different types in the dragstart event. Other 
thant that, you can't inspect the data during the dragenter/dragover 
events, because that would be a security risk (think of what happens if 
you drag a confidential file over a hostile web page to drop it on the 
other side of your browser).


On Sun, 15 Aug 2010, David Bruant wrote:
> 
> I have found completely counter-intuitive to have to register a listener 
> to the dragover event only to ".preventDefault()" the event.

Agreed. The new dropzone="" attribute should help a lot with this.


> The provided example in the spec seems to be written in a way that make 
> think that drag and drop operations occur within the same document (it's 
> never written specifically). If it's the case, I think that a different 
> example should be provided without dataTransfer.

You don't have to use dataTransfer, but there's no harm in using it as far 
as I can tell, especially with the expanded API that works with 
dropzone="" to let you control where you can drop something.


> * Regardless if the D&D spec example is changed (or another one added 
> for the "same document" use case without dataTransfer), please add a 
> comment next to the "event.preventDefault();" of the dragOverHandler 
> function to explain that the purpose of this is to define a drop target.

Done.


> * I am worried about the effectAllowed attribute. Needless to say that
> the number of value is exponencially proportional to the number of
> possible values for the dropEffect attribute. Wouldn't it be better to
> have a linear number of booleans ?

Yes, but it's about 10 years too late to change that.


> * About all the drag feedback generation, I am doubting that the current 
> API allows the rich effects that the JS librairies provide. Do anyone 
> have enough experience with these to tell if they see use cases covered 
> by the librairies that the native API wouldn't allow to cover ?

It's almost certainly the case that these libraries are more powerful than 
the API in HTML. Standardisation always follows innovation. If there are 
specific features that are used a lot, we should adopt them. Are there?


> I have seen in whatwg.org/issues that drag and drop is 
> "delayed-awaiting-more-implementation-experience". I haven't seen a wiki 
> page on http://wiki.whatwg.org/wiki/ though. Is there one ? Should one 
> be created to gather all the feedback and relevant resources regarding 
> D&D?

Please do feel free to use the wiki, but feedback should be sent to the 
list, so that I can track what has been dealt with and what has not.


On Thu, 26 Aug 2010, Charles Pritchard wrote:
> On 8/25/2010 2:02 PM, Ian Hickson wrote:
> > On Mon, 2 Aug 2010, Charles Pritchard wrote:
> > > > [ UAs can use<input type=file>  to let the user enter remote URLs ]
> > >
> > > When a user through selection, click+drag or manual entry of a URL 
> > > should the browser still submit an Origin request header? It seems 
> > > that CORS doesn't come into effect here -- but at the same time, 
> > > it'd be handy for logging purposes and added security.
> >
> > I don't think there'd be an origin, but that's rather up to the user 
> > agent. (In this case it's acting on behalf of the user, not the page, 
> > so I don't think it makes sense to give the page's origin.)
>
> Sounds like an implementer would not include a Referer header, either.

Possibly.


> Continuing on with tweaking URLs to work with with the File API:
> 
> Chrome has gone ahead with their setData proposal, enhancing the 
> event.dataTransfer object so that users may drag a file from within the 
> browser onto their desktop.
> 
> The extension uses setData with a key of DownloadURL and a value 
> including a mime type, file descriptor and URI.
> 
> I'd like this interface to work within ondrop; if getData(DownloadURL) 
> is set, then a FileList would be returned in event.dataTransfer.files, 
> much like it is when users drag files from their desktop into the 
> browser.
> 
> This would of course require Origin checks; whereas dragging onto the 
> desktop does not require an Origin check.

I'm not quite sure I follow what you are proposing. However, in a future 
version of this API we should definitely add a "promise"-like feature that 
lets you specify drag data without having it already downloaded, so that 
when the user drops the data somewhere, the browser can then ask the JS 
for the data. I'm not sure using setData('DownloadURL') is a good way to 
do it; that just seems to add more hacks to an already pretty hacky API.


On Mon, 18 Oct 2010, Daniel Cheng wrote:
>
> I've been working on better support of arbitrary MIME types in WebKit 
> for some time, and I had some implementation questions. In the past, UAs 
> seem to have gone out of their way to make sure full filesystem paths 
> aren't exposed to the Javascript (e.g. in the file input control). When 
> I did the work for WebKit, I implemented the web dragging clipboard as a 
> simple reflection of the native dragging clipboard.
> 
> However, this leads to issues like file system paths being exposed 
> through properties like "x-special/gnome-icon-list" or even 
> "text/plain". What is the expected behavior here? Mirroring the native 
> dragging clipboard allows for a much richer interaction with the system, 
> but I'm not sure if we need to go out of our way to try to scrub all 
> paths from the drag. After all, if you're dropping the file on the page, 
> you're already exposing the contents of the file, which are probably 
> much more interesting than just the path. Thoughts?

On Mon, 18 Oct 2010, Tab Atkins Jr. wrote:
> 
> Could you provide some more detail?  I have no idea what you mean by 
> "However, this leads to issues like file system paths being exposed 
> through properties like "x-special/gnome-icon-list" or even 
> "text/plain"."  Those aren't properties, nor do they expose file-system 
> paths.
> 
> Do you perhaps mean that they expose generally the origin of the file, 
> such that you know if you see "x-special/gnome-icon-list" that the file 
> is probably coming from wherever gnome stores that kind of file?

On Mon, 18 Oct 2010, Daniel Cheng wrote:
>
> Sorry, I'm using "properties" as a generic term for different types of 
> data that might be set in a drag. A lot of file managers try to be 
> helpful and populate alternative metadata for a file. Some of this 
> metadata contains file system paths. If the web dragging clipboard 
> mirrors the native dragging clipboard, then the metadata will be visible 
> to web apps. In this example, if you were on Linux with my patch, you 
> could call event.dataTransfer.getData("x-special/gnome-icon-list") while 
> handling a drop and the returned string would contain the file system 
> path.

On Tue, 19 Oct 2010, Anne van Kesteren wrote:
> 
> It seems wrong to expose it in a way native to a particular operating 
> system so it seems better to filter it out for now even if that is more 
> work. We should keep web platform APIs OS-independent.

On Wed, 20 Oct 2010, Daniel Cheng wrote:
>
> To clarify, I wasn't proposing that pages need to know details of a 
> particular OS. Things like "text/plain", "text/uri-list", "text/html", 
> etc. are automatically mapped by the UA to whatever the appropriate 
> platform idiom is.
> 
> I just thought it would be useful to also expose things that the UA 
> itself doesn't natively understand--it just gets passed through to the 
> web content. However, this led to the above problem with filenames being 
> exposed. This can, to some extent, be mitigated by blacklisting certain 
> types; I'm just wondering if people feel that the additional utility is 
> worth the risk of potentially exposing file paths because of a chatty 
> file manager, or if anyone has any ideas on how to mitigate this risk.

On Tue, 26 Oct 2010, Anne van Kesteren wrote:
> 
> I was saying that if you get this on one OS but not another you might 
> get pages that depend on a particular OS if not coded carefully.

On Fri, 29 Oct 2010, Daniel Cheng wrote:
>
> In that case, I'd like to propose a set of MIME types that the spec
> explicitly mentions for interoperability with native apps:
> - text/plain for compatibility with IE
> - text/uri-list for compatibility with IE
> - text/html for rich text formatting. One potential usage--a reference site
> such as Wikipedia could implement a drag-out handler which automatically
> encapsulates the dragged snippet in a "quote" box and links back to the
> source.
> - application/rtf for rich text data. RTF is called out in particular since
> it allows embedded images, which HTML does not. This would allow someone to
> drag-and-drop cells and a graph from a spreadsheet into an email.
> - image/png for image transfers. Native drag and drop usually involves
> bitmaps, so the UA would automatically perform a lossless conversion from
> the native platform-dependent bitmap format to a PNG if the page requested
> this type. One example usage is uploading a picture of your desktop--simply
> hit 'Print Screen', navigate to the image sharing site, and ^V.
> - image/svg+xml for vector image transfers. It should be possible to convert
> the various vector formats (WMF, PS, PDF) into SVG, but I'm not sure how
> valuable doing this would be.
> 
> Those 6 types seem to cover a fairly wide variety of use cases without 
> being too domain-specific. What do people think?

On Tue, 19 Oct 2010, Robert O'Callahan wrote:
> 
> The path can expose interesting metadata, such as the local username 
> (useful for dictionary attacks!), the names of file servers, names of 
> projects, etc. Obviously the filename can expose some too, but hopefully 
> the user's more aware of that.

I'm not exactly sure what we should do here. A whitelist seems like the 
only solution, since a blacklist is just asking for trouble. However, a 
whitelist dramatically reduces the options in terms of what can be done 
with a Web app -- for example, the list above doesn't include dragging a 
vCard to or from an address book app.

Also, what types should we support going out vs going in?

I've added a comment in the spec source to the effect that this needs 
specifying, but I don't really know what it should say.

-- 
Ian Hickson               U+1047E                )\._.,--....,'``.    fL
http://ln.hixie.ch/       U+263A                /,   _.. \   _\  ;`._ ,.
Things that are impossible just take longer.   `._.-(,_..'--(,_..'`-.;.'