[whatwg] Rich Paste & DataTransfer / DataTransferItems API

[Jeb Boniakowski]

I would like to be able to paste images from the system clipboard into
web apps.  E.g. an annotated screengrab right into the bug tracker, a
pic into emails, an image from twitpic into a blog CMS's post screen.

(Apologies if this has been discussed recently and conclusions made.
I searched the archives and couldn't find anything.  Also, this email
recapitulates a lot of stuff that will be familiar to participants in
this list, but I'm hoping that by posting a summary of the state of
affairs and then hopefully having errors/misconceptions pointed out,
it will serve as good archive fodder for future people like me.)

This is of course a special case of richer interaction with the system
clipboard, but I think its a very special case, for two reasons:
 - The general case of richer paste is fraught with peril: what the is
your webapp supposed to do with pointers to various system data
structures?
 - The second-most common type of data, after text, manipulated on the
web is images.  They are a huge part of people's interaction with the
browser.  People move tons of images all over the web all the time,
and having to bounce off the local filesystem for everything is a
pain.

Right now, there are effectively two intertwined APIs in the spec that
relate to being able to paste images:
 - The Old API, Microsoft style, which includes DataTransfer.getData(format)
 - The New API: DataTransferItems, etc.

-=The Old API=-

With the Old API, the spec reads as if it could be possible to get
this data, should the browser feel like giving it to you.  There's
nothing that specifically limits what data the UA can give you, and it
sounds like you should be able to ask for whatever the UA tells you it
has: http://dev.w3.org/html5/spec/Overview.html#dom-datatransfer-getdata
(note: there seems to a small typo in the current Editor's Draft: the
getData() method only takes 'format', not a second parameter called
'data', right?  That's what the summary tables above say.  In either
case, these should probably agree).

There's a range in what the browsers return for types.  In Safari
(r80833), having a png on the clipboard shows you:
 - com.apple.pasteboard.promised-file-url
 - public.tiff
 - NSPromiseContentsPboardType
 - com.apple.webarchive
 - public.utf8-plain-text
 - dyn.ah62d4rv4gu8yc6durvwwaznwmuuha2pxsvw0e55bsmwca7d3sbwu
 - text/uri-list
 - Apple files promise pasteboard type
 - application/x-webarchive
 - dyn.ah62d4rv4gu8y6y4usm1044pxqzb085xyqz1hk64uqm10c6xenv61a3k
 - dyn.ah62d4rv4gu8zs3pcnzme2641rf4guzdmsv0gn64uqm10c6xenv61a3k
 - WebURLsWithTitlesPboardType
 - dyn.ah62d4rv4gu8yc6durvwwa3xmrvw1gkdusm1044pxqyuha2pxsvw0e55bsmwca7d3sbwu
 - CorePasteboardFlavorType 0x75726C6E
 - CorePasteboardFlavorType 0x75726C20
 - text/plain
 - public.url-name
 - NeXT RTFD pasteboard type
 - public.url
 - com.apple.flat-rtfd
 - com.apple.pasteboard.promised-file-content-type
 - image/tiff

Whereas the Chrome I have (10.0.648.133) shows me:
 - text/html
 - text/uri-list
 - url

Firefox (3.6.15):
 - text/x-moz-url
 - text/x-moz-url-data
 - text/x-moz-url-desc
 - text/uri-list
 - text/_moz_htmlcontext
 - text/_moz_htmlinfo
 - text/html
 - text/plain

I don't think this contravenes the spec, except that for all of those
rich types in Safari, you don't actually get anything when you ask for
it.  The spec doesn't really mandate anything that you have to do with
any specific types of data on the clipboard, which especially makes
sense when considering data that originates outside the browser.

This has one interesting implication though for data that originates
inside the browser: drag/drop or paste with images loaded from data
URIs.  Currently, the clipboard behavior of these is interesting but
varies:
 - Chrome will return an img tag with the data uri intact, with the
Base64-encoded data for type "text/html" (preceded by a meta tag
telling you you're getting html)
 - Firefox does pretty much the same thing, minus the meta tag
 - Safari doesn't present a "text/html" type in this case.  However,
it does provide a "text/plain", which contains *just* the data uri
itself.

Now, this situation isn't preventing anyone from doing anything, as
far as I can tell, but would it be crazy to try to normalize this in
the HTML5 spec? Decide whether a dragged/pasted data uri-backed <img>
counts as HTML or or text or neither, and when it does, what the HTML
representation is appropriate to hand back?  For what its worth, FF
and Chrome both return the plain data uri Safari-style if you ask for
text/uri-list.

As far as I can tell, there is little interest in expanding the Old
API, since its an acknowledged hack-pile, but just to enumerate
options, I think it would be cool if there was a way to get image data
from here.  The obvious ways to do this would be:
 - actually expose mime types like image/tiff and return a String of 1s and 0s.
 - actually expose mime types, but when you ask for a binary one, you
get a Blob.
 - expose base64-encoded mime types.  I realize it seems a bit weird
to push base64 encoding into the clipboard logic, but there are
already several places in the current spec and implementation where
base64-encoded binary data is given special treatment which provides
both (a) precedent and (b) use cases.  For example, many of the
browsers can load resources from base64-encoded data uri's.  The
Canvas object exposes a getDataUrl() method that returns a
base64-encoded png of the Canvas.  Having base64 data quick 'n' easy
to get here would make it easy to push this data around in useful ways
within apps.

My guess is that no one wants to either add this to the current
implementations, even to accept patches that implement this behavior,
and that no one will want to add anything new to this part of the
spec, given the focus on the newer area.  Also, requiring support for
handling of specific binary formats seems to go against HTML's
heritage.  So while spec-wise, there's nothing to keep implementors
from adding application base64 versions of various image types to this
API, and returning same from getData(), they probably won't, and it
seems...unlikely to me that the spec will all of the sudden require
say four image types here.

So, moving on...

-=The New API=-
By the New API, I'm referring to the items member of DataTransfer,
DataTransferItems, and DataTransferItem.

Briefly, it looks to me like the use cases were more thought out here
for interacting with system drag-drop than with copy-paste.  Excuse me
if that isn't the case, and the idea of richer copy-paste was just
considered to be out of scope.

These features aren't supported by any of the shipping browsers I
looked at, but it looks like someone has started implementing them in
Chromium, there's talk on Firefox's list, and I'm not sure about
Safari/don't know how/if Chromium patches work their way up and down
into WebKit.  I haven't looked at the other browsers yet.  The point
is, I haven't actually been able to play with this, so I'm just going
by the spec here.

According to the spec, with this API implemented, there will be no way
to copy an image from a tab, and paste it into another tab (assuming
the browser itself doesn't do something crazy like...bounce the system
clipboard to a temporary file on paste and give you the filename
somewhere).  The DataTransferItem can be of two kinds: a string, or a
string referring to a file, so this lets you drag String-y data off
your system and into a tab, and it also lets you presumably use
Drag/Drop instead of the system's file browser to attach files.

Is there any chance of adding something like a getBlob() method here?

My initial thought about the trade-offs here vs. something like adding
Base64'd binary types to the old DataTranfer.getData() is that its
more in keeping with the overall direction of HTML5, but also more
wrapped up in additional new APIs and less easy to immediately use.

However, given the way system clipboards generally work, its pretty
easy to imagine how you could:
 - Map from certain system types to MIME types (or in some cases you
already have MIME types handy, and they don't necessarily have to be
in the spec)
 - Add more types to DataTransferItems to show these (again, don't
HAVE to be in spec)
 - have a DataTransferItems.getBlob(format) that returns a Blob.

For small objects you could handle this immediately or fail with a
"too big" error, and with Web Workers you could handle larger pastes
without blocking the UI thread the entire time.

Again, sorry if this was discussed in the current cycle and punted on
already, maybe I'm using the wrong search terms. Any thoughts about
changing the spec so that it becomes possible to paste images?

Thanks,
jeb.