[whatwg] Rich Paste & DataTransfer / DataTransferItems API

[Jeb Boniakowski]

Awesome!  Yeah I saw some of your patches in the public webkit trac
last week, but only the bits that were clearly related to implementing
the current spec.

The great thing about doing it as getAsBlob(), besides just being
cleaner is you keep the door open for expanding, through adding more
entries to DataTransferItem types and just passing back Blobs.
Naturally this is less likely to be immediately interesting in
non-image/file cases because the clipboard data isn't as naturally
self-contained and portable, but it could grow to be extremely useful,
especially if you factor in mobile devices, where you don't
necessarily have the filesystem-dribble as a fallback.

I guess the negative is you're using up official namespace without it
being in the spec?

Anyway, that is pretty cool.

jeb.

On Mon, Mar 14, 2011 at 9:22 PM, Daniel Cheng <dcheng at google.com> wrote:
> I'm currently 60-75% complete landing the patches for image paste support in
> Chrome. I've chosen to expose image/png instead of a raw bitmap through
> event.clipboardData.items in Chrome as a Blob. DataTransferItem::getAsFile()
> is currently specced to return a File; in my local changes, it secretly
> returns a Blob instead. I think it may make sense to rename it to
> getAsBlob() and change the corresponding add() method in DataTransferItems
> to take a Blob. I think these basic changes should support the use case you
> want if it gets more widely implemented.
> Daniel
>
> On Mon, Mar 14, 2011 at 15:39, Jeb Boniakowski <jebjeb at gmail.com> wrote:
>>
>> I would like to be able to paste images from the system clipboard into
>> web apps.  E.g. an annotated screengrab right into the bug tracker, a
>> pic into emails, an image from twitpic into a blog CMS's post screen.
>>
>> (Apologies if this has been discussed recently and conclusions made.
>> I searched the archives and couldn't find anything.  Also, this email
>> recapitulates a lot of stuff that will be familiar to participants in
>> this list, but I'm hoping that by posting a summary of the state of
>> affairs and then hopefully having errors/misconceptions pointed out,
>> it will serve as good archive fodder for future people like me.)
>>
>> This is of course a special case of richer interaction with the system
>> clipboard, but I think its a very special case, for two reasons:
>>  - The general case of richer paste is fraught with peril: what the is
>> your webapp supposed to do with pointers to various system data
>> structures?
>>  - The second-most common type of data, after text, manipulated on the
>> web is images.  They are a huge part of people's interaction with the
>> browser.  People move tons of images all over the web all the time,
>> and having to bounce off the local filesystem for everything is a
>> pain.
>>
>> Right now, there are effectively two intertwined APIs in the spec that
>> relate to being able to paste images:
>>  - The Old API, Microsoft style, which includes
>> DataTransfer.getData(format)
>>  - The New API: DataTransferItems, etc.
>>
>> -=The Old API=-
>>
>> With the Old API, the spec reads as if it could be possible to get
>> this data, should the browser feel like giving it to you.  There's
>> nothing that specifically limits what data the UA can give you, and it
>> sounds like you should be able to ask for whatever the UA tells you it
>> has: http://dev.w3.org/html5/spec/Overview.html#dom-datatransfer-getdata
>> (note: there seems to a small typo in the current Editor's Draft: the
>> getData() method only takes 'format', not a second parameter called
>> 'data', right?  That's what the summary tables above say.  In either
>> case, these should probably agree).
>>
>> There's a range in what the browsers return for types.  In Safari
>> (r80833), having a png on the clipboard shows you:
>>  - com.apple.pasteboard.promised-file-url
>>  - public.tiff
>>  - NSPromiseContentsPboardType
>>  - com.apple.webarchive
>>  - public.utf8-plain-text
>>  - dyn.ah62d4rv4gu8yc6durvwwaznwmuuha2pxsvw0e55bsmwca7d3sbwu
>>  - text/uri-list
>>  - Apple files promise pasteboard type
>>  - application/x-webarchive
>>  - dyn.ah62d4rv4gu8y6y4usm1044pxqzb085xyqz1hk64uqm10c6xenv61a3k
>>  - dyn.ah62d4rv4gu8zs3pcnzme2641rf4guzdmsv0gn64uqm10c6xenv61a3k
>>  - WebURLsWithTitlesPboardType
>>  -
>> dyn.ah62d4rv4gu8yc6durvwwa3xmrvw1gkdusm1044pxqyuha2pxsvw0e55bsmwca7d3sbwu
>>  - CorePasteboardFlavorType 0x75726C6E
>>  - CorePasteboardFlavorType 0x75726C20
>>  - text/plain
>>  - public.url-name
>>  - NeXT RTFD pasteboard type
>>  - public.url
>>  - com.apple.flat-rtfd
>>  - com.apple.pasteboard.promised-file-content-type
>>  - image/tiff
>>
>> Whereas the Chrome I have (10.0.648.133) shows me:
>>  - text/html
>>  - text/uri-list
>>  - url
>>
>> Firefox (3.6.15):
>>  - text/x-moz-url
>>  - text/x-moz-url-data
>>  - text/x-moz-url-desc
>>  - text/uri-list
>>  - text/_moz_htmlcontext
>>  - text/_moz_htmlinfo
>>  - text/html
>>  - text/plain
>>
>> I don't think this contravenes the spec, except that for all of those
>> rich types in Safari, you don't actually get anything when you ask for
>> it.  The spec doesn't really mandate anything that you have to do with
>> any specific types of data on the clipboard, which especially makes
>> sense when considering data that originates outside the browser.
>>
>> This has one interesting implication though for data that originates
>> inside the browser: drag/drop or paste with images loaded from data
>> URIs.  Currently, the clipboard behavior of these is interesting but
>> varies:
>>  - Chrome will return an img tag with the data uri intact, with the
>> Base64-encoded data for type "text/html" (preceded by a meta tag
>> telling you you're getting html)
>>  - Firefox does pretty much the same thing, minus the meta tag
>>  - Safari doesn't present a "text/html" type in this case.  However,
>> it does provide a "text/plain", which contains *just* the data uri
>> itself.
>>
>> Now, this situation isn't preventing anyone from doing anything, as
>> far as I can tell, but would it be crazy to try to normalize this in
>> the HTML5 spec? Decide whether a dragged/pasted data uri-backed <img>
>> counts as HTML or or text or neither, and when it does, what the HTML
>> representation is appropriate to hand back?  For what its worth, FF
>> and Chrome both return the plain data uri Safari-style if you ask for
>> text/uri-list.
>>
>> As far as I can tell, there is little interest in expanding the Old
>> API, since its an acknowledged hack-pile, but just to enumerate
>> options, I think it would be cool if there was a way to get image data
>> from here.  The obvious ways to do this would be:
>>  - actually expose mime types like image/tiff and return a String of 1s
>> and 0s.
>>  - actually expose mime types, but when you ask for a binary one, you
>> get a Blob.
>>  - expose base64-encoded mime types.  I realize it seems a bit weird
>> to push base64 encoding into the clipboard logic, but there are
>> already several places in the current spec and implementation where
>> base64-encoded binary data is given special treatment which provides
>> both (a) precedent and (b) use cases.  For example, many of the
>> browsers can load resources from base64-encoded data uri's.  The
>> Canvas object exposes a getDataUrl() method that returns a
>> base64-encoded png of the Canvas.  Having base64 data quick 'n' easy
>> to get here would make it easy to push this data around in useful ways
>> within apps.
>>
>> My guess is that no one wants to either add this to the current
>> implementations, even to accept patches that implement this behavior,
>> and that no one will want to add anything new to this part of the
>> spec, given the focus on the newer area.  Also, requiring support for
>> handling of specific binary formats seems to go against HTML's
>> heritage.  So while spec-wise, there's nothing to keep implementors
>> from adding application base64 versions of various image types to this
>> API, and returning same from getData(), they probably won't, and it
>> seems...unlikely to me that the spec will all of the sudden require
>> say four image types here.
>>
>> So, moving on...
>>
>> -=The New API=-
>> By the New API, I'm referring to the items member of DataTransfer,
>> DataTransferItems, and DataTransferItem.
>>
>> Briefly, it looks to me like the use cases were more thought out here
>> for interacting with system drag-drop than with copy-paste.  Excuse me
>> if that isn't the case, and the idea of richer copy-paste was just
>> considered to be out of scope.
>>
>> These features aren't supported by any of the shipping browsers I
>> looked at, but it looks like someone has started implementing them in
>> Chromium, there's talk on Firefox's list, and I'm not sure about
>> Safari/don't know how/if Chromium patches work their way up and down
>> into WebKit.  I haven't looked at the other browsers yet.  The point
>> is, I haven't actually been able to play with this, so I'm just going
>> by the spec here.
>>
>> According to the spec, with this API implemented, there will be no way
>> to copy an image from a tab, and paste it into another tab (assuming
>> the browser itself doesn't do something crazy like...bounce the system
>> clipboard to a temporary file on paste and give you the filename
>> somewhere).  The DataTransferItem can be of two kinds: a string, or a
>> string referring to a file, so this lets you drag String-y data off
>> your system and into a tab, and it also lets you presumably use
>> Drag/Drop instead of the system's file browser to attach files.
>>
>> Is there any chance of adding something like a getBlob() method here?
>>
>> My initial thought about the trade-offs here vs. something like adding
>> Base64'd binary types to the old DataTranfer.getData() is that its
>> more in keeping with the overall direction of HTML5, but also more
>> wrapped up in additional new APIs and less easy to immediately use.
>>
>> However, given the way system clipboards generally work, its pretty
>> easy to imagine how you could:
>>  - Map from certain system types to MIME types (or in some cases you
>> already have MIME types handy, and they don't necessarily have to be
>> in the spec)
>>  - Add more types to DataTransferItems to show these (again, don't
>> HAVE to be in spec)
>>  - have a DataTransferItems.getBlob(format) that returns a Blob.
>>
>> For small objects you could handle this immediately or fail with a
>> "too big" error, and with Web Workers you could handle larger pastes
>> without blocking the UI thread the entire time.
>>
>> Again, sorry if this was discussed in the current cycle and punted on
>> already, maybe I'm using the wrong search terms. Any thoughts about
>> changing the spec so that it becomes possible to paste images?
>>
>> Thanks,
>> jeb.
>
>