[whatwg] API for encoding/decoding ArrayBuffers into text

[Joshua Bell]

On Mon, Mar 19, 2012 at 7:00 AM, Henri Sivonen <hsivonen at iki.fi> wrote:

> On Wed, Mar 14, 2012 at 12:49 AM, Jonas Sicking <jonas at sicking.cc> wrote:
> > Something that has come up a couple of times with content authors
> > lately has been the desire to convert an ArrayBuffer (or part thereof)
> > into a decoded string. Similarly being able to encode a string into an
> > ArrayBuffer (or part thereof).
> >
> > Something as simple as
> >
> > DOMString decode(ArrayBufferView source, DOMString encoding);
> > ArrayBufferView encode(DOMString source, DOMString encoding,
> > [optional] ArrayBufferView destination);
>
> It saddens me that this allows non-UTF-8 encodings. However, since use
> cases for non-UTF-8 encodings were mentioned in this thread, I suggest
> that the set of supported encodings be an enumerated set of encodings
> stated in a spec and browsers MUST NOT support other encodings.


I believe we have consensus on the above.


> The
> set should probably be the set offered in the encoding popup at
> http://validator.nu/?charset or a subset thereof (containing at least
> UTF-8 of course). (That set was derived by researching the
> intersection of the encodings supported by browsers, Python and the
> JDK.)


I have edited the proposal to base the list of encodings on
http://dvcs.w3.org/hg/encoding/raw-file/tip/Overview.html - is there any
reason that would not be sufficient or appropriate? (this appears to be a
superset of the validator.nu/?charset list, with only a small number of
additional encodings)


> > would go a very long way.
>
> Are you sure that it's not necessary to support streaming conversion?
> The suggested API design assumes you always have the entire data
> sequence in a single DOMString or ArrayBufferView.


I agree that this is a sticking point, and I'm not sure how to resolve it.
Some are advocating for a simpler UI with the above assumption, others for
a broader solution that allows streaming conversion. The draft text as
written now is in the middle - supporting writing into an existing buffer,
but simply failing on overflow - and is thus not satisfactory to either
group.

> The question is where to stick these
> > functions. Internationalization doesn't have a obvious object we can
> > hang functions off of (unlike, for example crypto), and the above
> > names are much too generic to turn into global functions.
>
> If we deem streaming conversion unnecessary, I'd put the methods on
> DOMString and ArrayBufferView. It would be terribly sad to let the
> schedules of various working groups affect the API design.
>
> --
> Henri Sivonen
> hsivonen at iki.fi
> http://hsivonen.iki.fi/
>