[whatwg] [canvas] matrix based changes on bitmaps

[Rob Manson]

Hi Tyler,

really glad you started this discussion otherwise I never would have
heard of the DSP API 8)


> On Sep 21, 2012, at 3:34 AM, Jussi Kalliokoski
> <jussi.kalliokoski at gmail.com> wrote:
> > Please see the DSP API [1]. It's currently developed unofficially
> > under the W3C Audio WG [2], so if you have input, please post it to
> > the audiowg public mailing list. This should scratch your itch and
> > more. ;)
> > 
> > Cheers,
> > Jussi
> > 
> > [1] http://people.opera.com/mage/dspapi/
> > [2] http://www.w3.org/2011/audio/  
> 
> 
> This stuff looks really interesting, and actually lower level than I
> was asking for which is cool but I'm not sure how this could be
> abstracted out so that it is useful for things outside of Audio? I'm
> likely just not putting it all together.

I agree that this looks really useful and I don't think it's intimately
tied to Audio as it simply manipulates Typed Arrays.  In fact that
the page at link [1] doesn't seem to even contain the term "audio"
at all. So it's definitely just a generic DSP model built upon Type
Arrays.  So it seems like there's no need to raise this as feedback on
the ArrayBuffer spec at all since Type Arrays are built on top of
ArrayBuffers.

 
> > It seems like a reasonable suggestion, but since the pixel data is 
> > available as an ArrayBuffer, it seems like the more reasonable
> > thing to do is to provide generic ArrayBuffer manipulation routines.
> > 
> > I recommend raising this as feedback on the ArrayBuffer spec.  
> 
> I like the idea of abstracting this into an ArrayBuffer, I like the
> sound of this but ArrayBuffers seem to promote nested arrays, where
> as the Canvas spec uses a simpler structure with r,g,b,a,… pixel
> values. It would be awesome to have these types of inconsistencies
> worked out so we didn't need to transform our data into different
> structures before applying it. 
> 
> An abstraction between all of these contexts (Canvas,Element
> Transforms,Audio,WebGL,...) seems doable but if it means sacrificing
> performance later because what is output by the generic structure
> can't be used in each of these APIs as is, I would rather have more
> targeted transformation methods on each of these APIs. 

Perhaps I'm misunderstanding you here...but isn't that really what
Typed Arrays are meant to be?  

For 2D context canvas you have...

  var imageData = ctx.getImageData(0,0, 200, 100);
  var typedArray = imageData.data // data is a Uint8ClampedArray

For 3D context (webgl) canvas you have...

  var floatArray = new Float32Array([1,2,3,4,5,6,7,8]);
  gl.bufferData(gl.ARRAY_BUFFER, floatArray);

xhr now support this...

  xhr.responseType = 'arraybuffer';

And now with transferable objects it's possible to pass ownership of a
type array to a web worker without copying any data at all...just the
ownership.  That a major efficiency optimisation.

You probably know all this but here's a nice concise overview that I
stole those examples above from 8)
http://www.html5rocks.com/en/tutorials/webgl/typed_arrays/


roBman