"decode" attribute on <img>

[smfr]

Decoding of large images can block the main thread for hundreds of milliseconds or more, interrupting fluid animations and user interaction. Currently, there's no way for a web author to specify that they want an image to be decoded asynchronously, so there are scenarios where it is impossible to avoid UI stalls.

To solve this problem we propose an "async" attribute on image elements. This attribute is a hint to the UA that the author has requested asynchronous decoding. This implies that if the UA paints an image after the "load" event has fired, but before the image has been decoded, the UA is allowed to not paint the image (rather than block on decoding it).

To notify authors when a decoded image frame is available, we propose firing a new event, "ready", on the image element. This would allow authors who require a fully-decoded image, in content that is sensitive to UI stalls, to wait for the "ready" event before doing something that brings the image into view (such as a CSS transition).

Some images repeatedly decode frames, for example, animated GIFs. In addition, the UA can throw away the decoded frame for a still image, requiring a re-decode. In these cases, we propose that the "ready" event only fires once, the first time a frame is available for display.

ISSUES:
"async" is currently used to imply async loading, and it's possible that we'd want to use it in this sense for images too. Maybe call the new attribute "asyncDecode" or something else.

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

[grorg]

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

We could maybe follow CSS animations/transitions here. Make a new "imageDecoded" or "imageReady" event (I suck at names) and fire it on the target that has the new CSS property.

I wonder if you'll ever want some parts of the element to have async images, and other parts to not. e.g. borders vs background. If so, it seems like it would have to go in the new image() function. (It already has tags for rtl and ltr).

[grorg]

This only solves the problem for image elements, not CSS images. We could add a CSS property to allow async decoding for CSS images, and fire events on the element to which it applies.

I do think it is confusing that async here is a slightly different meaning from script (which is "Set this Boolean attribute to indicate that the browser should, if possible, execute the script asynchronously." )

Or does it?... Would you consider image decoding to be the equivalent of script executing?

[grorg]

Of course, the HTML5 specification has a more complete definition of script async.

[smfr]

Actually reading that, I think 'async' here is quite a good match.

[duanyao]

Isn't off main thread image decoding already possible in browsers without a spec change? E.g.:
https://bugzilla.mozilla.org/show_bug.cgi?id=716140
https://bugs.webkit.org/show_bug.cgi?id=90375

Does "async" also means lazy download of the image file?

[smfr]

Isn't off main thread image decoding already possible
Possible, yes, but the main thread may still have to block on the decoder thread if it is still decoding an image when the main thread wants to paint the image. If a UA chooses not to block, then the user gets a a blank or partial image even though the load event fired (I'm not sure what Gecko does, and Mac/iOS WebKit have used main thread decoding for the past few years).

This proposal allows an author to present images with a guarantee both that the main thread is never blocked on image decoding, and that the image is ready to display when the "ready" event fires.

Does "async" also means lazy download of the image file?

No. It's purely about decoding.

[duanyao]

Once a UA implemented off main thread decoding, there is little reason for them to keep the old behavior, so still no need to specify "async" in HTML. Currently there seems no UA which (intentionally) blocks layout/paint on image download/decoding.

As to "ready" event, I think the situation can be compicated, because UA might drop decoded images that are no longer visible (out of viewport or in background tabs) to save memory. For completeness, "unload" event may be also needed.

[smfr]

Currently there seems no UA which (intentionally) blocks layout/paint on image download/decoding.

First, we're not talking about downloading here. This proposal doesn't change anything about how image downloading works.

Second, Mac and iOS WebKit block painting on image decoding (I work on WebKit for Apple, BTW). That's the problem we're trying to address. If other UAs have solved this problem, I would be interested to hear how.

[duanyao]

I am confused because I observed that UAs may display images increamentally on a slow connection, so I think decoding is not blocking painting, at least in such situation.

So does "async" also gurantee that the image is not visible when the downloading/decoding is in progress?

[smfr]

UAs can decode and paint an image that is not fully downloaded, yes. It may appear partial or low-resolution in this state, but the decode itself can still take time (100ms or more). I think the "ready" event should fire when the first full-resolution frame is available.

[annevk]

@igrigorik @yoavweiss may have opinions.

I think the API should be <img>.ready returning a promise. No opinions on the attribute, though we might in the future also want something that influences when the image is downloaded (e.g., an indication to the user agent that it's fine to load the image later).

[domenic]

Let's set aside the bikeshedding on the name and style of the event/promise for now, and try to get some multi-implementer interest in the base feature of using a new attribute to control image decoding. I think the key question is:

Second, Mac and iOS WebKit block painting on image decoding (I work on WebKit for Apple, BTW). That's the problem we're trying to address. If other UAs have solved this problem, I would be interested to hear how.

For example, a UA could choose to simply make all images have this "async" behavior and draw placeholders. Presumably this might not be great for authors (why, exactly?), thus this proposal to make it opt-in, if I am understanding correctly. If we were to, in certain circumstances, flip the default to async, this is starting to sound intervention-ish; /cc @RByers @ojanvafai

Another aspect that this is related to is off-main-thread image decoding. Some of the recent work on ImageBitmap has enabled this to be done in JavaScript; see https://developers.google.com/web/updates/2016/03/createimagebitmap-in-chrome-50. But this requires awkward contortions because then you have to draw it onto a canvas, instead of using a simple <img>. I am unsure to what extend this async proposal overlaps---would this allow off-main-thread image decoding? Is it in some sense sugar over the ImageBitmap technique that makes it less weird? /cc @junov since he's our ImageBitmap guy, both for Chrome and for this spec.

My sense is that this area is of definite interest to Chrome, and as such they're good candidates for second-vendor interest. I've tried to pepper @-mentions of people who might know more above. Any thoughts on Mozilla or Edge?

[smfr]

For example, a UA could choose to simply make all images have this "async" behavior and draw placeholders. Presumably this might not be great for authors (why, exactly?)

Some authors need a guarantee that if an image is ready, it gets painted. An image popping in later is not acceptable in some kinds of content.

If we were to, in certain circumstances, flip the default to async

Interesting idea, but I don't think it would fly compat-wise.

I am unsure to what extend this async proposal overlaps---would this allow off-main-thread image decoding?

I'm proposing this now specifically because WebKit has been doing work on off-main-thread image decoding, for some subset of images (webkit.org/b/155322 and related). I don't think a UA would be required to do off main thread decoding to implement this (for example, they could trigger decodes in something like a requestIdleCallback), but pages would still suffer from UI stalls.

[domenic]

Some authors need a guarantee that if an image is ready, it gets painted. An image popping in later is not acceptable in some kinds of content.

I think I understand what you're saying, but can you check if the following expansion is correct?

As a web developer it seems like all of my images just pop in at random times when the browser has finished with them and is ready to show them. The difference between showing a placeholder while it's fetching and then janking while decoding and then painting, vs. showing a placeholder while fetching and decoding, and then painting, is almost unobservable to me.

The difference is in the specific case where I end up listening to the load event. For example, if I create an image out of document, then wait for the load event, I anticipate that when I insert it into the document it will be painted synchronously. (This might even affect from-script-observable things like offsetWidth.)

But as an author, it seems like any time I'm not listening for the load event and assuming the image is fully ready at that time, I'd be able to sprinkle async on the <img> elements, and there would be no observable difference. Right?

[smfr]

... anticipate that when I insert it into the document it will be painted synchronously

That's exactly it. This is mostly useful for images created in script.

any time I'm not listening for the load event and assuming the image is fully ready at that time, I'd be able to sprinkle async on the elements, and there would be no observable difference

Yes. The UA may choose to do async decoding for some or all elements. Addition of the "async" attribute to an may be treated as a hint or may do nothing.

[domenic]

OK, cool, glad I understand!

What do you think of an alternate proposal, to delay the load event until decoding is finished, in all cases?

[smfr]

What do you think of an alternate proposal, to delay the load event until decoding is finished, in all cases?

You can't do that, because loading and decoding are two separate steps. Not all images that are loaded are decoded (we never decode those that don't get painted).

Note that fetching metadata from an image, to get its size, for example, doesn't count as decoding.

[ojanvafai]

I'm pretty sure Chrome strongly supports doing something roughly like this. I'm consulting with folks to make sure I say something representative of our opinion since we've had a lot of complicated discussion about this issue. Will report back soon (please ping me if I lose track of it).

[igrigorik]

Naive question: how would "ready" event work in cases where we have hardware decoding?

[smfr]

I don't think it matters. Hardware decoding happens under a software call to the decoder, just like software decoding. It just might be faster.

[junov]

Responding to @domenic 's earlier comment that there is overlap with ImageBitmap... Yes and no.

I've experimented with getting jank-free scrolling and canvas draws under intense image loading. I used XHR -> creatImageBitmap -> canvas. It worked like a charm, but it has one non-negligible drawback: it pins all the decoded images in RAM unless the app explicitly discards them, so implementing something like Facebook's infinite scrolling on top of this is a bit involved. It would require the web app to be responsible for evicting and triggering predictive redecodes in order to avoid extreme memory bloat (and OOM crashes). Considering that JS code has no visibility into the system's memory contention, using ImageBitmap as a blanket solution for de-janking image-intensive pages is really not that great.

This proposal lets the UA continue to manage the memory occupied by decoded resources, which is a big win for many use cases.

On the other hand, the ImageBitmap approach guarantees that the image will be drawn and that it will be fast. This is useful for cases that don't jive with this statement of the proposal: "if the UA paints an image after the "load" event has fired, but before the image has been decoded, the UA is allowed to not paint the image (rather than block on decoding it)"

[smfr]

If the author really want the image pixels to be available on first paint, they need to either insert the image in script after receiving the "ready" callback, or toggle some CSS so that the image becomes visible.

However, there's a problem with my proposal; there's no way for the author to indicate that the UA should start decoding an image (if that image is unparented), and I don't think UAs will want to start decoding all unparented images with "async".

[domenic]

However, there's a problem with my proposal; there's no way for the author to indicate that the UA should start decoding an image (if that image is unparented), and I don't think UAs will want to start decoding all unparented images with "async".

Since this is most useful for images created in script anyway, maybe an imperative "decode me now" API? Like

img.ensureDecoded().then(() => {
  // OK, now you can insert it into the document
});

Maybe "ensureReady" if we want to hide the implementation details a little more.

[smfr]

That could work, yes. I think that means you don't need an "async" attribute for images created in JS.

It could still be useful as a hint on content images to say that you're OK with an image temporarily painting blank even after the load event fires, trading off against possible UI stalls. No doubt people will want CSS pseudo classes to style the various states of an image.

[vmpstr]

As @ojanvafai mentioned, here are our (Chrome) thoughts after having discussed with our team:

We agree that this is an important area to look into and we’re excited to work on solutions in this space!

We see two important use cases with different characteristics.
(1) Deferring: Ability to defer decodes of images in dom order to avoid janking the page.
(2) Predecoding: Ability to know when images have been decoded so they can be inserted into dom without jank

An async attribute on images solves deferring, but not predecoding. An on ready event solves predecoding, but not deferring. It seems like these could potentially be solved separately.

We have some concerns about tricky parts of the ready event:
* Does every image get "on ready" or just ones marked async?
* Would a browser need to predecode all out of dom images so that they would receive this event? Or just ones marked async?
* Is there an "on unready" event?
* If an image is evicted and then re-decoded, is there a second ready event? The original proposal said it’s just the first time, but we’re worried this is too confusing and can lead to jank in subsequent uses.
* If an author has a series of images (e.g. carousel) out of dom, how does an author communicate which ones are important to decode first?

Here's two rough proposals to handle each of these use cases.

(1) Proposal for handling deferring images:
Give the async attribute three values: async, auto, and sync. sync behaves as today’s image elements do without any attribute specified, where once an image has been loaded it will appear immediately (and be decoded synchronously) if inserted into the document. Images marked async can get loaded/decoded best-effort without janking. auto would involve browser heuristics to decide if the image could and should be loaded async or if it needs to be sync. async is therefore just a more aggressive version of auto in practice. sync is mostly just there as a safety valve for developers in case browser heuristics get it wrong.

There’s no ready event here, as it’s not part of this use case. In particular, this helps avoid confusion around what happens when a decoded image is discarded and then later decoded again (e.g. because the image was scrolled out of view and then back in). The browser can treat it the same way regardless of whether it was the first time the image was being decoded or not.

We’re not 100% sure we’ll need the async value without more experience on trying to ship changes here. It might be that we can make all images that we want to async and just have auto and sync in the end.

(2) Proposal for predecoding images (similar to what @domenic suggested):
Provide an explicit decode function for an image with some sort of callback when the image is decoded. This would notify the author that it's ready to be appended to the page without jank. The browser would promise to keep that image decoded for at least that raf frame. It would also allow an author to call this decode function again in the future if needed, and to prioritize images in an image carousel case. Finally, we might want to make this decode function return some success/failure (in case of an overloaded cache).

Possible sample code:
var img = new Image();
img.src = ‘...’;
// The decoded image is guaranteed to be kept in memory until the frame
// after the decode promise resolves.
img.decode().then(function() {
// This is guaranteed to paint the image without flicker or decoding jank.
document.body.appendChild(img);
});

Other side questions:
* What about large images that are scaled and cropped? Do these parameters need to be specified? We found that scaling and gpu uploading can be a significant part of image preparation time. Do we just punt on this?
* How is image data that isn’t image elements handled, e.g. should we add an async keyword for background images? Divorcing async from decode makes it easier to do something consistent between image elements and CSS background images.

[smfr]

I like this analysis.

(1) Proposal for handling deferring images
... Images marked async can get loaded/decoded best-effort without janking

Presumably the effects of "async" on a content image is that it can show blank after its load event has fired, right? Seems fine if so. "auto" is weird as an attribute; I would prefer that we say that "async" is a hint and the UA is free to ignore it; if it becomes a cargo cult, I don't want to have to respect it on thousands of images in a document.

(2) Proposal for predecoding images
I like the decode() Promise form; it makes it easy to figure out the behavior with repeated calls on the same image.

What about large images that are scaled and cropped? Do these parameters need to be specified? We found that scaling and gpu uploading can be a significant part of image preparation time. Do we just punt on this?

I say punt; doing scaling/cropping is usually a paint-time operation, and if you tried to do something for these, you'd end up having to do lots of things in the painting pipeline.

• How is image data that isn’t image elements handled, e.g. should we add an async keyword for background images?

Since you can't detect load on CSS images directly, it seems less important to support "async" for them.

[vmpstr]

Presumably the effects of "async" on a content image is that it can show blank after its load event has fired, right? Seems fine if so. "auto" is weird as an attribute; I would prefer that we say that "async" is a hint and the UA is free to ignore it; if it becomes a cargo cult, I don't want to have to respect it on thousands of images in a document.

To clarify a bit, the proposal is to have a single attribute called async, which could have three possible values (async/sync/auto). I agree that these values should still be just hints to the UA and it can ignore them.

The idea behind auto value and how it's different from async is that it allows the UA to adjust its heuristics to either include or exclude more images from being async based on whatever criteria it decides. In other words, async value would mean it's always safe to show a blank image after the load event fired; auto would mean that the UA should decide whether or not it is OK do this.

It's not entirely clear that we need both auto and async, but if auto is the default it would allow the UA to make these decisions for content that didn't explicitly specify what it wants.

[vmpstr]

decoding=sync don't actually require sync decoding that blocks the main thread -- they just require an event that fires when decoding is complete

I'm not sure that there is even a requirement for the event to fire when decoding is complete. The way the proposal specs this, there is no requirement for any javascript code to be able to determine the effect of different values of "decoding" attribute (other than inspecting the attribute itself of course).

I agree with @othermaciej that I think the cases where the event is desired are best suited for the img.decode() api.

[aghassemi]

@vmpstr @smfr @othermaciej One common use-case is hiding a placeholder (e.g. small, in-lined pixalated version of an image) or loading indicator as soon as image is painted.

If we use decoding=async and use load event to hide the placeholder, we may hide it too early.
If we use image.decode() before appending the image instead of attr, we lose progressive decoding.

An event to know decoding is finished combined with decoding=async seems to be needed.

@smfr, as a workaround, I am curious if we can still use image.decode() purely as a signal to know when decoding is done and combine it with the decoding=async attribute. Can this cause double work?

[smfr]

An event to know decoding is finished combined with decoding=async seems to be needed.

But the browser doesn't want to decode images that it doesn't know are going to get painted. This is extra work and possibly massive memory use.

@smfr, as a workaround, I am curious if we can still use image.decode() purely as a signal to know when decoding is done and combine it with the decoding=async attribute. Can this cause double work?

image.decode() is for forcing an (async) decode, not just for detecting when it's done. I don't think using the two things together makes much sense.

[aghassemi]

@smfr I didn't have the expectation that it will fire for every image. Maybe an event name like painted is more suitable than decoded to convey that behavior. This will still be very useful to hide partial loading overlays or background placeholders perfectly in sync with image painting.

[vmpstr]

The decision to paint (rasterize) content can happen pretty far down the pipeline and independently of the main thread. I think bubbling up an event whenever we decode an image is a lot of overhead, especially if there are only a few cases where it is useful. It's also a bit unclear if the event should be fired for subsequent decodes that may happen on the same image.

In general, the situation you describe would be best suited for (intended use of) the decode api. That is, show a low res image, call a decode() on a full res image and append/replace the content in the promise resolution. You are right that since this implies the load event, we'd lose the progressive decoding.

Just thinking out loud, but I don't readily see an elegant way to get all three of the following:

1. async behavior
2. progressive decoding
3. a signal that the final image is decoded

1+2 sounds like it could be handled by decoding=async
2+3 is implied by not specifying anything (ie decoding=auto or maybe even decoding=sync)
1+3 is the decode() api

[annevk]

Also thinking out load: I guess in the future we could add something to decode() that roughly means "take your time"? So you can invoke it early, but the browser knows you're just using the resolved promise as a signal and is therefore not in a hurry to decode.

[aghassemi]

@vmpstr curious why bubbling up an asyc flag from down the stack is expensive? I assume events like playing for video come from deep parts of the stack as well.

For now we are doing 1+2 and will assess if the transition between placeholder/image would flicker. /cc @cramforce

@annevk that's what I sort of assumed decode() already does since I was picturing it as being just the imperative version of the decoding=async attribute.

[vmpstr]

The difficulty with something like a decoded attribute is that its value may change at times that I would say are not interesting. For example, in Chromium images are sometimes decoded outside of the viewport based on some heuristics that suggests that we may soon need to rasterize content that uses these images.

However, due to user interactions we may not actually use the image that we decoded. Furthermore, the decode's memory backing may be discarded meaning the image is no longer decoded, but doing the query for this information is done only if we actually need the image to rasterize. Once we use an image to rasterize content, it may also be evicted since its memory backing is no longer necessary to present content.

If I'm understanding this correctly, to implement a decoded attribute, each event that causes a decode or eviction would have to come with corresponding communication with the main thread so that javascript can observe the change. (and this would have to happen for every image?)

So perhaps saying that this is a lot of overhead was premature. I wouldn't necessarily claim that this is either cheap or expensive; I'm simply skeptical that the value this attribute brings warrants this complexity. If there's an interest in this type of attribute, maybe we can split off the discussion into a separate issue?

[cramforce]

I think a decode event is too coupled to the implementation. What would be interesting is an event for "I'm about to or I could without extra decoding work paint pixels for this image. Either the full image or the initial progressive render."

[othermaciej]

If the requirement is to have a signal when the image is decoded enough to paint, even if not fully decoded, maybe that could be a parameter to image.decode(), which would then start decoding as usual, but trigger the promise at minimally ready to paint time instead of fully decoded time.

[othermaciej]

(I don't think you want to trigger based on the image actually being painted, since it might not even be inserted into the DOM until ready to paint, and since layout engines might be smart enough not to paint it at all if it's fully covered by an opaque placeholder.)

[chrishtr]

Hi all,

I propose to consider this issue complete for the purpose of committing the spec update in
#3221. The discussion happening now is good, but probably
best in a new issue to separate concerns.

Any objections?

[annevk]

I also lean in that direction. We can have a new issue to discuss a signal of sorts for the scenarios discussed above. I suggest we merge #3221 (and close this issue) somewhere Tuesday to give folks a few more days to raise concerns with this line of action.

[sideshowbarker]

Added https://wpdev.uservoice.com/forums/257854-microsoft-edge-developer/suggestions/32541025-img-decoding-attribute