Coordinated Image Loading with HTTP/2

Summary

This post gives the technical details needed to handle effectively the ShimmerCat feature coordinated image loading.

Introduction

A site heavy in images, or even a rather normal website these days loading over a slow network connection may have troubles visualizing the images as they load. For example, a dark background image won’t be apparent until there are at least some bytes to show, producing a flash in the site late in its load sequence. Or a web application site with an image gallery may end up showing thumbnails that happen to also be in the view, before a more important image which is expected to receive the immediate attention of the user. Browsers do their best to come up with sensibles ways of managing bandwidth, but they don’t always get it right.

To help in those cases, ShimmerCat is the first web server to allow for manual allocation of bandwidth for images over HTTP/2 connections. We call the feature coordinated image loading, and we will dedicate a coming blog post to a working example. This page just gives the technical details needed to handle effectively this technique.

The simplest case

At its core, ShimmerCat manages data that is about to be sent to a browser through queues. The elements of the queues are HTTP/2 frames, and for images, those frames are grouped into segments.

For non-progressive, non-interlaced images, there are only two definite segments: an initial segment that contains the image metadata, and the rest of the bytes up to the end of the file.

For progressive or interlaced images, there is an initial segment for metadata, and another segment for each block of data that would allow the browser to render a pass of the image. That is, eight segments in total for .png files, and a segment for each scan (denoted by SOS markers) in JPEG files.

The utility of this can be seen with a simple example:

<img src="/red.png">
<img src="/blue.png">

Notice that these <img> tags do not include size information, so it is very likely that the browser will need to re-layout the document as soon as any of them is received. Wouldn’t it be useful if the metadata of each image, including its size, were received by the browser as soon as possible? Then re-layouts would happen just at the very beginning of the page load, and not when the user has already started scanning or reading the document.

That’s exactly what coordinated image loading does by default: it assigns very high priority to the first segment of each image. In the example above, it means that the size information related to blue and red will be prioritized, while the bytes with pixel data will come later.

When more control is required

It is possible to assign an explicit priority to the first segment of each image, and then a decrease in priority for each following segment. There is a quirk however: we represent a value which is semantically the opposite of priority. Let’s call it calm. The higher the value of the calm, the lower the priority.

Therefore, to assign a segment the highest priority, we give it the lowest calm, zero.

We set the value of the calm through a URL query parameter, bil. Then in the same parameter, we specify increases to the calm value after an exclamation mark. To give increases for multiple consecutive segments, we separate the increases with commas.

For example, assuming red.png and blue.png are progressive PNGs:

<img src="/red.png?bil=0!100,50,20">

<!--
Some more contents sit in the middle
-->

<img src="/blue.png?bil=0!,1">

This tells the server that the metadata block for both images should go with the highest priority (“minimum calm”). After that, red.png receives a calm boost of 100, so the first segment of pixel data will go with a calm value of 0 + 100 = 100. Then the second will go with calm 100 + 50 = 150, and the third with calm 150 + 20 = 170.

For blue.png, the segment after the metadata goes with calm 0 + 1 = 1, and that value of the calm is kept for all the following segments, if any.

Notes

Availability

Coordinated image loading is available from version 1.3 of ShimmerCat. Notice that this only works over HTTP/2 connections.

Performance

Scanning the images to delimit their segments implies very little work. Furthermore, dividing the file in segments is done only the first time that an image is requested, and from there on ShimmerCat stores the metadata in its representations cache.

Why only with HTTP/2

This feature is made possible by the multiplexing nature of HTTP/2. In an HTTP/1.1 world, to achieve the same effect a whole TCP socket has to be given to a resource while it is downloading. Since the number of sockets is limited both for the server and for the browser, this technique is unaceptable for HTTP/1.1 connections.