Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Canvas

Servo supports four types of canvas context:

Each canvas context implements the CanvasContext trait, which requires contexts to implement some common features in a unified way:

  • context_id
  • resize: this method clears the painter's image by setting it to transparent alpha (all bytes are zero)
  • get_image_data: used when obtaining the canvas image, usually by calling toDataUrl, toBlob, createImageBitmap on the canvas or indirectly by drawing one canvas in another
  • update_the_rendering: for triggering update of image (usually by swapping screen-buffer and back-buffer)
  • canvas: obtain connected canvas element (this can be HTMLCanvasElement or OffscreenCanvas, which can also be connected to HTMLCanvasElement with context set to placeholder) while also providing some good default implementations (onscreen, origin_is_clean, size, mark_as_dirty). mark_as_dirty is called from functions that affect the painter's image and tells layout to rerender the canvas element (by marking HTMLCanvasElement as dirty node).

HTML event loop and rendering

flowchart TB
    subgraph Content Process
        subgraph Script flow
            JS-->utr[Update the rendering]-->Layout
        end
    end

    subgraph Main Process
        subgraph Painters
            WGPU[WGPU thread]
            WEBGL[WebGL thread]
            CPT[Canvas Paint Thread]
        end

        %% actual update in painters
        Painters--CreateImage-->Compositor
        Painters--UpdateImage-->Compositor

        Compositor-->WR

        WR[WebRender]--lock,unlock-->Painters
    end

    %% init canvas
    JS--create context-->Painters--ImageKey, CanvasId-->JS

    %% update canvas rendering
    utr<--Update rendering-->Painters

    %% rendering
    Layout--DisplayList (contains ImageKey)-->Compositor

As part of the HTML event loop, the script thread runs a task (parsing, script evaluating, callbacks, events, ...) and after that it performs a microtask checkpoint that drains the microtasks queue. In the window event loop we queue a global task to update the rendering if there is a rendering opportunity (usually driven by compositor based on hardware refresh rate). In Servo, we do not actually queue a task, but instead we run update the rendering after any IPC messages in the ScriptThread and then perform a microtask checkpoint too as the event loop would have done after a task is completed. Update the rendering does various resize, scroll and animations steps (which also includes performing a microtask checkpoint to resolve outstanding promises) and then run the animation frame callbacks (callbacks added with requestAnimationFrame). At this point draw commands are issued to painters to create a new frame of animation. Finally, we trigger reflow (layout), which first updates the rendering of canvases (by flushing dirty canvases) and animated images, then traverses the DOM and its styles, builds a DisplayList, and sends that to WebRender for rendering.

When canvas context creation is requested (canvas.getContext('2d')), the script thread blocks on the painter thread as it initializes and creates a new WebRender image (CreateImage), finally sending the associated ImageKey back to script.

sequenceDiagram
    Script->>Constellation: Create Context
    Constellation->>Painter: Create Context

    Painter->>Compositor: GenerateImageKey
    Compositor->>WebRender: GenerateImageKey

    opt 
        Painter<<->>Compositor: ExternalImageId
    end

    WebRender->>Compositor: ImageKey
    Compositor->>Painter: ImageKey

    Painter->>Compositor: CreateImage
    Compositor->>WebRender: CreateImage

    Painter->>Script: PainterIPCSender, ImageKey, CanvasId

Each canvas context implements LayoutCanvasRenderingContextHelpers, which returns the ImageKey that layout will use in its DisplayList, or None if the canvas is cleared or otherwise not paintable due to its size. WebRender will read the resultant image data when rendering, based on the provided ImageKey. In WebGL and WebGPU painters this is done by implementing a custom WebrenderExternalImageApi; this provides lock and unlock methods for WebRender to obtain the actual image data. For 2D canvases, image data is directly provided via CreateImage and UpdateImage IPC messages.

sequenceDiagram
    Script->>Painter:Update rendering (flush)
    Painter->>Compositor:UpdateImage
    Compositor->>WebRender: UpdateImage
    opt
        Painter->>Script: Done
    end
    
    Note over Script: Layout
    Script->>Compositor: DisplayList
    Compositor->>WebRender: DisplayList
    opt
        Compositor<<->>WebRender: Query ExternalImage Registery
        WebRender->>+Painter: lock ExternalImage
        WebRender->>Painter: unlock ExternalImage
        deactivate Painter
    end

2D canvas context

flowchart LR
    CanvasRenderingContext2d --- HTMLCanvasContext
    subgraph OffscreenCanvasRenderingContext2D
        subgraph CanvasRenderingContext2d
            CS'[CanvasState]
        end
    end
    OffscreenCanvasRenderingContext2D --- OffscreenCanvas
    PaintRenderingContext2D --- PaintWorklet
    subgraph PaintRenderingContext2D
        CS''[CanvasState]
    end

While most canvases use the same DOM type for their onscreen and offscreen contexts, this is not the case for 2D canvases due to their long history. Web standards define three types of 2D canvas context:

  • CanvasRenderingContext2D (connected to HTMLCanvasContext)
  • OffscreenCanvasRenderingContext2D (connected to OffscreenCanvas)
  • PaintRenderingContext2D (only available in PaintWorklet)

CanvasRenderingContext2D and PaintRenderingContext2D are implemented as wrappers around CanvasState, while OffscreenCanvasRenderingContext2D is implemented as a wrapper around CanvasRenderingContext2D because of similar logic to avoid duplication.

flowchart LR
    HTMLCanvasElement --getContext('2d')--> CanvasRenderingContext2d
    CanvasRenderingContext2d --strokeRect--> CanvasState
    CanvasState --IPC
    strokeRect--> CanvasPaintThread
    CanvasPaintThread --Done--> CanvasState

CanvasState implements the actual logic of 2D drawing, by setting appropriate state and sending IPC messages to the Canvas Paint Thread. Some commands only change internal state, but don’t need to send any messages until there is an actual draw command.

All "dirty" 2d canvases are stored in Document and are flushed during reflow, by sending IPC messages that trigger the update_the_rendering method on each canvas.

When drawing one 2D canvas into another 2D canvas, we send DrawImageInOther, a special IPC message that avoids copying the bitmap out of the canvas paint thread.

WebGL canvas context

flowchart LR
    WebGLRenderingContext --- c["HTMLCanvasElement
    OffscreenCanvas"]
    subgraph WebGL2RenderingContext
        WebGLRenderingContext
    end

WebGL(2) canvas contexts are WebGLRenderingContext or WebGL2RenderingContext, and in Servo WebGL2RenderingContext wraps and extends WebGLRenderingContext. These contexts store state and send IPC messages to the WebGL thread, which executes actual OpenGL (or OpenGL ES) commands and returns results via IPC. The script thread blocks on the WebGL thread, waiting for each operation to complete.

All "dirty" WebGL canvases are stored in Document and are flushed as part of reflow, by sending one IPC message containing all dirty context ids, then blocking on the WebGL thread until all canvases are flushed. Flushing swaps the framebuffer, where one is for presentation (that is read by WebRender) while the other is used for drawing as the target of GL commands.

WebGPU canvas context

WebGPU presentation is the most special as it is fully async (non-blocking). More info about how async is done in WebGPU can be read in the WebGPU chapter.

sequenceDiagram
    loop Context Creation
        Script->>WGPU: CreteContext
        WGPU->>WebRender: CreateImage
        WebRender->>WGPU: ImageKey
        WGPU->>Script: ImageKey
    end

    alt animation Callback
        Note over Script: getCurrentTexture
        Script-)WGPU:CreateTexture
        activate Script
        Note over Script: draw operations into current texture
    else update the rendering
    Note over Script: expire current texture
        Script-)WGPU:SwapchainPresent
        WGPU-)WGPU: Copy texture to one of staging buffer
        WGPU-)+WGPU poller: Map stagging buffer to CPU as GPUPresentationBuffer
        opt presentationId is newer than existing
            WGPU poller -)-WebRender: UpdateImage
        end
        WGPU poller -)WGPU poller: Unmap GPUPresentationBuffer

        Script-)WGPU:DestroyTexture
        deactivate Script

    end
    loop rendering
        WebRender<<->>+WGPU: lock ExternalImage and read GPUPresentationBuffer
        WebRender->>WGPU: unlock ExternalImage
        deactivate WGPU
    end

All onscreen WebGPU contexts have their update_the_rendering executed as part of updating the rendering in the HTML event loop. This expires (destroys) the current texture, but before that we send a SwapChainPresent request, which copies texture data into one of 10 presentation buffers on the GPU. After copying is done, we async map the new buffer to CPU. Because this process is async, we mark each presentation buffer with an incrementing u64 id, and only replace the active presentation buffer if our buffer’s id is newer. The inactive presentation buffer gets unmapped.

flowchart TD
    S[Staging Presentation Buffer] --copy_texture_to_buffer, mapAsync-->
    Mapping --mapAsync done-->
    UpdateWR --yes-->
    Mapped[Mapped, Unmapped old]

    UpdateWR--else unmap-->S

This is also modeled in TLA+: https://gist.github.com/gterzian/aa5d96a89db280017b04917eee67f6ac

Both WebRender's lock and get_image_data will use content of the active presentation buffer.

Resources