✨ feat(video-engine)!: a browser video engine, and annotations that draw on the frame it presents

[cfviotti]

Description

The problem

supervision already plays a video. It opens the file through mediabunny, and the renderer asks the
source for a sample at a time the renderer picked.

What it does not have is exact frame identity. The frame index it reports is
round((mediaTime - firstTimestamp) * estimatedFrameRate), and its own type documents that as an
estimate. That arithmetic names the wrong frame on:

• fractional frame rates, 29.97 and 59.94;
• non-integer frame timestamps;
• container timebases that are not milliseconds;
• variable frame rate.

The failure is quiet. The pixels stay correct and only the annotations move. It survives pausing and it
looks like bad model output.

This pull request adds a browser video engine. The engine, not the renderer, decides which frame is
on screen. It publishes that frame with its own media time and its identity in the container's
timebase, and every annotation layer draws against that one value.

Who this is for

• Applications that review model output on video: annotation tools, evaluation views, demos.
• Hosts that already ship their own decoder and want to keep it. The push path is public.
• Existing image and camera consumers. The engine is not on their path: they keep their own
  renderer sources and the renderer keeps picking the time. What does reach them is the
  maxDevicePixelRatio default in What breaks, which caps every presentation surface at 2.

The pipeline

flowchart LR
  SRC["Media source"] --> ENG["Video engine"]
  ENG -->|"presented frame + media time"| PRES["Video presentation"]
  ENG -->|"presented frame + media time"| DET["Temporal detections"]
  INF["Inference or fixtures"] --> DET
  PRES --> R["Renderer"]
  DET --> R
  R --> CAN["Canvas"]

Loading

Which path a source takes

flowchart TD
  A["Source opened"] --> B{"Which renderer source<br/>did the caller pass?"}
  B -->|"the default one"| PULL["Pull path"]
  B -->|"createWebVideoEngineMediaRendererSource"| C{"Engine chunk loaded?"}
  C -->|"no"| ERR["Throws, and names supervision/web-video-engine"]
  C -->|"yes"| D{"H.264 with an avcC record?"}
  D -->|"yes"| S["One decode session, held across seeks"]
  D -->|"no"| K["mediabunny sinks, re-positioned per request"]
  S --> PUSH["Push path"]
  K --> PUSH
  PULL --> P1["The renderer picks the time.<br/>It reads the sample timestamp."]
  PUSH --> P2["The engine picks the time.<br/>It publishes the presented frame."]

Loading

The library does not read the media and choose. The caller chooses by which source it passes. A
video file passed as a URL opens through mediabunny and takes the pull path.

The pull path is what existing consumers use. Its shape is unchanged: the renderer picks the time
and reads the sample timestamp, as it does on main. Its behaviour is not. A container that opens
with no parsed track now fails as UnsupportedFormat, where main failed it as NoVideoTrack.
errorKind is the field to branch on, so a consumer that reads it sees this. The default changes in
What breaks reach the pull path as well.

What the engine does

packages/video-engine is a new workspace package. It is private, so npm never publishes it. Its
build is staged into supervision, and consumers import it from supervision/web-video-engine. It
does not depend on the renderer.

Capability	Detail
Decode	One long-lived decode session across seeks for H.264 tracks with an avcC record. Other codecs decode through mediabunny's sinks, which re-position on each request.
Seek	Keyframe-aware. A target behind the read head restarts from the keyframe before it and decodes forward.
Scrub	Continuous, forward and backward. Gesture direction is read from a bounded ring of the last 8 samples, so one jittery pointer move cannot swing it and a reversal keeps a share of the prefetch window behind the pointer.
Cache	Two tiers: full-resolution frames near the playhead, downscaled frames over more of the timeline. Both are written from one decode. Both tiers are sized from the device's reported memory inside fixed byte clamps; the decode resolution sets the per-frame cost and the floor.
Frame identity	An index and a tick count in the container's own time grain. Built from each packet's own timestamp.
Ownership	Every decoded frame has exactly one owner responsible for closing it. A host that never closes one is told so.
Zero copy	Where WebGPU exposes importExternalTexture and the strategy decodes at the source's native size, a frame is sampled without a copy. Naming a display box makes the engine decode at the display size instead, which rules the path out. The demo names one.
Diagnostics	A trace recorder exports a capture as JSON. armTrace(windowMs) sizes the ring from the broadcast rate.
Frame extraction	The analysis entry point opens a source and pulls frames without a player.

Presented-frame identity

The frame table is built from each packet's own timestamp. That is what makes identity survive a
fractional rate or an unusual timebase.

The position published for a frame comes from the packet that was submitted. It does not come from
the timestamp the platform decoder returns. A decoder that counts from its own origin, or reorders,
therefore cannot slide the annotations off the picture.

A tripwire in the present throws if any layer is handed a media time other than the presented one.
It is armed in every build, production included. It costs about eleven comparisons per presented
frame.

Two rate-derived indexes remain. Both are documented as estimates, not identity:
estimatedFrameIndex in the renderer state, and the NearestFrameIndex detection selection mode.

Temporal detections

Detections are temporal data, independent of decoding. They can be precomputed, appended while
playback runs, or composed from several sources. Overlapping results update the active range without
rebuilding the whole annotation state.

Two producers exist in this repository: precomputed fixtures, and a remote model that pulls frames
from the engine's sample sink.

Prepared annotation rendering

The renderer prepares annotation artifacts ahead of the playhead. It keeps prepared frames on both
sides of it in a bounded cache, so a reversing scrub finds work already done.

On the push path, rendering is event-driven. Pixi's ticker is unused. The scene draws only on a
change: a new presented frame, a detection change, a prepared artifact landing, a hover or selection
change, or a presentation change. A paused scene nobody touches submits no frames. The pull path
still repaints on the ticker.

Masks

A worker builds one byte per pixel holding a detection id. A shader colours those ids from a palette
on the GPU. Where that raster cannot be built, the same worker produces an RGBA composite.
Boxes, labels and vectors draw as before; the hover silhouette is the one thing lost, because
the ids it needs are what the raster carries. The mask layer reports that state rather than
leaving it silent.

The palette holds 80 entries. One entry is the background, so a raster can name 79 detections. It is
keyed on detection index: a mask writes its detection's index plus one. A frame past the ceiling
falls back to the RGBA composite. That path walks each mask's runs rather than the whole plane
once per detection, which on 81 masks over 1920x1080 costs 19 ms for fills and 69 ms with
outlines, where walking the plane cost 151 and 337. The raster path is 1.5 ms. On the 2113-frame horse trail clip, 75
frames used to fall back and no longer do.

A host can declare the box it paints masks into, through
renderPreparation.maskFrame.display. The raster is then built at the size that box can show. Left
unset, masks are built at the detections' own resolution.

The push path runs on WebGPU where images stay on WebGL. Every shader therefore carries a WGSL
variant, and a test requires each shader to carry a program for both backends.

What is new in the public API

37 exported names are added and none is removed. The largest group answers a question a host could
not ask before: why will this file not play?

Added	For
WebVideoEngineErrorCode	Say why a file will not play. Ten codes, from DecodeUnsupported to RateUnsupported. Reached at supervision/web-video-engine.
createWebVideoEngineMediaRendererSource, openWebVideoEngineMediaSource, WebVideoEngineMediaSource	Open a video file through the engine.
PresentedFrameChannel, PresentedFrameSource, PresentedFramePlayhead and their signal types	Write your own push source.
PreparedAnnotationWindowSnapshot, PreparedAnnotationWindowFrame, PlaybackGateReach	Read how far preparation and the gate have reached.
resolveMediaSessionDefaults, ResolvedMediaSessionDefaults	Show a viewer the buffering numbers the session resolved, rather than a copy that drifts.

DecodedMediaSource declares both drive modes, and both are public. sampleSink answers
getSample(timestamp) for a time the renderer picked. engine is a PresentedFrameChannel: the
source announces each frame it puts on screen. A host with its own decoder can implement the push
path rather than only consume this engine's. sampleSink stays required either way, and the engine
supplies a real one over its batch analysis path, which is what serves thumbnails and one-off frame
grabs.

Out of scope

• Choosing a model. Storing detection data. Application-level editing.
• Audio. The renderer is video-only and audio playback is deferred.
• Reverse playback. Rates outside 0.25x to 8x throw, negative rates included.
• Moving presentation off the main thread. The cost is measured and the decision is deferred to its
  own pull request.

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation or example update
• Refactor or maintenance
• Performance improvement

Validation

• Tests added or updated where behavior changed
• Documentation updated where the public API or workflow changed
• Screenshots or recording attached for visual changes

How to run it

npm ci
npm run verify      # the full gate: boundary, format, lint, typecheck, tests, builds, smoke tests
npm run dev         # package watchers plus the demo server
npm run docs:serve  # the documentation site

Numbers, measured at this commit

Measured	Value	How
Test files	214, all passing	npx vitest run
Tests	2,444, all passing	npx vitest run
Engine test files	38	packages/video-engine/src/*.test.ts
Documentation contract checks	31, all passing	npm run docs:check
Browser evaluation metrics gated	30, across 11 families	METRICS in tools/demo-eval/baseline.mjs
Frame-table ceiling	1,000,000 frames: 9.3 h at 30fps, 4.6 h at 60, 2.3 h at 120	FRAME_TIMELINE.MAX_FRAMES
Mask palette ceiling	80 entries per frame	MAX_ID_MASK_PALETTE_ENTRIES
Playback rate range	0.25x to 8x, forward only	PLAYBACK_RATE
Detection selection tolerance	0.5 ms of playhead quantization	PLAYHEAD_QUANTIZATION_TOLERANCE_SECONDS

The engine's 38 test files cover decoding, timelines, cache behaviour, scrub trajectories, playback
scheduling, frame ownership, worker communication and presentation. They run in Node against fake
browser APIs and a recorded packet table.

Where the numbers come from

Every performance figure in this description was measured on an Apple M3 Max, 16 cores, 64 GB, in
Chrome, against the 70-second horse trail clip at 30fps through the WebGPU renderer. The clip
carries 2,113 frames and 98,115 detections, which is 46 a frame.

Nothing here is measured on a slower machine, at a higher frame rate, or on a denser clip.

The browser evaluation harness

npm run eval:demo drives the running player and gates 30 metrics against a recorded baseline. It
exits non-zero on a regression. The baseline records the machine, the commit, the clip and whether
the tree was dirty. The repository ships no baseline file and gitignores it: one recorded on a
given processor is only meaningful on that processor, so it stays local.

Ten families: sync, latency, layers, cadence, throttle, blanking, drag, playhead,
backscrub, focus.

The layers family carries a hard budget: zero frames over 34 ms, in every layer combination.

Playback rate, presented-frame identity, cache behaviour and cache memory ceilings are covered by
engine unit tests instead. The harness only measures playback at 1x.

Reviewer checklist

• npm ci && npm run verify from a clean clone.
• Load every fixture in the demo picker and confirm each one draws. Use Chrome. Firefox cannot
  open the HEVC fixture.
• Drag the timeline backwards with masks on.
• Drag a detection, then resize it.
• Change a style through setPresentation while keeping the same detection array.
• Read the breaking-change table below against any custom MediaRenderer or interaction-style
  code.

Notes For Reviewers

Answers to review feedback

1. "Polyline rendering broke on a docs page."

You were right, and it is fixed. It was the fixture, not the polyline renderer.

The polylines page embeds the demo with the basketball_sam3 fixture. The page filters to
className === "basketball" and metadata.trajectoryTrackId === "basketball-track:0".

SAM3 returns a whole-scene answer for that prompt alongside the ball. The fixture's trajectory
step accepted the whole-scene mask as the tracked ball and stamped the track id on it. The page's
filter then kept it faithfully. The precise shape of the defect:

On the broken fixture	Measured
basketball-track:0 detections with rect exactly 1920x1080	199 of 225
Polyline points within 40 px of frame centre	4,857 of 5,499 (88.3%)
Where the trace head parks	Frame centre from frame 26 (t = 1.04 s) onward

af35486 rebuilt the trace and refuses any candidate covering 50% or more of the frame.
demo/src/fixtures/demo-fixtures.test.ts now gates it: widestFrameCoverage must stay under 0.5.
That assertion evaluates to 1.0 on the old data, so it is the regression gate for exactly this.

At HEAD the trail is a ball trail: 216 polylines, all on the ball track, footprint 0.013% to 0.221%
of the frame.

The polyline renderer itself is untouched by this branch apart from the new shadowStroke default,
which landed after your report.

2. "Make 'buffered by detections' part of the createMediaSession API."

Done. createMediaSession takes playbackGate, a plain boolean. You either want that playback
mode or you do not, which is the shape you asked for.

createMediaSession({ playbackGate: false }); // start at once, draw annotations as they land

It is an umbrella switch over two gates. Either gate can still be set on its own, through
detections.playbackGate and renderer.renderPreparation.playbackGate.

Gate	Default when you pass nothing	What it holds for
Render preparation	On	Prepared raster artifacts: masks, polygons
Detections	Off, unless the session has appendable detections, or you pass playbackGate: true	Detection frames arriving and covering the playhead

Neither default changed. Both were already resolved this way on main. What was missing was a way
to say yes or no to the whole thing in one place, and a gate that reached a source presenting its
own frames at all: on main the wait lived in the renderer's sample pump, which such a source never
enters.

The docs page you saw playing bare now waits. The masks page embeds the demo. The demo opens a
sample on the Mediabunny media path, which the renderer pulls samples from. A pull source is held at
every frame whenever any gate is on, so playbackGateReach reports EveryFrame. The sample passes
no session gate, so render preparation is the gate holding it. The detection gate stays off, because
a sample ships its annotations with it.

One nuance worth knowing before you rely on it. On a source that presents its own frames, the two
gates reach different distances:

flowchart TD
  P["play()"] --> G{"playbackGate"}
  G -->|"off"| RUN["Frames arrive at once"]
  G -->|"on, pull path"| E["Held at every frame<br/>reach is EveryFrame"]
  G -->|"on, push path"| Q{"Which gate?"}
  Q -->|"render preparation"| S["Held at the start only<br/>reach is StartOfPlayback"]
  Q -->|"detections"| E

Loading

The render-preparation gate is awaited once inside play() on a push source, because stopping the
producer mid-run needs an answer about coverage that render preparation cannot give without waiting
for it. The detection gate does stop and restart the producer, so it holds every frame on both
paths. A pause or a scrub during the wait abandons the play, so readiness landing later does not
start a picture the viewer stopped.

What breaks

This takes supervision to 0.2.0-next.0, published on the next tag. latest stays on 0.1.7 until
0.2.0 goes out from main. The pinned public surface goes from 405 exported names on main to 425:
20 added, none removed. The engine's own names are not among them: they reach consumers at the
supervision/web-video-engine subpath. Every break below is a change to the shape of a type, or to what a
default does. Rows are ordered by how easily each slips past a consumer.

Change	How you find out	What you do
Four fields are gone from BaseInteractionStyleOptions: shape, cornerRadius, stroke, fill. All four were already @deprecated on main.	TypeScript stops the build. Plain JavaScript says nothing, and your custom highlight silently becomes the built-in one.	Move them into hovered.boxStyle and selected.boxStyle, which reach mask, label, keypoint, polygon and polyline highlights too.
requiredForPlayback is now requiredForCoverage.	TypeScript stops the build. Plain JavaScript says nothing, and a false reverts to the default true, so the composed source waits on that entry again.	Rename it. Polarity and default are unchanged.
Five protected resolvers are gone from BaseInteractionStyle: resolveBoxInstruction, resolveShape, resolveCornerRadius, resolveStroke, resolveFill.	Nothing, unless you compile with noImplicitOverride. A subclass that overrode one keeps compiling and stops being called.	Style through hovered.boxStyle and selected.boxStyle.
A container that opens with no parsed track now fails as UnsupportedFormat, where main failed it as NoVideoTrack.	Nothing. A branch on NoVideoTrack stops matching that file and falls through to your generic handler.	Branch on UnsupportedFormat as well. A container whose tracks read and carry no video still fails as NoVideoTrack.
The detection chunk cache raises its own ceiling to twice the widest buffer window it has served, from a floor of 12 chunks, and never lowers it.	Nothing. Backward scrubbing finds more in memory. A long session holds more of it.	Pass maxCachedChunks for a fixed cap.
MediaRenderer gains four required members: togglePlayback(), scrub(), getRenderCount(), getPreparedAnnotationWindow().	TypeScript stops the build, in your code.	Implement them, or narrow the annotation to Pick<MediaRenderer, ...>. Anyone who only calls createMediaRenderer() is untouched.
maxDevicePixelRatio left unset now caps the presentation surface at 2, where main rasterized at the display's own ratio.	Nothing, above 2x: the picture is drawn at 2 and looks slightly softer. Below 2x nothing changes.	Pass window.devicePixelRatio explicitly for the old behaviour. The cap is what puts the surface, the mask rasters and the decode on one grid. A mask raster can only be sampled nearest, so a grid it did not share showed as stair-stepped edges.
A trajectory drawn with the default polyline style now sits on a dark contrast stroke.	Nothing. An orange ball trail over a wooden court becomes readable. A path already on a contrasting background gains a thin outline.	Pass shadowStroke: null to BasePolylineStyle to draw the path bare.
Detections for a file are re-derived every 2.5 s, where main re-derived every 0.5 s.	Nothing. A window that does not reach the playhead still reloads at once, so this only changes how often covered ground is derived again.	Pass detections.buffer.refreshIntervalSeconds for the old cadence. Streams are unchanged at 0.25 s.
A file session buffers ten seconds of detections ahead of the playhead and five behind, where main buffered ten ahead and half a second behind. Core's own defaults move the same way, from five and half a second.	Nothing breaks. Annotations survive a backward scrub where they used to blink out. The lookahead is main's; what changed is how much ground behind the playhead stays buffered. A narrower lookahead was measured and rejected: over 48 runs six seconds ahead lost to ten in 11 of the 12 backward cells and tied in all 12 forward ones, so the window was widened rather than shifted. If you measured memory, the window is 15 seconds against 10.5.	Nothing. To pin the old window, pass detections: { buffer: { bufferAheadSeconds: 10, bufferBehindSeconds: 0.5 } }.
VideoSource.id is removed from UrlVideoSource, BlobVideoSource and StreamVideoSource.	TypeScript stops the build if you set it.	Drop the property from source literals. Nothing in the engine ever read it. These three types are new to supervision, and reach consumers only at supervision/web-video-engine, so no released consumer can be holding it.

playbackGate is not on this list, and that is deliberate. The render-preparation gate already
defaulted to enabled on main, and the detection gate already defaulted on for appendable sessions.
Both are unchanged. What is new is the playbackGate boolean itself: an off switch, and a way to
turn the detection half on for a session that is not appendable. Nothing an existing consumer does
starts behaving differently.

Two more are changes in output rather than removals.

• Detection frame selection now tolerates 0.5 ms of playhead quantization. main compares the
  playhead against the frame's media time exactly. On a source whose frame timestamps are not whole
  milliseconds, a playhead that rounds down selected the previous detection frame. Sources on
  exact-millisecond timestamps are unchanged.
• In NearestFrameIndex mode the grid step is measured from the buffered frames' own media
  times. frameRate is the fallback when the buffered indexes cannot give a step. With no indexed
  frame at all the mode does not apply, and selection matches by interval instead. A caller whose
  rate matched the clip sees no change. A caller who passed a nominal rate the clip does not run at
  was previously walked off the grid by the accumulating difference.

MediaRendererState gains five optional fields, so an existing renderer still satisfies the type.

Field	Reports
drawnMaskFrameTime	The frame the visible mask belongs to.
maskHeldStale	That frame is not the one the active detections describe.
playbackGateReach	How far the gate reaches on this source: Off, EveryFrame or StartOfPlayback.
seeking	A seek is still in flight, where playbackState cannot say so.
scrubbing	A drag is open on the playhead, so the viewer leads the picture rather than waits for it.

seeking answers for the transport. The transport settles one message before the landed frame
reaches the main thread. A host that needs "is the right picture up" must compare the presented
frame's own media time instead. A scrub sets seeking on every tick, so a host that draws a wait
indicator must read scrubbing first.

Deprecated

Deprecated	Still works?	Removal
MediaRendererOptions.muted	Yes. Nothing ever read it.	Delete at your convenience.
MediaSessionRendererOptions.muted	Yes. Nothing ever read it.	Delete at your convenience.
DetectionFrameSelectionOptions.frameIndexOriginTime	Yes. Selection does not read it.	Delete at your convenience. Each buffered frame carries the media time its index sits at.

Neither muted option was ever read, so nothing sounded different before or after. Audio playback
is deferred.

The main-thread cost

Every annotation is drawn on the page's own thread. The engine decodes off it. The picture and the
boxes, masks, labels, polygons, keypoints and focus over it are composited by Pixi on the main
thread, in one synchronous block per presented frame.

Measured on the reference machine and clip, playing from t=5s, three runs of a 6.0-second window
holding 180 presented frames:

Per presented frame	Annotations off	Annotations on
presentVideoFrame, entry to return	1.05 to 1.10 ms	1.08 to 1.14 ms
Main thread busy, all causes	7.23 to 7.27 ms	7.23 to 7.50 ms
Main thread occupancy	21.7 to 21.8%	21.7 to 22.5%

The frame period is 33 ms. Annotations cost 0.03 to 0.07 ms of the block.

What you see when the budget runs out is the picture falling behind. You never see annotations from
the wrong moment: the frame and every layer over it are drawn from one media time, in one block
nothing can interrupt.

A host application shares this thread with its own work. The direction is to make the block smaller
rather than move it to a worker, and the ceiling on what moving it would buy is known: the block is
1.08 to 1.14 ms of the 7.23 to 7.50 ms the thread is busy, so the rest of the thread bounds the win.
docs/internal/video-engine-presentation.md documents the mechanism. The figures above come from
a CDP profiling run over the demo, which is not committed.

Tradeoffs

We chose	It costs
Walk every packet on open to build an exact frame table.	Load is slower on a long file, and there is a hard ceiling of one million frames. A 70-second 30fps source walks 2113 packets in a measured 5.7 ms.
Composite every annotation on the page's own thread.	The block runs 1.08 to 1.14 ms per presented frame on an M3 Max. It gets expensive on a slower machine, at a higher frame rate, or on denser detections.
Cap the presentation surface at 2x device pixel ratio by default.	A display above 2x draws slightly softer. It is what puts the surface, the mask rasters and the decode on one pixel grid.
Ship the engine inside supervision, on its own import path.	Every consumer downloads the engine. The published tarball grows from 654,101 to 1,732,755 bytes. A dynamic import() keeps it out of the bundle, so an app that only creates a media session emits no engine asset.
Refuse a file we cannot index exactly, rather than guess.	Some files that would play in a <video> element are refused here. WebVideoEngineErrorCode names which limit was hit.
Commit the fixture media and its raw model output.	demo/fixtures is 306 MB tracked over 107 files. Every clone and every CI run pays it.

Known limitations

• Firefox cannot open an HEVC clip. Firefox 154 plays an HEVC file in its own <video>
  element, but its VideoDecoder reports both hvc1 and hev1 configurations unsupported. The
  engine refuses the file at load with DecodeUnsupported, before any frame is presented. There is
  no software fallback in the package. This is that decoder, not a rule about non-Chromium browsers:
  Safari 18.6 reports both supported and plays the same file. The demo's default fixture, the 70s
  horse trail, is HEVC Main 10, so Firefox errors on it. The 9-second basketball fixture is H.264
  and plays.
• Without a usable WebGPU device, every presented frame is copied through a staging canvas.
  This is the generic fallback, not a browser-specific path. Safari 18.6 reaches it because it has
  no WebGPU at all. Firefox reaches it for a different reason. The cost is characterised in
  docs/internal/video-engine-presentation.md: the upload runs once per presented frame and
  dominates the wall clock during playback, where a straight VideoFrame upload costs a small
  fraction of that. It carries no number.
• Masks are built at the detections' own resolution unless a host declares a display box.
  Passing renderPreparation.maskFrame.display is what makes the raster follow what the screen can
  show. The demo passes one. The presentation numbers above are optimistic for an integration that
  has not opted in.
• A frame whose mask is still being prepared may keep the previous frame's mask, for exactly one
  frame. The bound is one frame of the buffered detection timeline, whatever media distance
  separates them, and holds cannot accumulate. Nothing scales it: a budget in seconds multiplied by
  playback rate once left a mask four tenths of a second stale after an 8x run, and there is no rate
  on this path. Boxes, labels and the other layers draw regardless, because readiness is tracked per
  layer rather than per session. drawnMaskFrameTime and maskHeldStale report a hold while it is
  happening. Measured on a slow backward scrub, this takes frames drawn with no mask from 14% to 4%
  and the flicker from ten a second to under three; at 4x it changes nothing, which is what one frame
  is worth at that speed.
• The StreamVideoSource variant is declared but no test or demo exercises it. A stream cannot
  be re-opened, so the decoder-recovery path degrades instead of rebuilding on one.
• Reverse playback is refused rather than clamped. Rates outside 0.25x to 8x throw.

Three defects that ship on main today

All three are fixed here, and none of the fixes is on main. The code each one lives in was there
first: region effects and their fixture landed on main before this branch, the prepared-window
timeline has been there since the shape-primitives work, and the interaction layer has followed a
selected detection across frames since before this branch opened.

The region-effects lens jumped off a player's head, frame after frame. Some lenses floated over
the crowd with nobody under them. A head the model did not see was moved by however far the player's
whole bounding box moved, and that box is set by whichever limb reaches furthest, usually a raised
arm. An invented head now sits between the two real observations on either side of it.

Invented heads	Median error	Badly placed
Before	7.2 px	15.7%
After	2.8 px	2.8%
Real heads, for scale	2.8 px	3.5%

A detection selected while scrubbing vanished for good the first time its annotations were late.
Scrubbing backward is where they are most often late, so the selection usually died within a frame or
two of the first drag, and picking the detection again was the only way back. An absent frame and a
detection that had genuinely left the video both rebased to nothing, and the caller wrote that empty
result over the selection. The follow step now leaves a selection alone while data is missing and
adjudicates on the next frame that has any.

On a looping clip the prepared render window ranked a frame from the previous lap as the furthest
thing prepared. Seventy seconds of footage reported 66.86 seconds of readiness for 211 frames
covering seven. That number is not a readout: it is compared against the lookahead a session asks
for before playback is considered ready, so a wrong value can hold or release the gate for the wrong
reason.

Reading the diff

The pull path is unchanged. The push path, the transport, the frame-present walk and the
prepared-annotation window are new files, reached only through a presented-frame channel. Today only
the video engine drives that channel. The pull path keeps its three ticker callbacks and its draw
order. That is the split worth holding in mind while reading the renderer diff.

Almost every deletion is fixture data. 2,191,256 of 2,197,601 deleted lines sit under
demo/fixtures, because the detection payloads are no longer pretty-printed. Outside those
fixtures the diff is 425 files, 88,464 insertions against 6,345 deletions. That is the code to
review.

The fixture data itself differs from main. The SAM3 fixtures are generated against the source
videos at their native frame rate rather than a resampled proxy. The clearest case is the basketball
sample. On main its manifest reads 270 frames at 30fps against basketball_sample.normalized.webm.
Here it reads 225 frames at 25fps against basketball_sample.mp4, the clip's own rate. Loading every
fixture in the demo picker covers this better than reading the diff does.

What the fixtures cost a clone. demo/fixtures is 306 MB tracked over 107 files, in a
repository whose .git is 772 MB.

Fixture	Tracked	Largest single file
horse_trail	231 MB	1min-horse-video.mov, 128 MB, the media the demo plays
basketball_sam3	28 MB	raw-sam3.jsonl, 11 MB
basketball_sample	28 MB	basketball_sample.mp4, 22 MB
basketball_regions	19 MB	head-detections.json, 9 MB

horse_trail/raw-sam3.jsonl is 44 MB of raw model output kept for provenance beside the 59 MB of
chunked detections derived from it. Nothing loads it at runtime. It is worth deciding deliberately,
since it is what every reviewer and every CI run pays to clone.

Packaging and release

The engine does not publish on its own. packages/video-engine is a private workspace, and its
browser build is staged into supervision under dist/web-video-engine. Consumers reach it by
import path:

import { createWebVideoEngineMediaRendererSource } from "supervision/web-video-engine";

The subpaths are supervision/web-video-engine, supervision/web-video-engine/analysis and
supervision/web-video-engine/worker. createWebVideoEngineMediaRendererSource and
openWebVideoEngineMediaSource are exported from the package root as well, and are the same function
in both places.

There is no second install and no optional peer dependency. Installing supervision installs the
engine, because the staged build is inside the tarball. The tarball grows from 654,101 to 1,732,755
bytes, and every consumer pays that download even if it never imports the engine. The bundle cost
stays conditional. The engine is reached by a dynamic import(), so an app that imports only
createMediaSession emits 1,750,666 bytes and no engine asset, while adding the engine adapter
emits 3,278,684 bytes with the engine in its own 1,503,131-byte chunk. Still images and camera input
never load it. Opening a video file does. If that chunk does not load, the video path throws an
error naming supervision/web-video-engine and saying the engine is a lazily loaded chunk of
supervision, rather than a bundler stack trace naming a hashed asset.

The release workflow publishes one package. It builds the video-engine workspace, stages that build
into dist/web-video-engine, and deletes the engine's file: devDependency from the packed
manifest. It then builds the portable tarball, smoke-tests it in a clean consumer, and publishes
supervision. A released supervision therefore names no engine package and no engine version.
After the upload the workflow polls npm view supervision@<dist_tag> up to twelve times at
five-second intervals, until the dist-tag resolves to the version it just published. The workflow
publishes from main, or from a release/* branch when dist_tag is next.

No release step needs a person. supervision is already on npm, so its trusted publisher is
already attached. The workflow publishes the generated tarball with npm publish and authenticates
through OIDC. It needs no npm login and no NPM_TOKEN. The engine is private and is never
published, so there is no second name to register.

Two things that will not warn you

A custom workerFactory must match the host's version. The mask preparation protocol changed.
The artifact kind is idMask rather than pngIdMask, the payload field is raster rather than
png, and the job carries a maxRasterWidth. None of those types is exported, so nothing warns.
Point the factory at supervision/render-preparation-worker and this cannot happen.

Content Security Policy is unaffected. This package already spawns classic blob workers for mask
preparation and for tracking. The engine's worker needs the same directive and no new one.

Documentation status

docs/public is the published documentation and it is checked against the code. npm run docs:check
runs 31 checks: every path a document names exists, every npm script it runs is declared, every flag
matches the script that reads it, every checksum matches the file beside it, every version matches
the manifest, every symbol it imports is exported, and every copyable integration example
typechecks. All 31 pass.

Eighteen files under docs/public change here:

Page	Covers
guides/browser-support.md	New. The four limits an integration has to plan around.
api/video-engine.ts	New. The engine subpath's own surface, pinned.
guides/media-sessions.md, guides/detections-and-rendering.md, guides/media-preparation.md, recipes/streaming-detections.md, recipes/multiple-detection-sources.md	Which distance each gate reaches on which source.
guides/application-integration.md	The single install, the engine's import path, and the download that carries the engine either way.
guides/public-api.md, concepts.md, annotation-renderers/polylines.md	The push path, the presented frame, and the polyline trail.
api/media-preparation.ts, api/rendering.ts, api/sessions.ts	The 20 added exported names, pinned.
guides/presentation-styles.md, recipes/interactive-picking.md, recipes/progressive-upload-normalization.md	Engine references renamed, and the picking and upload recipes kept in step.
typedoc-icons.js	The generated icon set the API pages render with.

One gap I know about and have not closed:

• guides/browser-support.md says Firefox refuses every HEVC profile. What was measured is that
  Firefox 154 reports hvc1 and hev1 configurations unsupported. Every HEVC codec string starts
  with one of those two, so the claim follows, but it is inferred rather than tested profile by
  profile.

What this pull request does not have

• No screenshots and no recording. This is a visual change and it should have one. The demo is
  the artifact worth recording: load a fixture, scrub backwards with masks on, and watch every
  annotation stay on its frame.
• The evaluation harness leaves no artifact in the repository. It was run on this commit,
  clean tree, engine path pinned, on the 70s horse-trail clip: eleven scenarios pass and
  battery skips itself, because it needs a Storybook this repository does not ship. Detections
  land on their frame with a worst offset of 0 ms, a backward scrub keeps every mask
  (maskInkRatio 1.000) and settles at 91 ms p95, no presented frame is dropped, and there
  are no skipped intervals or engine stalls. Seek p95 is 3.7 ms against a 250 ms limit and step
  p95 is 53.5 ms against 80 ms. None of that is in the diff: .gitignore excludes
  tools/demo-eval/report.json and tools/demo-eval/baseline.json, because those are one
  machine's numbers and only mean anything on that machine. Reproduce with
  npm run eval:demo -- --url 'http://localhost:5173/?mediaPath=engine'.
• No second machine. Every performance number here is one M3 Max in Chrome. Nothing is measured
  on a slower machine, and the main-thread block is the number most likely to move on one.
• No comparison against the alternative. The cost of moving presentation to a worker is priced
  on one side only. The other side needs a harness story that lives in the engine repository.