fix: close the gaps the widened mask union and open producer opened

Widening `DetectionMask` and opening the live producer contract both moved a
boundary, and each left a consumer on the far side that had never been asked to
handle what now crosses it.

**A dense mask crashed the editable annotation session.** Routing every reader
through `decodeDetectionMask()` was necessary and not sufficient:
`createEditableAnnotationFrameSession()` never decodes, it deep freezes, and
`Object.freeze()` throws on an array buffer view with elements. Every mask had
been a `counts` string, so nothing in core had met a typed array nested in a
`Detection`. The freeze now steps over views; the buffer is documented immutable
and shared, so there was nothing there to protect. Rebasing onto main surfaced
the same class of bug from the other direction — #90's `regionCoverageMask` is
typed `DetectionMask` and assigned straight from `detection.mask`, so it decodes
through the union now too.

**The producer contract accepted more than it drew.** Only dense masks reach the
fill loops, which is the right call — decoding RLE per detection per frame is
the cost the dense encoding exists to avoid — but a producer publishing cold
storage got a blank overlay and no signal. Skipped masks are now counted and
reported through the readout. Keypoint edge indices are bounds-checked rather
than asserted, since the skeleton is no longer a constant this package owns, and
keypoint color resolves per detection from `className` the way boxes already
did; one frame-wide color was an artifact of pose having had exactly one class.
`createExecutorchPoseKeypointInstructions` returns as a deprecated forwarding
alias, so the rename breaks no import.

**The media clock paid twice for held frames.** It re-reported identical
detections to React on every presented frame, turning a ~1.4Hz cross-thread hop
under `analysis` into ~30Hz under `media`, and rebuilt the ID-mask artifact from
scratch each time — ~15ms per frame with the JS builder on a Pixel 10 Pro.
Those milliseconds come out of the wait time the pacing budget banks, so
refilling an identical artifact was lowering the inference cadence it existed to
protect. `skia.ts` now splits the fill from the upload; the pump caches the
build keyed on the detections array identity held frames already share.
Uniforms and the `SkImage` stay per packet, so `PreparedFrameStore` ownership is
unchanged and a reused artifact reports `fillMs: 0` rather than a cost it did
not pay. Under `analysis`, `shouldInfer` is always true, so that lane is
bit-identical.

Also corrects the record on the spin loop. The pump runtime does have
`setTimeout` — `createWorkletRuntimeForThread()` delegates to worklets'
`createWorkletRuntime()`, whose `enableEventLoop` defaults to true. What it
cannot do is fire one from inside `runPump`, a synchronous loop that never
returns to the run loop. That makes the spin a structural choice with a known
exit at Phase 3, not a missing primitive to go hunting for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: lucas-fochesatto

docs/internal/react-native-architecture.md

@@ -157,8 +157,16 @@ serialization/preparation worklets, and shared file/live defaults.
 controls, state subscription, and capability surface while its strict-sync
 native-pointer pump and Skia presentation lanes remain package-private.
 `createReactNativeVideoSession()` is retained only as a deprecated forwarding
-alias. File sessions are analysis-paced, support pause/play/stop, and report
-seeking as unsupported.
+alias. File sessions support pause/play/stop and report seeking as unsupported.
+
+Pacing is a session option, not a fixed property. `clock: "analysis"` remains
+the default and infers on every decoded frame, so playback runs at inference
+speed — correct for producing a fully annotated video. `clock: "media"`
+presents frames on their own timeline, so a ten second clip takes ten seconds,
+inferring on whatever subset the measured model cost affords and holding the
+most recent detections across the frames in between. See
+[`react-native-unified-session-plan.md`](react-native-unified-session-plan.md)
+for the pacing policy and what remains.
 
 Android saved-video decoding is implemented (experimental) as a Nitro C++
 `VideoFrameSource` under `packages/react-native/android/`: `AMediaExtractor`

docs/internal/react-native-live-rendering.md

@@ -208,6 +208,10 @@ Same-day iPhone 15 comparison (debug build, same code, same Basketball
 sample, 6 masks, CoreML INT8, native mask prep): SEG 72ms, FILL 1ms, TICK
 88ms (~7.8 fps) — consistent with the pre-extraction 46-58ms baseline plus
 dev-mode overhead. The cross-platform gap is entirely the detection
-producer's backend (ANE INT8 vs CPU fp32); both platforms are strict-sync
-analysis-paced by design, so file playback intentionally runs at inference
-speed rather than media speed.
+producer's backend (ANE INT8 vs CPU fp32).
+
+Both figures above were measured under the analysis clock, where playback runs
+at inference speed by design. Saved-video sessions now also accept
+`clock: "media"`, which holds the clip to its real duration and infers on the
+subset that fits; the presented-frame ratio under that clock is the number to
+record here next, per device.

docs/internal/react-native-unified-session-plan.md

@@ -104,6 +104,17 @@ or `SyncMediaRendererAdapter` directly.
 processor and renderer are both sync is proven to run its whole frame path
 with no microtask in it, and the async default is proven to still yield.
 
+**Widening a union means auditing its consumers, not just its decoders.**
+Routing every reader through `decodeDetectionMask()` was necessary and not
+sufficient: `createEditableAnnotationFrameSession()` never decodes, it deep
+freezes, and `Object.freeze()` throws on an array buffer view with elements. A
+dense mask crashed it. Every mask had been a `counts` string, so nothing in
+core had ever met a typed array nested in a `Detection`. The freeze now steps
+over views, and the same question — what does this do with bytes rather than a
+string? — is worth asking of any new `DetectionMask` consumer. Main added one
+during this branch's life (`regionCoverageMask`, in the web mask compositor)
+and it needed the same fix.
+
 ## Phase 1 — Open the producer contract (live camera first)
 
 Today `UseReactNativeLiveInferenceOptions` requires
@@ -129,6 +140,18 @@ containing "Executorch" appears in the live hook's public options.
 published geometry, so `ReactNativeLiveInferenceMode` is gone. Keypoint drawing
 moved out of the adapter as `createReactNativeKeypointDrawInstructions`, which
 was the last vendor name the hook imported.
+`createExecutorchPoseKeypointInstructions` stays as a deprecated forwarding
+alias, so the rename does not break an import.
+
+**What an open contract had to admit.** Opening the door meant being honest
+about what is behind it. Only `DenseBitmapDetectionMask` reaches the fill
+loops, so an RLE mask is skipped rather than decoded per frame — but that is now
+counted and reported through the readout's `skippedRleMaskCount`, because a
+blank overlay with no signal is not an open contract. Keypoint edge indices are
+bounds-checked instead of asserted, since the skeleton is no longer a constant
+this package owns. And keypoint color resolves per detection from `className`,
+the same way boxes already did; a single frame-wide color was an artifact of
+pose having had exactly one class.
 
 Two notes for whoever picks this up. Detections still cross into the ID-mask
 fill through `serializeReactNativeLiveDetectionFrame`, a shallow bridge to the
@@ -171,9 +194,29 @@ until the schedule recovers, and another inference becomes affordable. The
 policy lives in `sessions/media-clock-policy.ts` as pure functions, because
 nothing inside the pump worklet is reachable from a test.
 
-Two limits carried forward. The wait spins — the pump runtime has no sleep
-primitive — which costs less than `analysis` (never waits, never stops
-inferring) but is worse than pacing presentation off vsync. And held
+**Two costs the first cut paid, since removed.** Held frames re-reported
+identical detections to React on every presented frame, turning a ~1.4 Hz
+cross-thread hop under `analysis` into a ~30 Hz one under `media`; reporting is
+now gated on `shouldInfer`, which is the only rate at which the payload
+changes. And every held frame rebuilt the ID-mask artifact from scratch — ~15 ms
+per frame with the JS builder on a Pixel 10 Pro, taken directly out of the wait
+time the budget banks, so refilling an identical artifact was lowering the
+inference cadence it was supposed to protect. `skia.ts` now splits into
+`buildReactNativeSkiaMaskArtifact()` and
+`createReactNativeSkiaMaskFrameFromArtifact()`; the pump caches the build keyed
+on the detections array identity that held frames already share. Uniforms and
+the `SkImage` are still per packet, so `PreparedFrameStore` ownership is
+unchanged, and a reused artifact reports `fillMs: 0` rather than repeating a
+cost it did not pay.
+
+**Two limits carried forward.** The wait spins. That is a structural choice,
+not a missing primitive: `createWorkletRuntimeForThread()` delegates to
+worklets' `createWorkletRuntime()`, whose `enableEventLoop` defaults to true
+and installs `setTimeout` on the runtime. What cannot happen is firing one from
+inside `runPump`, a single synchronous loop that never returns to the run loop.
+Removing the spin means restructuring the pump into a per-frame continuation,
+which also moves pause, resume, teardown, and source close off the guarantee
+that loop exit currently provides — so it belongs with Phase 3. And held
 detections lag their frame; propagating with a tracker is the next step, worth
 judging against device numbers rather than in the abstract.
 

packages/core/src/detections/editable-annotation-frame-session.test.ts

@@ -4,6 +4,7 @@ import {
   AnnotationFrameMutationKind,
   createEditableAnnotationFrameSession,
 } from "#detections/editable-annotation-frame-session";
+import { DetectionMaskEncoding } from "#types/detections";
 
 describe("editable annotation frame session", () => {
   const initialFrame = {
@@ -128,4 +129,36 @@ describe("editable annotation frame session", () => {
       "preserve the stable id",
     );
   });
+
+  it("snapshots a dense bitmap mask without freezing its bytes", () => {
+    // `Object.freeze()` throws on an array buffer view with elements, so the
+    // snapshot's deep freeze has to step over mask bytes. Every mask used to
+    // arrive RLE-encoded, where the payload is a string, which is why this
+    // only became reachable once the encoding widened.
+    const data = new Uint8Array([0, 255, 0, 255]);
+    const session = createEditableAnnotationFrameSession({
+      detections: [
+        {
+          id: "dense",
+          mask: {
+            data,
+            encoding: DetectionMaskEncoding.DenseBitmap,
+            height: 2,
+            width: 2,
+          },
+        },
+      ],
+      mediaTime: 0,
+    });
+
+    const snapshot = session.getSnapshot();
+    const mask = snapshot.detections[0]?.mask;
+
+    expect(Object.isFrozen(snapshot)).toBe(true);
+    expect(Object.isFrozen(mask)).toBe(true);
+    // Shared, not deep-copied: duplicating a full-resolution mask per frame is
+    // exactly the cost this encoding exists to avoid.
+    expect(mask && "data" in mask ? mask.data : null).toBe(data);
+    expect(Object.isFrozen(data)).toBe(false);
+  });
 });

packages/core/src/detections/editable-annotation-frame-session.ts

@@ -229,7 +229,16 @@ function createSnapshot(frame: DetectionFrame): DetectionFrame {
 }
 
 function deepFreeze<T>(value: T): T {
-  if (value && typeof value === "object" && !Object.isFrozen(value)) {
+  // Typed arrays are skipped, not frozen: `Object.freeze()` throws on any
+  // array buffer view that has elements. A dense mask nests its bytes in one,
+  // and `DenseBitmapDetectionMask` already documents `data` as immutable and
+  // shared rather than copied, so there is nothing here left to protect.
+  if (
+    value &&
+    typeof value === "object" &&
+    !ArrayBuffer.isView(value) &&
+    !Object.isFrozen(value)
+  ) {
     Object.freeze(value);
 
     for (const nested of Object.values(value)) {

packages/react-native/src/adapters/executorch.ts

@@ -6,6 +6,7 @@ import {
   type DetectionFrame,
 } from "supervision-js-core";
 import type { ReactNativeLiveSerializedDetection } from "../index";
+import { createReactNativeKeypointDrawInstructions } from "../renderers/keypoint-draw-instructions";
 import type { ReactNativeLiveDetectionProducer } from "../types/live-producer";
 import type { ReactNativeVideoFrameHandle } from "../video-frame-source";
 
@@ -704,3 +705,16 @@ export function createDetectionFrameFromExecutorchCocoPoses(
     mediaTime: options.mediaTime ?? 0,
   };
 }
+
+/**
+ * @deprecated Renamed to `createReactNativeKeypointDrawInstructions` and moved
+ * to `renderers/`. Keypoint drawing reads only core types, so nothing about it
+ * was ever specific to ExecuTorch; the vendor name here was an artifact of
+ * pose support having been written in this adapter first.
+ *
+ * Kept as a forwarding alias so the rename does not break an import. The
+ * replacement additionally resolves color per detection instead of taking a
+ * single color for the whole frame.
+ */
+export const createExecutorchPoseKeypointInstructions =
+  createReactNativeKeypointDrawInstructions;

packages/react-native/src/react/use-live-inference.ts

@@ -1,7 +1,6 @@
 import { useCallback, useEffect, useMemo, useRef } from "react";
 import {
   KeypointMarkerShape,
-  resolveDetectionClassColorStyle,
   type DetectionFrame,
   type KeypointDrawInstruction,
   type PolygonDrawInstruction,
@@ -69,6 +68,12 @@ export interface ReactNativeLiveInferenceReadout {
   readonly segmentationMs: number;
   readonly serializationMs: number;
   readonly shaderActive: boolean;
+  /**
+   * Detections whose mask the producer published RLE-encoded. Only dense masks
+   * reach the fill loops, so a non-zero value here means masks were produced
+   * but nothing was drawn for them.
+   */
+  readonly skippedRleMaskCount: number;
   readonly syncMode: "synced";
   readonly timestamp: number;
   readonly visibleKeypointCount: number;
@@ -686,6 +691,7 @@ function reportLiveInference(
     segmentationMs: metrics.segmentationMs.value,
     serializationMs: metrics.serializationMs.value,
     shaderActive: metrics.shaderActive.value,
+    skippedRleMaskCount: metrics.skippedRleMaskCount.value,
     syncMode: "synced",
     timestamp: frame.timestamp,
     visibleKeypointCount: metrics.visibleKeypointCount.value,
@@ -753,8 +759,22 @@ export function useReactNativeLiveInference(
   // model for a moment.
   //
   // Each closure captures its own generation and compares it against the
-  // shared value, which React updates immediately. A superseded closure sees
-  // the mismatch and skips the frame instead of running a stale model.
+  // shared value, which the effect below writes on commit — ahead of the
+  // camera's own effect reinstalling `onFrame`, which is the ordering that
+  // matters. A superseded closure sees the mismatch and skips the frame
+  // instead of running a stale model.
+  //
+  // The obvious simplification — put `producer` itself in a shared value and
+  // compare identity — is deliberately avoided. A shared value serializes what
+  // it holds, and a producer closes over a JSI HostFunction; that is exactly
+  // the recursive-capture case `react-native-live-rendering.md` records as
+  // serializing successfully and then being non-callable on the worklet
+  // runtime. An integer carries no such risk.
+  //
+  // Incrementing inside `useMemo` is a render-phase write, which a StrictMode
+  // double render inflates. That is harmless here because only equality is
+  // ever tested, and a recomputed memo rebuilds `onFrame` and republishes the
+  // shared value together.
   const producerGenerationRef = useRef(0);
   const producerGeneration = useMemo(() => {
     producerGenerationRef.current += 1;
@@ -767,7 +787,6 @@ export function useReactNativeLiveInference(
     activeProducerGeneration.value = producerGeneration;
   }, [activeProducerGeneration, producerGeneration]);
 
-  const poseInstructionColor = resolveDetectionClassColorStyle("person").fill;
   // Capture initialized worklet functions before the callback is serialized.
   // Worklets' Babel transform does not preserve normal function hoisting.
   const createPoseInstructions = createReactNativeKeypointDrawInstructions;
@@ -847,10 +866,9 @@ export function useReactNativeLiveInference(
             );
           }
           stage = "pose-instruction-conversion";
-          const poseInstructions = createPoseInstructions(
-            detectionFrame,
-            poseInstructionColor,
-          );
+          // No color argument: each skeleton takes its own class color, so a
+          // producer publishing more than one class does not draw them alike.
+          const poseInstructions = createPoseInstructions(detectionFrame);
           stage = "pose-prepare-vector";
           const vector = presentation.prepareVector({
             frameHeight: frameSize.height,
@@ -882,7 +900,11 @@ export function useReactNativeLiveInference(
         }
 
         stage = "detection-serialization";
-        const detections = serializeDetections(detectionFrame);
+        const serialized = serializeDetections(detectionFrame);
+        const detections = serialized.detections;
+
+        metrics.skippedRleMaskCount.value = serialized.skippedRleMaskCount;
+
         const segmentationMs = inferenceMs;
         const extensionResult = activeExtension.active
           ? evaluateLiveInferenceObjectExtension({
@@ -970,7 +992,6 @@ export function useReactNativeLiveInference(
       mediaRect,
       metrics,
       mosaicCellPx,
-      poseInstructionColor,
       presentation,
       activeProducerGeneration,
       privacyContourWidth,
@@ -1025,6 +1046,7 @@ function useLiveInferenceMetrics() {
   const segmentationMs = useReactNativeSharedValue(0);
   const serializationMs = useReactNativeSharedValue(0);
   const shaderActive = useReactNativeSharedValue(false);
+  const skippedRleMaskCount = useReactNativeSharedValue(0);
   const visibleKeypointCount = useReactNativeSharedValue(0);
 
   return useMemo(
@@ -1045,6 +1067,7 @@ function useLiveInferenceMetrics() {
       segmentationMs,
       serializationMs,
       shaderActive,
+      skippedRleMaskCount,
       visibleKeypointCount,
     }),
     [
@@ -1064,6 +1087,7 @@ function useLiveInferenceMetrics() {
       segmentationMs,
       serializationMs,
       shaderActive,
+      skippedRleMaskCount,
       visibleKeypointCount,
     ],
   );