Skip to content

README.md

Latest commit

 

History

History
150 lines (129 loc) · 8.61 KB

README.md

File metadata and controls

150 lines (129 loc) · 8.61 KB
Raw

Video Sources POC

End-to-end proof of concept for video sources as a platform primitive. This file is the runbook; for the full context — what we're proving, architecture, flows, data model, known gaps — read HANDOFF.md first. For the next phase — relay capacity, multi-cell placement, dedicated cells, remote execution, and workload-aware GPU admission — read MULTI_CELL_SCALING_RFC.md.

connector (Go, laptop/LAN)  --RTSP push-->  mediamtx  --RTSP-->  processor (Python, warm worker)
        |                                       |                        |
        |  outbound HTTP poll (commands+ack)    |  WHEP (browser         |  MJPEG + SSE events
        v                                       |  live preview)         v  (browser, via job.processorUrl)
   Roboflow app  (Video Sources page; functions: token.js + deviceApi.js; Firestore)

Components in this folder (branch hansent/video-poc):

  • connector — Go agent, now in its own repo: roboflow/rf-video-connector (single binary + bundled ffmpeg, released via goreleaser; install script in that repo). Discovers USB cameras (avfoundation/v4l2), RTSP URLs (flags), and video files (a folder); polls the platform for commands; pushes streams via ffmpeg. Stdlib only.
  • processor/processor.py — warm worker. Polls the platform for jobs, runs InferencePipeline.init_with_workflow on the assigned source in one of two modes (batch = every frame as fast as possible; stream = real-time with drops, files replayed via ffmpeg -re through mediamtx), splits outputs into an SSE events stream (images redacted to refs) and per-output MJPEG previews (/preview.mjpeg?output=, available outputs advertised in /status). Managed-pool claims carry a platform-minted processorAccessToken; job endpoints require it as a Bearer token (or access_token query parameter for native browser media elements). Bare /status and /metrics expose only aggregate health/metrics without auth.
  • mediamtx.yml, fetch-deps.sh, bin/ — local media plane (RTSP ingest :8554, WHEP :8889).
  • benchmarks/ — relay fan-out load harness, provisional workflow corpus, standalone processor job generator, and aggregate-metric runbook. Active load is for local, staging, or a dedicated performance cell—not production.

The platform half lives in the roboflow repo (branch hansent/video-sources-poc): Video Sources page + /query/video-sources* routes (token.js) + connector/processor endpoints (deviceApi.js) + Firestore collections video_sources, video_connectors, video_jobs.

Staging and the first feature-flagged production cell (crusoe-use1) are live as of 2026-08-10. This runbook remains intentionally local; current cluster deployment mechanics live in the roboflow-infra helm/roboflow-video-proc/ chart and crusoe/video-proc Terraform stack.

Running the demo (everything local)

Terminal 0 — deps (once):

cd development/video_poc && ./fetch-deps.sh
cd ../.. && uv venv development/video_poc/.venv --python 3.11
uv pip install --python development/video_poc/.venv/bin/python -e . onnxruntime requests
# connector lives in roboflow/rf-video-connector now — install a release:
gh api repos/roboflow/rf-video-connector/contents/install.sh -H "Accept: application/vnd.github.raw" | sh

Terminal 1 — the app (roboflow worktree). Environment quirks discovered on 2026-07-06: node 24 required (n exec 24.11.0), firebase-tools 15 comes from the repo-root node_modules, the API-key auth path needs a local Redis on :6379 (brew install redis; avoids the docker compose stack, which also wants nginx), and build:dev:once skips docker entirely.

redis-server --port 6379 &
cd ~/code/wt/roboflow-video-poc/app
n exec 24.11.0 npm run build:dev:once                       # one-shot webpack build
source ../secrets.sh staging gac && \
  PATH="/usr/local/n/versions/node/24.11.0/bin:$PATH" NODE_OPTIONS=--preserve-symlinks \
  ../node_modules/.bin/firebase emulators:start --only hosting,functions
# app: http://localhost:5000  ·  functions: http://localhost:5001

Terminal 2 — media plane:

cd development/video_poc && ./bin/mediamtx mediamtx.yml

Terminal 3 — warm processor. Two env vars point model downloads at staging (matching the staging API key): PROJECT=roboflow-staging for the classic inference paths, and ROBOFLOW_ENVIRONMENT=staging for the new inference_models weights provider (it has its own config and defaults to prod — without it model loads fail with UnauthorizedModelAccessError):

cd development/video_poc/processor && \
  PROJECT=roboflow-staging ROBOFLOW_ENVIRONMENT=staging ../.venv/bin/python processor.py \
  --api-url http://localhost:5001/roboflow-staging/us-central1/light-v2-device \
  --api-key $ROBOFLOW_API_KEY

Terminal 4 — connector (the thing a customer runs):

rfv-connector \
  --api-url http://localhost:5001/roboflow-staging/us-central1/light-v2-device \
  --api-key $ROBOFLOW_API_KEY \
  --files-dir ./videos          # plus e.g. --rtsp cam1=rtsp://... ; USB cams auto-discovered

The connector serves a local status/config UI at http://127.0.0.1:8070 (--ui-addr to change, empty to disable): platform connection status, per-source enable/disable toggles, add-RTSP-camera form, and API-key entry — the key can be omitted from the CLI entirely and set in the UI instead. Runtime changes persist to connector.json (--config).

Then open the app → Video Sources in the sidebar:

  1. Connector's cameras/files appear as sources within ~2s of it starting.
  2. Click a source → live preview (WHEP) or file playback.
  3. "Start processing with workflow" → pick a workflow → annotated MJPEG + JSON event stream.
  4. Stop processing → the platform reconciles and tells the connector to stop pushing.

Standalone processor test (no platform)

cd development/video_poc
./bin/mediamtx mediamtx.yml &
./bin/ffmpeg -re -f lavfi -i "testsrc2=size=1280x720:rate=30" -c:v libx264 -preset veryfast \
  -tune zerolatency -pix_fmt yuv420p -g 30 -f rtsp rtsp://127.0.0.1:8554/test &
cd processor && ../.venv/bin/python processor.py --job-file test-job-blur.json
# http://127.0.0.1:8890/status | /events | /preview.mjpeg

Measured on M-series laptop (720p30, blur workflow, CPU): 30 fps sustained, ~40 ms decode→result latency, ~12 s job→first-result (engine init dominates — a warm processor amortizes imports; model load is the remaining per-job cost).

Design notes

  • Control channel: outbound-only polling with commands in the healthcheck response and ack-by-id — the same pattern as device-healthcheck-v2 / device-manager.
  • Stream lifecycle: the platform reconciles desired state (active jobs + preview TTLs) against connector-reported streams on every healthcheck — start/stop commands are emitted from the diff. Registered ≠ streaming: video flows only while something needs it.
  • Events vs pixels: the processor redacts image outputs from the JSON events (SSE) and serves any image output as MJPEG on demand (?output=); /status advertises what exists so viewers can attach to a job after it started.
  • Benchmark telemetry: an authorized job /status response and the platform heartbeat retain the original stats.frames, stats.fps, and startup fields, and add a versioned, fixed-size stats.counters/stats.decodeToResultLatency report. It distinguishes source frames captured, decoded/selected, and dropped from inferred, rendered, and actually published output frames. The latency histogram has fixed bounds so runs can be merged without retaining per-frame samples. stats.runtime reports only an explicit non-secret allowlist (process/host/GPU visibility plus VIDEO_PROC_IMAGE and VIDEO_PROC_GIT_SHA when deployment injects them). Job identity remains out of Prometheus labels.
  • Two processing modes: batch (files as they are: every frame, in order, faster than real time is the goal) vs stream (real-time pacing, drops under load — cameras, or files standing in for cameras). Details and the is_file gotcha that motivated it: HANDOFF.md §5.
  • Media plane: mediamtx is one static binary doing RTSP ingest + WHEP browser preview; the processor consumes plain RTSP from it. This is the seam where the real relay/cell architecture slots in. The first production cell implements that seam; the next scaling contract is MULTI_CELL_SCALING_RFC.md.

An interactive architecture diagram with scenario walkthroughs (for demos) lives at architecture.html — open it directly in a browser.