Skip to content

feat(webrtc): process video files by URL with optional max_fps cap - #2896

Closed
digaobarbosa wants to merge 1 commit into
mainfrom
feat/webrtc-video-file-url
Closed

feat(webrtc): process video files by URL with optional max_fps cap#2896
digaobarbosa wants to merge 1 commit into
mainfrom
feat/webrtc-video-file-url

Conversation

@digaobarbosa

@digaobarbosa digaobarbosa commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Description

The objective here is to be able to process a video file that was already uploaded to GCS in asset library without a new download and upload through datachannel approach.

Adds two fields to WebRTCWorkerRequest (POST /initialise_webrtc_worker):

  • video_file_url — HTTP(S) URL of a video file (mp4/webm/ogv/...). The server streams it directly with ffmpeg, so no data-channel chunk upload is needed. The URL passes the same SSRF guard used for image URL inputs (_validate_url_destination).
  • max_fps — optional cap on the processed frame rate for video files. Frames are decimated with ffmpeg's fps filter (via PyAV's filter graph) inside ThreadedVideoFileTrack. Unset = all frames. The name reuses the established max_fps contract from InferencePipeline/VideoConfiguration.

With webrtc_realtime_processing: false the file is processed as fast as possible (existing ThreadedVideoFileTrack machinery, no pacing).

Implementation notes

  • The tail of the chunk-upload handler was extracted into _begin_video_file_processing, shared by the upload path and the new URL branch — no duplicated logic. max_fps also applies to chunk-uploaded files.
  • Effective FPS: when max_fps caps a file, _declared_fps (→ VideoFrame.fps / WebRTCVideoMetadata.declared_fps) reports the capped rate, so fps-sensitive workflow blocks (trackers, time-in-zone, velocity) see the real processed rate.
  • The fps filter duplicates frames above the native rate, so the cap is only applied when max_fps < detected fps. In realtime mode max_fps is ignored (realtime has its own frame-drop logic) with a warning.
  • URL processing is deferred until the peer connection and data channel are ready. A finite file processed at full speed finishes before the channel opens, silently dropping every output — caught in an end-to-end run, not by unit tests.
  • Modal: new fields ride the pickled request; billing video_source label video_file_url; request-summary logging. The modal image needs a redeploy before clients send the new fields.

Testing

  • Unit tests: request-model validation and fps-filter decimation on a synthetic mp4 (tests/.../webrtc_worker/test_video_file_source.py); full webrtc_worker suite passes (20/20).
  • E2E against a locally running server with an aiortc client and a 48s/24fps file over HTTP:
    • max_fps: 5 → exactly 240 outputs, declared_fps=5.0, done in 0.9s
    • no max_fps → all 1152 frames, declared_fps=24.0, done in 3.4s
    • processing_complete delivered in both runs

Reviewer notes

  • stream_output: [] is required for data-only sessions (None means "send video back"), same as before — the SDK already does this.
  • Docs updated in docs/webrtc-streaming.md (request example + max_fps semantics).

Allow creating a WebRTC session that processes a video file directly from
an HTTP(S) URL - ffmpeg streams it, so no data-channel upload is needed.
With webrtc_realtime_processing=false every frame is processed as fast as
possible; max_fps optionally caps the rate via ffmpeg's fps filter, and
workflows see the effective (capped) fps so temporal analytics stay correct.

The upload path and the new URL path share the same processing logic
(_begin_video_file_processing). URL processing is deferred until the peer
connection and data channel are ready, since a finite file processed at
full speed would otherwise finish before outputs can reach the client.

URL destinations are validated with the existing SSRF guard used for
image URL inputs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant