Action Recognition: new task, model ABC, and workflow block

[leeclemnet]

What does this PR do?

Adds Action Recognition: a video task that labels frame ranges. Output is (start_frame_idx, end_frame_idx, class) tuples, which can overlap.

The temporal contract travels with the model. VideoSampling states the window, the sample rate, and the limits training set, so a caller sends a clip and nothing else. No caller picks a window length or a frame rate, because those are training decisions. This is the idea the rest of the diff follows from.

Two kinds of checkpoint share one output type, so nothing downstream tells them apart:

• Fine-tune (from https://github.com/roboflow/roboflow-train/pull/880 / https://github.com/roboflow/roboflow-train/pull/884). Sliding windows, class-token spans, and a prefix_allowed_tokens_fn state machine that admits only tokens continuing a valid line, so the answer always parses. Runs on a stream and over HTTP.
• Zero-shot. Reads a whole clip in one call and names the events it finds in its own words. Endpoint only: a stream has no end to span, so the block refuses a whole-video package and says where to send it.

Four surfaces reach the task: the block roboflow_core/roboflow_action_recognition_model@v1, both HTTP routes (/infer/action_recognition, and the legacy /{model}/{version} that serverless serves today), and client.infer_action_recognition in the SDK. The server samples, so a request carries a clip and the response covers that clip.

Type of Change

• New feature (non-breaking change that adds functionality)

Testing

• I have tested this change locally
• I have added/updated tests for this change

Test details:

CI

• New unit tests

Local inference server build on L4 VM

• Reproduce NVIDIA's zero-shot temporal localization demo via v0+v1 API

v0  POST /{model}/{version}          [v0]  108.8s (warm start)

v1  POST /infer/action_recognition   [v1]  108.8s (warm start)

output in details below (same for both).

Details

{ "timeline": [ { "start_frame_idx": 0, "end_frame_idx": 15, "class": "The robot's arm scoops popcorn from the machine and begins lifting it toward the cup.", "class_id": -1 }, { "start_frame_idx": 15, "end_frame_idx": 45, "class": "The robot continues to lift the popcorn scoop, moving it closer to the cup while maintaining a steady grip.", "class_id": -1 }, { "start_frame_idx": 45, "end_frame_idx": 75, "class": "The robot's arm positions the popcorn scoop directly above the cup, preparing to release the popcorn.", "class_id": -1 }, { "start_frame_idx": 75, "end_frame_idx": 105, "class": "The robot releases popcorn into the cup, with the arm slightly adjusting its angle for precision.", "class_id": -1 }, { "start_frame_idx": 105, "end_frame_idx": 135, "class": "The robot's arm retracts slightly after releasing popcorn, readying for the next scoop.", "class_id": -1 }, { "start_frame_idx": 135, "end_frame_idx": 165, "class": "The robot scoops another portion of popcorn from the machine, lifting it toward the cup.", "class_id": -1 }, { "start_frame_idx": 165, "end_frame_idx": 195, "class": "The robot positions the popcorn scoop above the cup again, ensuring accurate placement.", "class_id": -1 }, { "start_frame_idx": 195, "end_frame_idx": 225, "class": "The robot releases popcorn into the cup, with the arm making minor adjustments for consistency.", "class_id": -1 }, { "start_frame_idx": 225, "end_frame_idx": 255, "class": "The robot's arm retracts slightly, preparing for the next cycle of scooping.", "class_id": -1 }, { "start_frame_idx": 255, "end_frame_idx": 285, "class": "The robot scoops another portion of popcorn, lifting it toward the cup.", "class_id": -1 }, { "start_frame_idx": 285, "end_frame_idx": 315, "class": "The robot positions the popcorn scoop above the cup, ensuring alignment for release.", "class_id": -1 }, { "start_frame_idx": 315, "end_frame_idx": 345, "class": "The robot releases popcorn into the cup, with the arm making slight adjustments for precision.", "class_id": -1 }, { "start_frame_idx": 345, "end_frame_idx": 375, "class": "The robot's arm retracts slightly, readying for the next scoop.", "class_id": -1 }, { "start_frame_idx": 375, "end_frame_idx": 405, "class": "The robot scoops another portion of popcorn, lifting it toward the cup.", "class_id": -1 }, { "start_frame_idx": 405, "end_frame_idx": 435, "class": "The robot positions the popcorn scoop above the cup, ensuring alignment for release.", "class_id": -1 }, { "start_frame_idx": 435, "end_frame_idx": 465, "class": "The robot releases popcorn into the cup, with the arm making minor adjustments for consistency.", "class_id": -1 }, { "start_frame_idx": 465, "end_frame_idx": 495, "class": "The robot's arm retracts slightly, preparing for the next cycle of scooping.", "class_id": -1 }, { "start_frame_idx": 495, "end_frame_idx": 525, "class": "The robot scoops another portion of popcorn, lifting it toward the cup.", "class_id": -1 }, { "start_frame_idx": 525, "end_frame_idx": 555, "class": "The robot positions the popcorn scoop above the cup, ensuring alignment for release.", "class_id": -1 } ], "source_fps": 30.0, "frame_count": 563, "windows_classified": 1 }

• Reproduce NVIDIA's zero-shot temporal localization demo via v1 API

• Fine-tuned model test via v0+v1 API peter-robicheaux/video-7-cosmos3-edge-t6 (lee-2.mov, in the train split)

v0   5.4s (warm start)
v1   5.7s (warm start) 

[same output both ways]
out  id=2  frames   0-436  =  0.00s -  7.78s     (labelled 0-8s)
in   id=0  frames 436-857  =  7.78s - 15.28s     (labelled 8.5-15.5s)

• Fine-tuned model test via numpy+torch workflow block peter-robicheaux/video-7-cosmos3-edge-t6 (lee-2.mov again, stride=15.5s to ensure inference triggers before the 16s stream ends)

{
  "model_output": {
    "timeline": [
      {
        "start_frame_idx": 1,
        "end_frame_idx": 450,
        "class": "in",
        "class_id": 0
      },
      {
        "start_frame_idx": 464,
        "end_frame_idx": 857,
        "class": "out",
        "class_id": 2
      }
    ],
    "error_status": ""
  }
}

note the frame ranges are similar but the class assignments flip vs the HTTP endpoint. I could not find a bug that explains this, so I believe the fine-tuned model most likely is not confident in the in vs out class.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code where necessary, particularly in hard-to-understand areas
• My changes generate no new warnings or errors
• I have updated the documentation accordingly (if applicable)

Additional Context

CI stays red until an inference-models release carries the new modules and requirements/ pins it. The branch was developed against a source overlay.

ACTION_RECOGNITION_ENABLED gates the v1 route. It currently defaults to on; flip the default before merge if this needs to land dark.

The prompt and decode settings come from measurement against NVIDIA's own demo assets. Zero-shot needs its reasoning pass to localize, which costs 12–80 s per clip depending on length. That is affordable for a clip and not for a stream, which is why the two modes divide the way they do.

🤖 Generated with Claude Code

[github-actions]

👋 Thanks for the pull request! Here is how automated Claude review works here, so you spend credits (and reviewer time) wisely.

🚦 This PR is marked Ready for review, so automated Claude review will run — and every pass spends real credits.

Warning

💸 The Claude reviewer bills in credits, not vibes

Automated review spins up a real agent that reads real code and spends real credits on every pass. It is glad to help — but it is not a rubber duck, a linter you poke in a loop, or a substitute for reading the contributing guide. Treat it like an expensive senior reviewer whose time you booked, and show up prepared.

Draft when unsure, Ready when you mean it:

• 🌱 Not sure the PR is in good shape yet? Keep it (or set it back) as a draft — drafts pause review, so you can push and iterate without burning credits on a moving target.
• 💪 Feel strong about the contents? Mark it Ready for review and the reviewer will take a look.

However you get there, arrive prepared:

• 🧱 Bring a SOLID, thorough PR. Point your local agent at our skills/ to tune it to our guidelines first — or, if you are one of those fabled carbon-based contributors, read them yourself. A half-baked diff costs exactly the same to review as a finished one.
• ✅ Resolve every comment before you re-request review. Re-requesting with threads still open means paying twice for the same conversation.
• 🔁 Do not use CI review as an inner loop for a local agent. The reviewer is not a step-by-step debugger — do the unfolding locally and arrive with the answer, not the search.
• 🙋 If something looks off, ask a human. One question to a maintainer is cheaper and faster than three rounds of agent re-review chasing a misread.

Reviews are not free. A draft costs nothing to review; a Ready PR is a promise that it is worth reviewing.

• Prefer to skip automated review entirely? Add the skip-claude-review label.

[github-actions]

🤖 Claude review started at commit 10768549ddff9c55c52aaeab0d5e3e88e55a4346.

New commits are not auto-reviewed. Add the claude-review label to request a re-review — the label is consumed when the review starts, so just add it again next time.