Skip to content

Commit 90c7f59

Browse files
japrescottstellasphere
japrescott
and
stellasphere
authored
feat: batch-processing skill (ETL staging + jobs via inference-cli and MCP tools) (#36)
* feat: batch-processing skill (ETL staging + jobs via inference-cli and MCP tools) * docs: batch-processing skill covers the master tool and webhook relay * docs: cover the full inference-cli lifecycle and drop the webhook relay The skill described a webhook relay hosted by the MCP server and a batch_processing_run that waited server-side for up to 55 seconds. Both are gone, so the skill was describing behavior that no longer exists. Webhooks are now bring-your-own: pass notifications_url to batch_processing_job_start or --notifications-url on the CLI, pointing at a receiver you control. Local video staging is the one ingest that does not accept the flag, which is now called out. batch_processing_run is single-shot. It reads state, advances what it can, and returns retryAfterSeconds; the MCP host does the waiting. Staging and result export are the two steps that can only run in the CLI, because the server can neither read nor write the user's disk. Everything between needs only an API key, so the skill now gives the commands for the whole run (stage, inspect, process, monitor, export) for images and videos alike, alongside the MCP equivalents, and says when to reach for which. Cloud storage and references files carry videos as well as images, so the choice of create-batch-of-images vs create-batch-of-videos is the author's to make rather than something inferred from a file extension. * fix: correct the job_start example and the multipart results recipe Review fixes from the companion MCP PR's source-verified review: - The job_start example mixed content_type="images" with max_video_fps, which the tool rejects; the video-only knobs moved to a note. - Job exports are multipart batches: the files-list recipe now explains the automatic sole-part selection and when part_name is needed. - CLI facts sharpened: the 20k reference chunking is images-only (videos cap at 5k, unchunked), Azure URLs ride the SAS token rather than a 24h presign, AWS_REGION applies alongside AWS_ENDPOINT_URL, and small local image batches (32 or fewer) also ignore --notifications-url. - Added batch_processing_jobs_list and _staging_batches_list to the tool map, plus a note that credits are checked before the idempotent-retry comparison. - cloud-storage skill now routes ETL-shaped asks (process without importing) to the batch-processing skill. * docs: spell out both inference-cli auth routes The CLI reads ROBOFLOW_API_KEY from the environment and every command accepts --api-key=<key>. A key stored by roboflow login (~/.config/roboflow/config.json) is not picked up by the inference-cli, which trips people who assume the login session carries over, so say it explicitly where the commands are introduced. * docs: add a cpu-vs-gpu decision recipe for machine_type Two cheap probes before any credits are spent: time a single-image workflows_run (second call, past the cold start; over ~1s of model time means CPU workers will crawl), and count the Workflow's model steps via workflows_get (several chained models or large foundation models mean gpu). Also notes the video-FPS multiplier and the workers_per_machine throughput/OOM trade-off. * Update Batch Processing skill for Asset Library jobs * Clarify Batch Processing resume semantics * docs: address review feedback from Mike - Spell out the full batch_processing_job_* tool names in the first inventory an agent sees; the abbreviated _job_start forms are not real MCP tool names. - Past the 10k files-list cap, keep the fallback inside ETL: paginate with nextPageToken and per-part part_name listings, or use the resumable export-batch. Importing source data with datasources never downloads job results, so that suggestion is gone. - The inference skill's Batch Processing section now defers to the roboflow-batch-processing skill for the full recipe. * docs: fold Asset Library into one batch-processing model Every job processes a Data Staging batch; the two input paths differ only in who fills it. The platform stages Asset Library selections for you; you stage external files yourself with the inference-cli. The intro now teaches that fold with a three-question decision tree, replacing the two competing sections (input-path choice and ETL vs ELT). ETL/ELT survive only as parentheticals on the external-file branches, and the old fork becomes a composition: mirror with a datasource (bucket mirror), then run an Asset Library job over the imported images. Also renames the staged creator to batch_processing_staged_job_start to match the MCP side, documents the inputSource label on job listings, and aligns the cloud-storage and inference skills with the same framing. * docs: bridge the who-stages model back to ETL/ELT terms One line so pipeline-minded readers map instantly: a staged job is the T of an ETL flow, an Asset Library job is the T of an ELT flow whose load already happened via uploads or a datasource mirror. --------- Co-authored-by: Leo Ueno <leo@roboflow.com>
1 parent f3b2cbd commit 90c7f59

3 files changed

Lines changed: 341 additions & 1 deletion

File tree

‎skills/batch-processing/SKILL.md‎

Lines changed: 332 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,332 @@
1+
---
2+
name: roboflow-batch-processing
3+
description: Use when running a Roboflow Workflow over many images or videos with batch processing — Asset Library jobs (the platform stages workspace images for you) or staged jobs (you stage external local/cloud files with the inference-cli), plus monitoring jobs, downloading results, and routing imports to datasources (bucket mirror) instead.
4+
---
5+
6+
> **For agents — source-of-truth:** This skill is authored in [`roboflow/computer-vision-skills`](https://github.com/roboflow/computer-vision-skills) and shipped with the Roboflow plugin. If your client has loaded the plugin (you'll see `roboflow:<name>` skills in your available skills list), use those local skills — they're read fresh from disk every session. The same content served as MCP resources at `roboflow://skills/<name>/...` is a fallback for clients without the plugin and may lag this repo. **Don't call `ReadMcpResourceTool` for `roboflow://skills/...` URIs when a local `roboflow:<name>` skill is available.**
7+
8+
# Batch Processing
9+
10+
Run a Roboflow Workflow over a very large set of images or videos on
11+
Roboflow's autoscaling compute. Every job processes a temporary Data Staging
12+
batch; the two input paths differ only in **who fills that batch**:
13+
14+
- **Asset Library job** — the files are already in Roboflow. You hand the
15+
platform a selection and it selects and stages them for you.
16+
- **Staged job** — the files are outside Roboflow (local disk, cloud bucket,
17+
references file), so you stage them yourself with the inference-cli: only
18+
your machine and credentials can reach them. Nothing is imported into the
19+
workspace, and staged data expires after ~7 days.
20+
21+
## Which path? Three questions
22+
23+
1. **Are the files already in Roboflow (Asset Library)?** Call
24+
`batch_processing_asset_library_job_create` with a stable idempotency key
25+
and exactly one selection: `image_ids`, `query`, or `all_images=true`. The
26+
platform performs access checks, selects the files, stages them, verifies
27+
Workflow compatibility, bills, and registers the durable job. Poll the
28+
returned `taskId` with `batch_processing_asset_library_task_get` until the
29+
task is terminal; then monitor its `jobId` with `batch_processing_job_get`.
30+
2. **Files outside Roboflow, and you only want the outputs?** Stage them
31+
yourself and drive the run with `batch_processing_guide` +
32+
`batch_processing_run` as described below (the classic ETL shape: nothing
33+
lands in the workspace). Staging and result export must run on the machine
34+
that can access the files.
35+
3. **Files outside Roboflow that you want INSIDE it** (labeling, curation,
36+
training)? That is an import, not a batch processing job: mirror the
37+
bucket with a datasource (`connect_cloud_storage`, see the `cloud-storage`
38+
skill). Once mirrored, the files are Asset Library images, so if you also
39+
want bulk predictions, run an Asset Library job over them (the classic ELT
40+
shape: load first, then transform).
41+
42+
"Datasource" and "bucket mirror" are one thing: a datasource is the
43+
user-facing name for a bucket-mirror config, the importer that fills the
44+
Asset Library. It never runs Workflows itself.
45+
46+
In pipeline terms: a staged job is the T of an ETL flow (no load ever
47+
happens), and an Asset Library job is the T of an ELT flow — the load
48+
already happened, via uploads or a datasource mirror.
49+
50+
## Prerequisites
51+
52+
- The workspace needs the batch-processing feature. Gated calls fail with a
53+
402 "Batch processing is not enabled for this workspace. Upgrade your plan
54+
or contact sales at https://roboflow.com/sales."
55+
- Jobs consume credits; the workspace must have a positive balance.
56+
- The Workflow must already exist in the workspace (`workflows_list`,
57+
`workflows_create`). Jobs reference it by `workflow_id`; there is no
58+
inline-spec option.
59+
- API key: the inference-cli reads `ROBOFLOW_API_KEY` from the environment,
60+
and every command also accepts `--api-key=<key>`. A key stored by
61+
`roboflow login` (`~/.config/roboflow/config.json`) is NOT picked up by
62+
the inference-cli: export it or pass `--api-key` explicitly. Mint one with
63+
the `api_keys_create` MCP tool. Never have the user paste a private key
64+
into chat.
65+
66+
## Where each step runs, and why
67+
68+
Two steps are inference-cli only, because the MCP server can neither read nor
69+
write the user's disk:
70+
71+
- **Staging** runs on the machine that can reach the files (local disk) or
72+
with the user's cloud credentials (bucket sources).
73+
- **Exporting results** (`export-batch`) downloads into a local directory.
74+
75+
Everything in between needs only an API key, so it works either way: the MCP
76+
tools (`batch_processing_run`, `batch_processing_staged_job_start`,
77+
`batch_processing_job_get`, `batch_processing_jobs_list`,
78+
`batch_processing_job_logs`, `batch_processing_job_abort`,
79+
`batch_processing_job_restart`, plus the
80+
`batch_processing_asset_library_*` and `batch_processing_staging_*`
81+
families) or the equivalent CLI commands. Prefer the MCP tools when the host has no shell or the user has
82+
no local `inference-cli`; prefer the CLI when the user is already in a
83+
terminal. The full command set for both content types is in sections 1-5.
84+
85+
The `batch_processing_guide` MCP tool routes a request: it settles the batch
86+
id, picks the staging source, and returns the exact ordered commands. It
87+
never touches files or the network, and it does not repeat this document.
88+
89+
## The master tool: `batch_processing_run`
90+
91+
Prefer `batch_processing_run(batch_id, workflow_id, content_type, ...)` to
92+
drive the flow. Each call reads current state, advances what it can, and
93+
returns immediately with a status:
94+
95+
- `staging_required` — no batch yet; the response contains the CLI commands
96+
for the whole run plus cloud-credential guidance. Run them, then call again.
97+
- `ingest_in_progress` — files still registering.
98+
- `ingest_failed` — ingest failed or returned an unknown shard state; no paid
99+
job was started.
100+
- `running` — the job was started or is still working; includes stage progress.
101+
- `completed` — includes the export batch id and the first result files with
102+
signed download URLs.
103+
- `failed` — includes logs and a restart hint.
104+
105+
**The server never waits on your behalf.** Non-terminal responses carry
106+
`retryAfterSeconds`; sleep that long on your side, then call again with the
107+
returned `job_id` and the same `batch_id` to resume. Omit optional creation
108+
settings on a read-only resume: the paid job's stored definition is
109+
authoritative. The server holds no state between calls.
110+
111+
The job is started under an id derived from (workspace, batch, workflow), so
112+
retrying after a lost response re-registers the same job instead of paying for
113+
a second run. (The platform checks credits before that idempotency comparison,
114+
so a retry can still see a 429 first.) A 409 means that id already exists with
115+
different batch/Workflow identity or conflicts with an optional setting you
116+
explicitly supplied. Inspect the job, omit optional settings to monitor it as
117+
stored, or pass a new explicit `job_id` for a separate run.
118+
119+
For the advanced knobs (`max_runtime_seconds`, `max_parallel_tasks`,
120+
`max_image_failure_rate`, `image_outputs_to_save`) use
121+
`batch_processing_staged_job_start` directly.
122+
123+
## Webhooks
124+
125+
Roboflow will POST job and ingest notifications to a URL you control. There is
126+
no MCP-side relay: pass `notifications_url` to `batch_processing_staged_job_start`, or
127+
`--notifications-url` on the CLI commands, pointing at your own receiver. The
128+
POST carries an `Authorization` header with your publishable key.
129+
130+
Caveat: local **video** staging does not support `--notifications-url` (the CLI
131+
prints a warning and drops it), and a small local **image** batch (32 files or
132+
fewer stages as a simple batch) ignores it too. Sharded local image,
133+
cloud-storage and references-file ingests all support it.
134+
135+
If you have no receiver, just poll `batch_processing_job_get` (or
136+
`batch_processing_run`, which reports progress on each call).
137+
138+
## 1. Install the CLI
139+
140+
```bash
141+
pip install inference-cli
142+
# For s3:// gs:// az:// sources:
143+
pip install 'inference-cli[cloud-storage]'
144+
```
145+
146+
Every `inference rf-cloud` command below authenticates via `ROBOFLOW_API_KEY`
147+
from the environment, or `--api-key=<key>` on the command itself.
148+
149+
Cloud credentials are picked up from the standard env chains on the machine
150+
running the CLI: AWS via the default credential chain (`AWS_PROFILE` honored;
151+
R2/MinIO work via `AWS_ENDPOINT_URL`, with `AWS_REGION` applied alongside it),
152+
GCS via `GOOGLE_APPLICATION_CREDENTIALS`, Azure via
153+
`AZURE_STORAGE_ACCOUNT_NAME` plus `AZURE_STORAGE_ACCOUNT_KEY` or
154+
`AZURE_STORAGE_SAS_TOKEN`. For S3/GCS the CLI generates presigned URLs (24h
155+
expiry); for Azure it appends your SAS token, so those URLs stay valid as long
156+
as the token does. Either way the URLs are handed to Roboflow and bucket
157+
secrets never leave the machine.
158+
159+
## 2. Stage a batch
160+
161+
Batch ids: lowercase letters, digits, `-` or `_`. Staged batches expire
162+
after ~7 days.
163+
164+
```bash
165+
# Local images (>32 images are packed into tar shards automatically)
166+
inference rf-cloud data-staging create-batch-of-images \
167+
--batch-id my-batch --images-dir ./images
168+
169+
# Local videos (uploaded one by one via signed URLs)
170+
inference rf-cloud data-staging create-batch-of-videos \
171+
--batch-id my-batch --videos-dir ./videos
172+
173+
# Cloud bucket (S3/GCS/Azure; glob over object paths).
174+
# Videos work exactly the same way: create-batch-of-videos.
175+
inference rf-cloud data-staging create-batch-of-images \
176+
--batch-id my-batch --data-source cloud-storage \
177+
--bucket-path 's3://my-bucket/images/**/*.jpg'
178+
179+
inference rf-cloud data-staging create-batch-of-videos \
180+
--batch-id my-batch --data-source cloud-storage \
181+
--bucket-path 's3://my-bucket/videos/**/*.mp4'
182+
183+
# References file: JSONL lines of {"name": ..., "url": "https://..."}
184+
inference rf-cloud data-staging create-batch-of-images \
185+
--batch-id my-batch --data-source references-file --references refs.jsonl
186+
```
187+
188+
Images vs videos is a choice you make, not something inferred from the path:
189+
`create-batch-of-images` and `create-batch-of-videos` both accept every data
190+
source. Pick the one matching the content.
191+
192+
Sharded, cloud-storage, and references ingests are asynchronous. Wait until
193+
the batch is fully ingested before starting a job:
194+
195+
```bash
196+
inference rf-cloud data-staging show-batch-details --batch-id my-batch
197+
inference rf-cloud data-staging list-ingest-details --batch-id my-batch
198+
```
199+
200+
or the `batch_processing_staging_batch_get` MCP tool, which returns the file
201+
count plus an `ingest` block with `pending` and `failed` flags. Do not start a
202+
job while `pending` is true or `failed` is true: the job costs credits and
203+
would run over incomplete input.
204+
205+
Practical limits: up to 20,000 image references per ingest request
206+
(auto-chunked; video references cap at 5,000 per request and are not chunked),
207+
~1,000 videos per batch suggested, image formats jpg/png/webp/bmp/jp2,
208+
video formats mp4/mov/avi/mkv/flv/wmv/m4v. `batch_processing_staging_batches_list`
209+
shows every staged batch in the workspace (inputs and job results).
210+
211+
## 3. Start the job
212+
213+
CLI, images:
214+
215+
```bash
216+
inference rf-cloud batch-processing process-images-with-workflow \
217+
--batch-id my-batch --workflow-id my-workflow --machine-type gpu
218+
```
219+
220+
CLI, videos:
221+
222+
```bash
223+
inference rf-cloud batch-processing process-videos-with-workflow \
224+
--batch-id my-batch --workflow-id my-workflow --machine-type gpu \
225+
--max-video-fps 5
226+
```
227+
228+
Shared optional flags: `--workers-per-machine 1|2|4|8`,
229+
`--aggregation-format jsonl|csv`, `--save-image-outputs`,
230+
`--image-outputs-to-save <name>`, `--image-input-name <name>`,
231+
`--workflow-params params.json`, `--max-runtime-seconds <n>`,
232+
`--max-parallel-tasks <n>`, `--job-id <id>`, `--job-name <name>`,
233+
`--notifications-url <url>`, `--part-name <part>`.
234+
235+
Images only: `--max-image-failure-rate 0.0-1.0` (the server rejects it on
236+
video jobs). Videos only: `--max-video-fps <n>`.
237+
238+
MCP equivalent:
239+
240+
```
241+
batch_processing_staged_job_start(
242+
job_id="my-stable-job-id", # required; reuse for retries
243+
batch_id="my-batch", workflow_id="my-workflow",
244+
content_type="images", # or "videos"
245+
machine_type="gpu", # cpu|gpu, optional
246+
workers_per_machine=4, # 1, 2, 4 or 8, optional
247+
aggregation_format="jsonl", # or "csv"
248+
save_image_outputs=True, # persist crops/visualizations
249+
)
250+
```
251+
252+
For videos, set `content_type="videos"` and optionally `max_video_fps=5`
253+
(prediction subsampling). The tool rejects `max_video_fps` on image jobs and
254+
`max_image_failure_rate` on video jobs, matching the platform.
255+
256+
### Choosing cpu vs gpu (`machine_type`)
257+
258+
Default compute is CPU. Decide with two quick checks before starting a paid
259+
job over the whole batch:
260+
261+
1. **Test-run the Workflow on one representative image** (`workflows_run` MCP
262+
tool, or the hosted API) and measure the wall time. Run it twice and time
263+
the second call (the first may cold-start). If a single image takes more
264+
than about a second of model time, CPU workers will crawl through a large
265+
batch: take `gpu`.
266+
2. **Inspect the Workflow spec** (`workflows_get`): count the model steps and
267+
note their sizes. One small fine-tuned detector/classifier: `cpu` is the
268+
cheapest and usually enough. Several models chained, or any large
269+
foundation model (SAM family, CLIP, OCR, VLM blocks): `gpu`.
270+
271+
Videos multiply per-frame work with `--max-video-fps`, so lean `gpu` there
272+
too. `workers_per_machine` (1/2/4/8) then scales throughput on one machine:
273+
more workers means better utilization but a higher OOM risk.
274+
275+
Same job_id plus an identical definition is idempotent; a divergent one is
276+
rejected with a 409. The tool checks that ingest is complete before the paid
277+
registration call. For a multipart input, pass `part_name`; it is also
278+
supported by `batch_processing_run`. Both tools default to the current
279+
`inference-models` backend; use `inference_backend="old-inference"` only for a
280+
known compatibility requirement.
281+
282+
## 4. Monitor
283+
284+
```bash
285+
inference rf-cloud batch-processing show-job-details --job-id my-job
286+
inference rf-cloud batch-processing fetch-logs --job-id my-job
287+
inference rf-cloud batch-processing abort-job --job-id my-job
288+
inference rf-cloud batch-processing restart-job --job-id my-job
289+
```
290+
291+
MCP equivalents, which return JSON rather than a rendered table:
292+
293+
- `batch_processing_job_get(job_id)` — status, current/planned stages,
294+
per-stage progress, output batches; `job.isTerminal` + `job.error` are the
295+
end states.
296+
- `batch_processing_job_logs(job_id)` — info/error logs for diagnosis.
297+
- `batch_processing_job_abort(job_id)` / `batch_processing_job_restart(job_id)`
298+
— stop a run, or retry a failed one (optionally overriding machine type,
299+
workers, timeout).
300+
- `batch_processing_jobs_list(search=...)` — the workspace's recent Workflow
301+
jobs, for finding a job id you did not keep. Internal TensorRT compilation
302+
jobs are excluded. Both this tool and `batch_processing_job_get` label each
303+
job with `inputSource`: `asset-library` (platform-staged selection) or
304+
`staged-batch` (a batch you staged yourself).
305+
306+
## 5. Download results
307+
308+
Results land in Data Staging as platform-generated batches:
309+
`<job-id>-processing` (raw per-shard outputs) and `<job-id>-export`
310+
(packaged, downloadable archives).
311+
312+
Downloading writes to disk, so this step is CLI only:
313+
314+
```bash
315+
inference rf-cloud data-staging export-batch \
316+
--batch-id my-job-export --target-dir ./results
317+
```
318+
319+
It is resumable; add `--override-existing` to re-pull content already
320+
exported, and `--part-name <part>` to fetch one part of a multipart batch.
321+
322+
The MCP tool `batch_processing_staging_batch_files_list(batch_id="<job-id>-export")`
323+
lists the same files with signed `downloadURL`s (~24h expiry) so a host with no
324+
shell can still fetch them. Export batches are multipart: the tool selects the
325+
part automatically when there is exactly one, and otherwise asks you to pass
326+
`part_name` (the parts are in `batch_processing_staging_batch_get`). Archives
327+
(`.tar` / `.tar.gz`) must be unpacked after download, and listings are capped
328+
at 10,000 entries per call — past that scale keep paginating with
329+
`nextPageToken` and per-part `part_name` listings, or pull everything with the
330+
resumable `export-batch`.
331+
332+
Do this within 7 days: staged inputs and results expire.

‎skills/cloud-storage/SKILL.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,12 @@ in automatically. Two pieces work together:
1919
> Imports land at the **workspace** level. The datasource API cannot target a
2020
> specific project/dataset today.
2121
22+
> Only want to run a model/Workflow over the files and collect the outputs,
23+
> without importing anything into Roboflow? That is batch processing — see the
24+
> `roboflow-batch-processing` skill. Mirroring composes with it too: imported
25+
> files become Asset Library images, and bulk predictions over them run as
26+
> Asset Library batch processing jobs (same skill).
27+
2228
## Fast path — one call
2329

2430
`connect_cloud_storage` does the whole flow in one shot: (optionally) create a

‎skills/inference/SKILL.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,8 @@ Mitigation strategies:
8282

8383
## Batch Processing
8484

85+
> For the full recipe (staging, jobs, monitoring, results, MCP tools), see the `roboflow-batch-processing` skill; this section is only the deployment comparison.
86+
8587
**What it is.** A Roboflow-managed cloud service that runs a Workflow over a batch of images or videos asynchronously, provisioning the infrastructure for you. *"Ideal for asynchronously processing large amounts of data."* — [Roboflow docs](https://docs.roboflow.com/deploy/batch-processing).
8688

8789
**Problem it solves.** Bulk inference over thousands to millions of files without standing up your own GPUs, queues, or autoscaler. You hand Roboflow a Workflow plus a batch of inputs, pay per job, and get JSON results back when the job finishes.
@@ -93,7 +95,7 @@ Surfaces: Roboflow web UI, `inference rf-cloud` CLI, and REST API.
9395
### Flow
9496

9597
1. Have a saved Workflow in your workspace.
96-
2. Stage inputs as a Data Staging batch (local directory, JSONL of signed URLs, or cloud-storage path on S3 / GCS / Azure).
98+
2. Provide inputs: an Asset Library selection (the platform stages it for you), or stage external files yourself as a Data Staging batch (local directory, JSONL of signed URLs, or cloud-storage path on S3 / GCS / Azure).
9799
3. Submit a job referencing the Workflow + input batch; choose CPU or GPU.
98100
4. Monitor — poll job status or register a webhook.
99101
5. Export the output batch as JSON.

0 commit comments

Comments
 (0)