feat: batch-processing skill (ETL staging + jobs via inference-cli and MCP tools) - #36
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new batch-processing skill page documenting how to run large-scale Workflow inference without importing data into a workspace, including client-side staging via inference-cli and server-side orchestration via MCP batch-processing tools.
Changes:
- Introduces
skills/batch-processing/SKILL.mdwith guidance on choosing batch processing (ETL) vs datasource bucket mirroring (ELT). - Documents end-to-end flow: CLI install + staging (local/cloud/references), job start/monitor, webhook events, and exporting/downloading results.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The skill described a webhook relay hosted by the MCP server and a batch_processing_run that waited server-side for up to 55 seconds. Both are gone, so the skill was describing behavior that no longer exists. Webhooks are now bring-your-own: pass notifications_url to batch_processing_job_start or --notifications-url on the CLI, pointing at a receiver you control. Local video staging is the one ingest that does not accept the flag, which is now called out. batch_processing_run is single-shot. It reads state, advances what it can, and returns retryAfterSeconds; the MCP host does the waiting. Staging and result export are the two steps that can only run in the CLI, because the server can neither read nor write the user's disk. Everything between needs only an API key, so the skill now gives the commands for the whole run (stage, inspect, process, monitor, export) for images and videos alike, alongside the MCP equivalents, and says when to reach for which. Cloud storage and references files carry videos as well as images, so the choice of create-batch-of-images vs create-batch-of-videos is the author's to make rather than something inferred from a file extension.
|
/jarbas review this pull request for the improvement of humanity |
|
/jarbas review, noting this is a docs-only change |
Review fixes from the companion MCP PR's source-verified review: - The job_start example mixed content_type="images" with max_video_fps, which the tool rejects; the video-only knobs moved to a note. - Job exports are multipart batches: the files-list recipe now explains the automatic sole-part selection and when part_name is needed. - CLI facts sharpened: the 20k reference chunking is images-only (videos cap at 5k, unchunked), Azure URLs ride the SAS token rather than a 24h presign, AWS_REGION applies alongside AWS_ENDPOINT_URL, and small local image batches (32 or fewer) also ignore --notifications-url. - Added batch_processing_jobs_list and _staging_batches_list to the tool map, plus a note that credits are checked before the idempotent-retry comparison. - cloud-storage skill now routes ETL-shaped asks (process without importing) to the batch-processing skill.
The CLI reads ROBOFLOW_API_KEY from the environment and every command accepts --api-key=<key>. A key stored by roboflow login (~/.config/roboflow/config.json) is not picked up by the inference-cli, which trips people who assume the login session carries over, so say it explicitly where the commands are introduced.
|
/jarbas review |
Two cheap probes before any credits are spent: time a single-image workflows_run (second call, past the cold start; over ~1s of model time means CPU workers will crawl), and count the Workflow's model steps via workflows_get (several chained models or large foundation models mean gpu). Also notes the video-FPS multiplier and the workers_per_machine throughput/OOM trade-off.
Mike-Medvedev
left a comment
Mike-Medvedev
left a comment
There was a problem hiding this comment.
Three docs issues. Accuracy vs MCP #129 is otherwise fine (no relay, retryAfterSeconds, CLI-only stage/export).
Also: skills/inference/SKILL.md still has its own Batch Processing section and does not point here. Agents that load roboflow-inference for "batch" will skip this skill. Add a one-line deferral to roboflow-batch-processing.
| (`.tar` / `.tar.gz`) must be unpacked after download, and listings are capped | ||
| at 10,000 entries per call — past that scale use `export-batch`, or import the | ||
| source data into the workspace with datasources (ELT) and work inside Roboflow. |
There was a problem hiding this comment.
Past the 10k files-list cap, the fallback should stay export-batch / pagination / part_name.
Importing source data with datasources (ELT) does not download job results. That path is for labeling/training, which this skill is explicitly not. Agents will abandon ETL and start connect_cloud_storage when they only needed more result files.
- Spell out the full batch_processing_job_* tool names in the first inventory an agent sees; the abbreviated _job_start forms are not real MCP tool names. - Past the 10k files-list cap, keep the fallback inside ETL: paginate with nextPageToken and per-part part_name listings, or use the resumable export-batch. Importing source data with datasources never downloads job results, so that suggestion is gone. - The inference skill's Batch Processing section now defers to the roboflow-batch-processing skill for the full recipe.
|
Addressed all three of Mike's docs issues in 9d8d453:
|
Every job processes a Data Staging batch; the two input paths differ only in who fills it. The platform stages Asset Library selections for you; you stage external files yourself with the inference-cli. The intro now teaches that fold with a three-question decision tree, replacing the two competing sections (input-path choice and ETL vs ELT). ETL/ELT survive only as parentheticals on the external-file branches, and the old fork becomes a composition: mirror with a datasource (bucket mirror), then run an Asset Library job over the imported images. Also renames the staged creator to batch_processing_staged_job_start to match the MCP side, documents the inputSource label on job listings, and aligns the cloud-storage and inference skills with the same framing.
|
Heads-up after the approval: 638da31 restructures the intro to resolve the Asset Library vs ETL/ELT tension. New teaching model: every batch processing job runs over a Data Staging batch, and the two input paths differ only in who fills it (the platform stages Asset Library selections; you stage external files with the inference-cli). The two competing top sections merged into one three-question decision tree, ETL/ELT are now parentheticals, and the datasource-vs-batch-processing fork became a composition (mirror, then Asset Library job). Matches the tool rename in roboflow/roboflow-mcp#129 (batch_processing_staged_job_start) and the new inputSource label on job listings. |
One line so pipeline-minded readers map instantly: a staged job is the T of an ETL flow, an Asset Library job is the T of an ELT flow whose load already happened via uploads or a datasource mirror.
Adds the
batch-processingskill, served by the MCP server atroboflow://skills/batch-processing/SKILLand shipped with the Roboflow plugin, and adds routing pointers to thecloud-storageandinferenceskills.What it covers
cloud-storagepoints back here and notes the composition: mirrored images become Asset Library images and can be batch-processed as Asset Library jobs.cloud-storageextra), staging from local disk, cloud buckets (S3/GCS/Azure with the credential env vars per provider; S3/GCS get 24h presigned URLs while Azure rides the SAS token) or references files, ingest inspection,process-images-with-workflow/process-videos-with-workflowwith their flags,show-job-details/fetch-logs/abort-job/restart-job, andexport-batch.batch_processing_runmaster tool, which is single-shot and returnsretryAfterSecondsfor the host to poll on rather than waiting server-side, plus the staging reads,jobs_list(entries labeled withinputSource:asset-libraryorstaged-batch), and the job lifecycle tools. Thestaged_job_startexample is valid as written (video-only knobs are called out separately, since the tool rejectsmax_video_fpson image jobs).notifications_url/--notifications-url, including the two ingests that drop the flag (local video staging, and simple local image batches of 32 files or fewer). The POST carries an Authorization header with the publishable key.<job-id>-exportstaging batch is multipart; the files-list tool auto-selects the part when there is exactly one and otherwise takespart_name(parts come frombatch_processing_staging_batch_get). Also covers the resumableexport-batchdownload, signed download URLs for shell-less hosts, the 10k files-list cap, and the 7-day staging TTL.Images vs videos is presented as an authoring choice rather than something inferred from a path, since
create-batch-of-imagesandcreate-batch-of-videosboth accept every data source.Companion change
The MCP server side (tools, instructions line for this skill) lands in roboflow/roboflow-mcp#129 for INC-318. That PR bumps its submodule pointer at this branch so the skill resource resolves during review; it should be re-pointed at the merged commit once this PR lands. Merge this PR first.