feat(vision-mcp): add local-first vision engine and MCP server - #1271
feat(vision-mcp): add local-first vision engine and MCP server#1271ARDA7787 wants to merge 15 commits into
Conversation
…ront end Adds a standalone `vision-mcp` package giving an agent structured access to live RF-DETR vision: inference, tracking, zone and line analytics, metrics, and event evidence. Two processes, one package: - `vision-engine` — long-lived asyncio daemon owning cameras, model weights, inference workers, SQLite and artifacts. Local HTTP API bound to 127.0.0.1. - `vision-mcp` — thin stdio MCP server holding no vision state; each tool is a typed wrapper over one engine endpoint. The split keeps the agent out of the per-frame path and keeps historical metrics available when no MCP client is running. The engine contract is defined once as Pydantic models shared by both sides. Includes lazy model loading with warm weight reuse and MPS/CPU device resolution; a stream pipeline with per-stream capture threads, bounded latest-frame queues and reconnect backoff; ByteTrack tracking with polygon zones, line crossings and dwell times; live and historical metrics aggregated into SQLite; and path-traversal plus URL allowlist checks with RTSP credentials redacted from logs. Additive only — no existing rfdetr code is modified, and the new dependencies are scoped to vision-mcp/.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #1271 +/- ##
=======================================
- Coverage 86% 86% -1%
=======================================
Files 114 114
Lines 14880 14880
=======================================
- Hits 12835 12753 -82
- Misses 2045 2127 +82 🚀 New features to boost your workflow:
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f78fcc0b86
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -0,0 +1,10 @@ | |||
| """Local-first RF-DETR vision engine and MCP server. | |||
There was a problem hiding this comment.
This file, representative of the newly added Python package, begins directly with a docstring, and the other added Python modules do likewise; they must all include the repository's required RF-DETR Apache license header before their module content.
AGENTS.md reference: AGENTS.md:L144-L152
Useful? React with 👍 / 👎.
Overall, before sending any PR, especially this large, it is better to open issues and discuss the scope and what would be the desired impact... Still, from the description, it is not clear what the benefit is for the RF-DETR package to have MCP, and what the MCO brings to the table for package users?
While you say you want to keep it separate, why should it be added to this package?
I think this would be a better fit - https://github.com/roboflow/computer-vision-skills |
|
It can offered as an extra package, core package can be kept separate. It provides a maintained, version-compatible adapter around RF-DETR for users using agentic vision workflows. "https://github.com/roboflow/computer-vision-skills" - this repo is for agent configuration managed by roboflow mcp, it does not contain the server implementation. |
|
I agree this is a product decision first, I should have opened an issue before proposing a change of this size. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d6a659453a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if not path.exists() or not path.is_file(): | ||
| raise VisionError(ErrorCode.INVALID_IMAGE, "File does not exist.", {"path": str(path)}) |
There was a problem hiding this comment.
When filesystem_roots has multiple entries and a caller provides a relative source or checkpoint such as image.jpg, resolve_within() succeeds for the first root, so this branch raises if root1/image.jpg is missing instead of trying root2/image.jpg; valid files in later configured roots become unreachable. Treat a missing file as a failed candidate and only raise after all roots have been checked.
Useful? React with 👍 / 👎.
docformatter can collapse blank lines around the docstrings it rewraps, so ruff-format needs the last word on blank-line and whitespace formatting. Running it first stops the two hooks from fighting on every pass. Split docformatter into two hooks: vision-mcp/ wraps at 110 to match the ruff line-length in vision-mcp/pyproject.toml, while the repo root keeps its default width. Re-wraps docstrings across vision-mcp/src accordingly.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8bdbff4688
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "CREATE TABLE IF NOT EXISTS stream_health (" | ||
| "id INTEGER PRIMARY KEY AUTOINCREMENT, stream_id TEXT NOT NULL, at REAL NOT NULL, " | ||
| "state TEXT NOT NULL, health TEXT NOT NULL, processed_fps REAL NOT NULL, " | ||
| "queue_depth INTEGER NOT NULL, dropped_frames INTEGER NOT NULL, last_error TEXT)", | ||
| "CREATE TABLE IF NOT EXISTS detection_summaries (" |
There was a problem hiding this comment.
For any configured stream, the aggregator's first metrics flush inserts into processing_metrics, and the detection-rate/latency/throughput/drop-rate queries select from the same table, but the migrations never create it. On a fresh database those paths raise no such table: processing_metrics instead of returning historical metrics, so add the table with the columns written by MetricsCollector._metrics_row before these writes can run.
Useful? React with 👍 / 👎.
| detections = sv.Detections(xyxy=boxes, confidence=scores, class_id=class_id) | ||
| detections.data["keypoints"] = xy |
There was a problem hiding this comment.
When detect_keypoints handles RFDETRKeypointPreview, RF-DETR's sv.KeyPoints already carries resolved labels in keypoints.data["class_name"] because keypoint class_id values can be sparse or shifted; rebuilding a Detections with only class_id drops that data. convert() then falls back to positional class_names[class_id], so keypoint detections can be reported with the wrong class name or be filtered out by classes=["person"]; copy the keypoint data (at least class_name) into the converted detections.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8bdbff4688
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| def __init__(self, config: TrackingConfig, processing_fps: float) -> None: | ||
| self._tracker = sv.ByteTrack( | ||
| track_activation_threshold=config.track_activation_threshold, | ||
| lost_track_buffer=max(1, round(config.lost_track_seconds * processing_fps)), |
There was a problem hiding this comment.
At the default 3 FPS, this passes lost_track_buffer=3, but Supervision 0.29 scales that value by frame_rate / 30, yielding a zero-frame retention window instead of the configured one second. A detection missing for one processed frame is therefore assigned a new ID when it returns, inflating unique-object, zone-entry, and occupancy metrics; convert lost_track_seconds to ByteTrack's 30-FPS buffer units rather than multiplying it by the stream FPS.
Useful? React with 👍 / 👎.
| return StreamingResponse( | ||
| services.preview.mjpeg(runtime), media_type="multipart/x-mixed-replace; boundary=frame" | ||
| ) |
There was a problem hiding this comment.
When debug preview is enabled, this response advertises boundary=frame, while PreviewService.mjpeg() emits chunks beginning with --visionmcpframe. Multipart clients search for the advertised delimiter, so /debug/stream cannot render frames; use the same boundary constant for the response content type and generated parts.
Useful? React with 👍 / 👎.
| "SELECT class_name, SUM(detections) AS n FROM detection_summaries " | ||
| "WHERE stream_id = ? AND bucket_start >= ? AND bucket_start < ? GROUP BY class_name", | ||
| (stream_id, query.start, query.end), |
There was a problem hiding this comment.
Whenever the rolling query start falls inside a stored aggregation bucket, this predicate excludes that entire bucket even though part of it belongs to the requested window. With the default 30-second aggregation cadence, get_counts_by_class can therefore omit almost 30 seconds of detections; select overlapping buckets and account for their overlap, as the detection-rate and throughput paths already do.
Useful? React with 👍 / 👎.
| @app.exception_handler(VisionError) | ||
| async def vision_error(request: Request, exc: VisionError) -> JSONResponse: | ||
| return JSONResponse(status_code=exc.http_status, content=exc.to_payload()) |
There was a problem hiding this comment.
When a tool request raises a VisionError, this handler only returns the error response; the repository has no callers of EventSink.record_error, which is the sole writer to the errors table. Consequently get_recent_errors remains empty even after model, database, or inference request failures, so the exception paths need to record the failure before responding.
Useful? React with 👍 / 👎.
2 participants
What does this PR do?
This PR adds
vision-mcp/, a standalone, local-first MCP integration for RF-DETR. It gives AI agents structured access to live vision workflows without modifying the existing RF-DETR package.The feature is split into two processes:
vision-engine: a long-running local daemon that owns camera streams, model loading, inference, tracking, SQLite metrics, and generated evidence artifacts. Its HTTP API is bound to127.0.0.1.vision-mcp: a lightweight stdio MCP server that exposes typed tools backed by the engine API, while keeping no vision state itself.It can be used to connect local image/video/RTSP streams to an agent workflow. The agent can request inference, manage streams, inspect tracked objects, configure polygon zones and crossing lines, query live or historical metrics, and retrieve event evidence.
Implementation details include lazy RF-DETR model loading with MPS/CPU resolution, threaded stream capture with bounded latest-frame queues and reconnect backoff, ByteTrack tracking, zone/line/dwell analytics, SQLite-backed metrics, and safeguards for path traversal, URL allowlisting, and RTSP credential redaction.
This contribution is additive: it is scoped entirely to
vision-mcp/and does not change existing RF-DETR code or dependencies.Related Issue(s): None
Type of Change
Testing
Test details:
Added
vision-mcp/tests/test_phase1_contract.py, covering the initial shared API-contract behavior. I have not yet completed a full local validation run, so I am leaving the local-testing checkbox unchecked.Checklist
Additional Context
I am applying to Roboflow, where contributing to an open-source repository was part of the application process. I chose to make this contribution because I wanted to build something genuinely useful around RF-DETR rather than submit a minimal or cosmetic change.
The goal was to explore a practical way for AI agents to use RF-DETR in a local, privacy-conscious vision system while preserving a clean separation between agent-facing tools and the performance-sensitive inference path. I would appreciate feedback on whether this architecture fits the project’s direction, especially since this is a relatively substantial additive feature and was not tied to a pre-approved issue.