feat(multicamera): add world-plane HOTA

[AlexBodner]

What does this PR do?

Adds AI City Challenge 2024 multicamera evaluation on world-plane coordinates. The implementation reads the benchmark's nine-column text format, evaluates each scene with HOTA using a 2 m zero-distance, and reports the protocol's unweighted mean across scenes.

This differs from MOT evaluation in three ways:

• Similarity is max(0, 1 - euclidean_distance / 2 m) on world coordinates rounded to three decimals.
• Camera filtering happens before duplicate (frame_id, object_id) rows are removed; the first row is kept.
• The headline result is the arithmetic mean of per-scene HOTA, DetA, AssA, and LocA. It is not the existing TP-weighted MOT aggregate.

Dataset download support is deferred to the dataset PR. The evaluation CLI and example remain in the follow-up CLI PR.

Scope

• Add load_multicamera_file and load_scene_camera_map for AI City 2024 inputs.
• Add single-scene and multi-scene evaluation entry points in a new trackers.eval.multicamera module.
• Add MulticameraBenchmarkResult and SceneMeanHOTA so the benchmark row carries exactly the four fields the protocol defines, leaving the shared MOT result types untouched.
• Keep a small local fixture with frozen NVIDIA goldens.
• Keep a direct parity script and a compact 30-scene comparison artifact.

The public evaluator expects per-scene inputs:

gt/<scene>/ground_truth.txt
pred/<scene>.txt

Missing prediction files raise an error so scenes cannot be skipped silently. An existing but empty prediction file is valid and produces a zero score for that scene.

Public API

from trackers.eval import evaluate_multicamera_scene, evaluate_multicamera_scenes
from trackers.io.multicamera import load_multicamera_file, load_scene_camera_map

evaluate_multicamera_scene returns a normal SequenceResult containing full HOTA metrics. evaluate_multicamera_scenes returns a MulticameraBenchmarkResult, whose scenes hold per-scene SequenceResult objects and whose aggregate is a SceneMeanHOTA with HOTA, DetA, AssA, and LocA.

The protocol defines no benchmark-level recall, precision, or detection counts under an unweighted mean, so those fields are absent from the aggregate rather than serialized as null. HOTAMetrics and BenchmarkResult are unchanged, MOT result JSON is unaffected, and the two serialized shapes are self-identifying: multicamera files have a top-level scenes key, MOT files have sequences.

Reviewer guide

1. src/trackers/io/multicamera.py

   • Loads and validates nine-column AI City 2024 files.
   • Applies camera filtering, rounding, keep-first deduplication, and NVIDIA's frame offset.
   • Builds per-frame world-plane similarity matrices.

2. src/trackers/eval/multicamera.py

   • Computes HOTA for one scene and resolves per-scene paths for a benchmark run.
   • Defines the two result types and the unweighted scene mean.
   • src/trackers/eval/evaluate.py is byte-identical to develop; MOT evaluation is not touched.

3. src/trackers/eval/results.py

   • Only change is extracting _format_metric_rows, the fixed-width table renderer now shared by the MOT and multicamera tables.
   • Rendered MOT output was checked byte-for-byte against develop's implementation loaded side by side, including long sequence names and single-column tables.

4. scripts/verify_multicamera_eval.py and tests/data/multicamera/

   • Compare the local implementation directly with NVIDIA's pinned evaluator.
   • The script now builds NVIDIA's monolithic input layout from the per-scene fixtures at run time, so the same rows are no longer committed twice.
   • Preserve the frozen local goldens and the compact 30-scene parity record.

The existing HOTA computation kernel is unchanged.

How to try it

Run the committed two-scene fixture:

uv run python - <<'PY'
from trackers.eval import evaluate_multicamera_scenes

result = evaluate_multicamera_scenes(
    gt_dir="tests/data/multicamera/gt",
    tracker_dir="tests/data/multicamera/pred",
    scene_camera_map="tests/data/multicamera/scene_camera_map.json",
)
print(result.table())
PY

Run direct parity against NVIDIA revision 1eebcf0f74a510994fe4c886f4fa77fbc6724ea8:

uv run --with pandas python scripts/verify_multicamera_eval.py \
    --nvidia-eval-dir /path/to/MTMC_Tracking_2024/eval

To check NVIDIA's monolithic full sample too, add:

--sample-dir /path/to/MTMC_Tracking_2024/sample

The script creates temporary per-scene files for that check; this conversion is not part of the public API.

Verification

• Focused multicamera suite: 23 passed.
• Full suite after merging the latest develop: 1,471 passed, 5 skipped.
• Pre-commit on all files, including Ruff, formatting, docformatter, codespell, and mypy.
• mkdocs build --strict.
• Direct local fixture parity against NVIDIA's pinned evaluator.
• MOT table rendering byte-identical to develop.
• All 30 scenes recorded in tests/data/multicamera/tier3_comparison.jsonl at 1e-4 relative and absolute tolerance.
• git diff --check.

The recorded full-benchmark headline values are HOTA 49.2825, DetA 49.1998, AssA 49.3655, and LocA 77.0547.

Type of change

• New feature (non-breaking)

[Copilot]

Pull request overview

Adds AI City 2024 multicamera world-plane HOTA evaluation, including streaming input preparation, scene-mean aggregation, and NVIDIA parity validation.

Changes:

• Adds multicamera parsing and Euclidean HOTA evaluation APIs.
• Adds aggregation and benchmark-coverage metadata.
• Adds fixtures, parity tooling, tests, documentation, and CI integration.

Reviewed changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/trackers/io/multicamera.py	Implements multicamera parsing and preparation.
src/trackers/eval/evaluate.py	Adds scene and benchmark evaluation.
src/trackers/eval/results.py	Adds aggregation and coverage metadata.
src/trackers/eval/hota.py	Validates aggregation arrays.
src/trackers/eval/errors.py	Defines aggregation exception.
src/trackers/eval/__init__.py	Exports new public APIs.
scripts/verify_multicamera_eval.py	Adds NVIDIA parity harness.
tests/io/test_multicamera.py	Tests parsing and preparation.
tests/eval/test_evaluate.py	Tests multicamera evaluation.
tests/eval/test_results.py	Tests result semantics and serialization.
tests/eval/test_integration.py	Adds real-scene parity coverage.
tests/cli/test_verify_multicamera_eval.py	Tests verification tooling.
tests/conftest.py	Adds pinned fixture downloads.
tests/data/multicamera/tier3_validation.md	Records Tier 3 validation summary.
tests/data/multicamera/tier2_expected.json	Stores Tier 2 recipe and golden.
tests/data/multicamera/provenance.json	Records validation provenance.
tests/data/multicamera/expected_results.json	Stores Tier 1 goldens.
tests/data/multicamera/scene_camera_map.json	Defines synthetic scene mapping.
tests/data/multicamera/scene_a_gt.txt	Adds scene A ground truth.
tests/data/multicamera/scene_a_pred.txt	Adds scene A predictions.
tests/data/multicamera/scene_b_gt.txt	Adds scene B ground truth.
tests/data/multicamera/scene_b_pred.txt	Adds scene B predictions.
tests/data/multicamera/combined_gt.txt	Adds combined oracle ground truth.
tests/data/multicamera/combined_pred.txt	Adds combined oracle predictions.
tests/data/multicamera/gt/scene_a/ground_truth.txt	Adds benchmark-layout scene A GT.
tests/data/multicamera/gt/scene_b/ground_truth.txt	Adds benchmark-layout scene B GT.
tests/data/multicamera/pred/scene_a.txt	Adds benchmark-layout scene A predictions.
tests/data/multicamera/pred/scene_b.txt	Adds benchmark-layout scene B predictions.
docs/api/io.md	Documents multicamera loaders.
docs/api/evals.md	Documents evaluation APIs and results.
.github/workflows/ci-integrations.yml	Requires external integration fixtures.
pyproject.toml	Adds Hugging Face development dependency.
uv.lock	Locks the new dependency.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.