Skip to content

Commit d824e44

Browse files
Bordaclaude
andcommitted
docs: update README scores + add OC-SORT Phase 1 journal
- SDP table: fill OC-SORT autotune+Optuna row (HOTA=58.905, IDF1=71.636, MOTA=66.396, IDSW=291) and SORT row (HOTA=58.026) - Journal: add SORT Phase 1 section (9 kept, 5 reverted, +8.4%) - Journal: add OC-SORT Phase 1 section (7 iters + Codex, +10.4%) with positive experiments table, code features, and key lesson - README Algorithms table: OC-SORT MOT17 HOTA (tuned) 57.9→58.9 - autotune/program.md: OC-SORT Phase 1 findings section — 7 kept changes, tuned best config (HOTA=58.905, +10.4% from 53.351 baseline), Optuna insights --- Co-authored-by: Claude Code <noreply@anthropic.com>
1 parent d3ed0ea commit d824e44

3 files changed

Lines changed: 158 additions & 15 deletions

File tree

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,7 @@ Clean, modular implementations of leading trackers. HOTA scores at default param
8080
| :-------------------------------------------: | :-------------------------------------------------------------: | :--------: | :----------------: | :------------: | :------------: | :-------------: |
8181
| [SORT](https://arxiv.org/abs/1602.00763) | Kalman filter + Hungarian matching baseline. | 55.7 | **57.7** | 70.9 | 81.6 | 45.0 |
8282
| [ByteTrack](https://arxiv.org/abs/2110.06864) | Two-stage association using high and low confidence detections. | 60.1 | 59.1 | **73.0** | **84.0** | 50.2 |
83-
| [OC-SORT](https://arxiv.org/abs/2203.14360) | Observation-centric recovery for lost tracks. | **61.9** | 57.9 | 71.7 | 78.4 | **51.8** |
83+
| [OC-SORT](https://arxiv.org/abs/2203.14360) | Observation-centric recovery for lost tracks. | **61.9** | **58.9** | 71.7 | 78.4 | **51.8** |
8484

8585
For detailed benchmarks and tuned configurations, see the [tracker comparison](https://trackers.roboflow.com/develop/trackers/comparison/).
8686

autotune/README.md

Lines changed: 115 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -120,20 +120,20 @@ Published reference points (MOT17-val, FRCNN, IoU-only): SORT ~45–50 (estimate
120120

121121
Bundled SDP detections; same ground truth as FRCNN. Full 7-sequence eval.
122122

123-
| Config | Metric | ByteTrack | OC-SORT | SORT |
124-
| ------------------- | ------ | ---------- | ----------- | ----------- |
125-
| Defaults | HOTA | 53.941 | 53.351 | 53.217 |
126-
| | IDF1 | 65.402 | 65.817 | 64.538 |
127-
| | MOTA | 62.464 | 58.731 | 61.917 |
128-
| | IDSW | 371 | 283 | 355 |
129-
| + Optuna (n=500) | HOTA | 56.115 | **57.747** | 56.083 |
130-
| | IDF1 | 68.077 | 70.330 | 67.517 |
131-
| | MOTA | 65.602 | 66.215 | 65.283 |
132-
| | IDSW | 329 | 303 | 326 |
133-
| + autotune + Optuna | HOTA | **59.092** | _(pending)_ | _(pending)_ |
134-
| | IDF1 | **71.993** | _(pending)_ | _(pending)_ |
135-
| | MOTA | **66.977** | _(pending)_ | _(pending)_ |
136-
| | IDSW | **259** | _(pending)_ | _(pending)_ |
123+
| Config | Metric | ByteTrack | OC-SORT | SORT |
124+
| ------------------- | ------ | ---------- | ---------- | ----------- |
125+
| Defaults | HOTA | 53.941 | 53.351 | 53.217 |
126+
| | IDF1 | 65.402 | 65.817 | 64.538 |
127+
| | MOTA | 62.464 | 58.731 | 61.917 |
128+
| | IDSW | 371 | 283 | 355 |
129+
| + Optuna (n=500) | HOTA | 56.115 | **57.747** | 56.083 |
130+
| | IDF1 | 68.077 | 70.330 | 67.517 |
131+
| | MOTA | 65.602 | 66.215 | 65.283 |
132+
| | IDSW | 329 | 303 | 326 |
133+
| + autotune + Optuna | HOTA | **59.092** | **58.905** | **58.026** |
134+
| | IDF1 | **71.993** | **71.636** | _(pending)_ |
135+
| | MOTA | **66.977** | **66.396** | _(pending)_ |
136+
| | IDSW | **259** | **291** | _(pending)_ |
137137

138138
> SDP is stronger than FRCNN — expect single-sequence defaults around 60–65 HOTA on MOT17-04, but the 7-sequence Optuna average is lower because the benchmark includes harder sequences that pull the mean down.
139139
@@ -303,6 +303,107 @@ xcycsr 7D Kalman state (−1.2%), anisotropic Q matrix (−0.5%), EMA position b
303303

304304
---
305305

306+
### SORT — Phase 1 Campaign (MOT17-val SDP, 9 kept + 5 reverted)
307+
308+
**Period**: 2026-04-05 → 2026-04-06 | **Baseline**: HOTA = 53.217 | **Final**: HOTA = 57.675 (+8.4%, +4.458 pts) | **Best commit**: `f873a75`
309+
310+
Full iteration log: `.experiments/state/sort-phase1/experiments.jsonl`. The table below shows **kept** experiments only.
311+
312+
#### Positive experiments
313+
314+
| Hypothesis | Commit | HOTA delta at defaults |
315+
| -------------------------------------------------------------------------------------- | --------- | ---------------------------- |
316+
| Kalman covariance dynamics: velocity_decay, q_miss_alpha, p_reset_threshold | `8d66fba` | +0.98% |
317+
| OC-SORT observation-centric velocity re-estimation (oru_threshold) | `de5704a` | +1.22% |
318+
| DIoU replaces IoU in association matrix | `c4de2c6` | +0.17% |
319+
| Confidence-weighted Hungarian assignment (conf_cost_weight) | `eefe13e` | enables Optuna headroom |
320+
| IoU age discount for lost tracks (iou_age_weight) | `c094bcb` | enables Optuna headroom |
321+
| Two-stage confidence-based association (high_conf_det_threshold, stage2_iou_threshold) | `e576b9e` | enables Optuna headroom |
322+
| conf_cost_weight wiring + gap interpolation activation | `25d00c5` | activates existing feature |
323+
| minimum_consecutive_frames 3→2 | `3555147` | faster confirmation |
324+
| Align defaults with Optuna-tuned best_config | `ce69432` | +1.18% (single biggest jump) |
325+
326+
#### Code features added
327+
328+
All features are in `trackers/core/sort/tracker.py` and wired through `optimize_tracking.py`:
329+
330+
| Parameter | Default | What it does |
331+
| ------------------------- | ------- | ---------------------------------------------------------------------------------- |
332+
| `velocity_decay` | 0.82 | Multiplicative attenuation of velocity components each missed frame |
333+
| `q_miss_alpha` | 0.8 | Per-frame Q inflation rate for missed frames: Q_eff = Q × (1 + α × t) |
334+
| `p_reset_threshold` | 10 | Reset error covariance P to identity on re-detection after this many missed frames |
335+
| `oru_threshold` | 3 | Minimum occlusion length (frames) to trigger ORU virtual trajectory replay |
336+
| `conf_cost_weight` | 0.2 | Confidence boost multiplier in Hungarian assignment cost matrix |
337+
| `iou_age_weight` | 0.0 | Age discount factor for stale lost tracks in stage-1 solver cost |
338+
| `high_conf_det_threshold` | 0.0 | Score gate for high-confidence first-stage detections |
339+
| `stage2_iou_threshold` | 0.05 | Independent IoU gate for second-stage low-confidence recovery |
340+
341+
#### What failed (reverted)
342+
343+
xcycsr Kalman state representation (−0.51%), velocity-adaptive Q scaling (−0.06%), Mahalanobis distance gate (regression), GIoU as association metric (regression), OC-SORT velocity correction duplicate (reverted).
344+
345+
#### Key lesson
346+
347+
**A single calibration wave** (aligning defaults with the Optuna best, `ce69432`) accounted for +1.18% — the single largest step in the campaign. Each individual code feature added only 0.17–1.22% on its own, but together they created 8 new tunable dimensions that Optuna could jointly exploit.
348+
349+
---
350+
351+
### OC-SORT — Phase 1 Campaign (MOT17-val SDP, 7 iterations + Codex co-pilot)
352+
353+
**Period**: 2026-04-06 → 2026-04-07 | **Baseline**: HOTA = 53.351 | **Final**: HOTA = 58.905 (+10.4%, +5.554 pts) | **Best commit**: `573e71b`
354+
355+
Full iteration log: `.experiments/state/20260406-225110/experiments.jsonl`. Team Mode (3 axes: kalman-covariance / association-pipeline / post-processing-lifecycle) + Codex co-pilot. Table shows **kept** experiments only.
356+
357+
#### Positive experiments
358+
359+
| Iter | Change | HOTA before → after | Δ pts | Δ % | IDSW |
360+
| -------- | ----------------------------------------------------------- | ------------------- | ------ | ------ | ---- |
361+
| i1 | Gap interpolation (max_interpolation_gap=20) | 53.351 → 55.361 | +2.010 | +3.77% ||
362+
| i2 | Kalman Q/R/P scalar multipliers (q_scale, r_scale, p_scale) | 55.361 → 55.361 | 0.0 | 0.0% ||
363+
| i2-codex | Codex param promotion (Optuna best → default_config) | 55.361 → 58.525 | +3.164 | +5.71% ||
364+
| i3 | DIoU in OCM + OCR association stages | 58.525 → 58.244 | −0.281 | −0.48% ||
365+
| i4 | Confidence-weighted Hungarian (conf_cost_weight) | 58.244 → 58.652 | +0.408 | +0.70% ||
366+
| i5 | IoU age discount for lost tracks (iou_age_weight) | 58.652 → 58.652 | 0.0 | 0.0% ||
367+
| i6 | P reset to identity on re-detection (p_reset_threshold) | 58.652 → 58.652 | 0.0 | 0.0% ||
368+
| i7 | Velocity decay + Q inflation during missed frames | 58.652 → 58.905 | +0.253 | +0.43% | 291 |
369+
370+
<details>
371+
<summary><strong>Experiment descriptions</strong></summary>
372+
373+
- **Gap interpolation** (i1) — post-processing pass that linearly interpolates tracklet positions across gaps up to `max_interpolation_gap` frames. Infrastructure already existed in `optimize_tracking.py`; only `default_config.json` needed updating.
374+
- **Kalman Q/R/P scalars** (i2) — expose global multipliers for the noise matrices to Optuna, replacing hardcoded paper defaults. No change at default params (multipliers=1); creates Optuna headroom.
375+
- **Codex param promotion** (i2-codex) — after the i2 guard's 500-trial run found HOTA=58.525, Codex co-pilot promoted the full-precision Optuna best into `default_config.json`. Largest single step in the campaign.
376+
- **DIoU** (i3) — replaced `sv.box_iou_batch` with `_compute_diou_matrix` in both OCM stage (`ocsort/utils.py`) and OCR stage (`tracker.py`). DIoU penalises centre distance, recovering near-miss associations. Registered as a slight regression at 1-trial defaults (−0.28 HOTA) but within guard tolerance; kept because Optuna later recovered and surpassed with tuned DIoU threshold.
377+
- **conf_cost_weight** (i4) — confidence boost in Hungarian cost matrix: `cost *= (1 + w × conf)`. Higher-confidence detections win ties; gate check uses raw IoU.
378+
- **iou_age_weight** (i5) — age discount for stale lost tracks in stage-1 solver: `cost *= 1/(1 + w × max(0, t−1))`. Pushes long-lost tracks to OCR recovery, keeps stage-1 for recently-seen tracks. Creates Optuna headroom.
379+
- **p_reset_threshold** (i6) — after a gap ≥ threshold frames, reset covariance P to identity on re-detection, discarding stale accumulated uncertainty. Creates Optuna headroom.
380+
- **velocity_decay + q_miss_alpha** (i7) — during missed frames: attenuate velocity components by `velocity_decay` and inflate Q by `(1 + alpha × t)`. Reduces prediction drift during occlusion. Optuna found velocity_decay=0.926, q_miss_alpha=0.512.
381+
382+
</details>
383+
384+
#### Code features added
385+
386+
All features are in `trackers/core/ocsort/tracker.py` + `tracklet.py` and wired through `optimize_tracking.py`:
387+
388+
| Parameter | Default | What it does |
389+
| --------------------------------- | ------- | --------------------------------------------------------------------- |
390+
| `q_scale` / `r_scale` / `p_scale` | tuned | Global multipliers for Kalman Q, R, P noise matrices |
391+
| `conf_cost_weight` | 0.9699 | Confidence boost in Hungarian cost; gate check uses raw IoU |
392+
| `iou_age_weight` | 0.4279 | Age discount for stale lost tracks in stage-1 solver cost |
393+
| `p_reset_threshold` | 8 | Reset P to identity when re-detected after ≥ this many missed frames |
394+
| `velocity_decay` | 0.9260 | Multiplicative attenuation of velocity components each missed frame |
395+
| `q_miss_alpha` | 0.5123 | Per-frame Q inflation rate for missed frames: Q_eff = Q × (1 + α × t) |
396+
397+
#### What failed (reverted)
398+
399+
Two Codex co-pilot passes hit transient guard failures (sort HOTA output missing during parallel 500-trial runs) and were reverted; neither represented a real code regression — re-runs passed. No algorithmic hypotheses were reverted.
400+
401+
#### Key lesson
402+
403+
**`direction_consistency_weight` is near-useless on SDP**: Optuna converged to 0.0006 (essentially zero) for the OCM direction signal. Confidence-based assignment (`conf_cost_weight` = 0.97) proved far more effective as a tiebreaker. The Codex param-promotion step (i2-codex, +5.71%) shows how much headroom existed in the pre-campaign hyperparameters — a single calibration wave after exposing Kalman scalars to Optuna captured the majority of the total gain.
404+
405+
---
406+
306407
## Target analysis
307408

308409
The ByteTrack Phase 2 campaign target of HOTA = 68.0 requires real architectural improvements, not parameter search — Optuna alone on FRCNN detections plateaus around 52–53.

autotune/program.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -213,6 +213,48 @@ Campaign run on `bemch/auto-research` using 3-team parallel strategy (Kalman / A
213213
}
214214
```
215215

216+
### OC-SORT Phase 1 findings — already in the code (do not re-implement)
217+
218+
Campaign run on `bemch/auto-research` using 3-team parallel strategy (Kalman / Association / Lifecycle + Codex co-pilot). Baseline: HOTA 53.351 → tuned (500 trials): **58.9** (+10.4%).
219+
220+
#### Kept changes
221+
222+
| Hypothesis | Commit | HOTA delta at defaults |
223+
| -------------------------------------------------------------------------------- | ---------- | ----------------------- |
224+
| Gap interpolation (max_interpolation_gap=20) | `(iter 1)` | +2.01% |
225+
| Kalman Q/R/P scalar multipliers exposed to Optuna (q_scale, r_scale, p_scale) | `(iter 2)` | enables Optuna headroom |
226+
| DIoU replaces IoU in OCM + OCR association stages | `(iter 3)` | within guard tolerance |
227+
| Confidence-weighted Hungarian assignment (conf_cost_weight) | `(iter 4)` | +0.22% |
228+
| IoU age discount for lost tracks in stage 1 (iou_age_weight) | `(iter 5)` | enables Optuna headroom |
229+
| P reset to identity on re-detection after gap (p_reset_threshold) | `9960dd5` | enables Optuna headroom |
230+
| Velocity decay + Q inflation during missed frames (velocity_decay, q_miss_alpha) | `9525885` | +0.43% to HOTA 58.905 |
231+
232+
**Optuna findings**: `direction_consistency_weight` converged near-zero (0.0006) — OCM direction signal hurts on SDP; `conf_cost_weight` converged high (0.97); `iou_age_weight` = 0.43 is effective; `velocity_decay` = 0.926 + `q_miss_alpha` = 0.512 reduce prediction drift.
233+
234+
**New OC-SORT constructor params**: `conf_cost_weight`, `iou_age_weight`, `p_reset_threshold`, `velocity_decay`, `q_miss_alpha`
235+
236+
#### Tuned best config (ocsort/sdp, 500 trials, HOTA=58.905)
237+
238+
```json
239+
{
240+
"lost_track_buffer": 74,
241+
"minimum_consecutive_frames": 1,
242+
"minimum_iou_threshold": 0.1488,
243+
"direction_consistency_weight": 0.000618,
244+
"high_conf_det_threshold": 0.6876,
245+
"delta_t": 1,
246+
"max_interpolation_gap": 42,
247+
"q_scale": 0.7203,
248+
"r_scale": 1.1889,
249+
"p_scale": 0.0952,
250+
"conf_cost_weight": 0.9699,
251+
"iou_age_weight": 0.4279,
252+
"p_reset_threshold": 8,
253+
"velocity_decay": 0.926,
254+
"q_miss_alpha": 0.5123
255+
}
256+
```
257+
216258
### Agent warning — Kalman patch and state representation
217259

218260
`_apply_kalman_patch` in `optimize_tracking.py` overwrites Q, R, and P with uniform identity-scaled matrices. If the state representation is changed (H-A), the patch must be redesigned to work with the new state dimension and matrix structure.

0 commit comments

Comments
 (0)