Skip to content

Add Qwen 3.6 Plus/Flash, 3.5 Plus/Flash snapshots, and Qwen3.6 27B benchmark results - #29

Open
SkalskiP wants to merge 3 commits into
mainfrom
feat/qwen-plus-flash-36-27b
Open

Add Qwen 3.6 Plus/Flash, 3.5 Plus/Flash snapshots, and Qwen3.6 27B benchmark results#29
SkalskiP wants to merge 3 commits into
mainfrom
feat/qwen-plus-flash-36-27b

Conversation

@SkalskiP

@SkalskiP SkalskiP commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Adds five Qwen models via OpenRouter: qwen-3.6-plus, qwen-3.6-flash, qwen-3.5-plus-20260420, qwen-3.5-flash-02-23, and qwen-3.6-27b, each with full-dataset runs for all six tasks at effort low plus reasoning at effort high (35 result files).
  • Adds a --concurrency option to vlm-exam run (thread pool, order-preserving, thread-local image-encode cache in the OpenRouter provider) and extends --resume-file to also re-run empty-content samples via the new is_incomplete_sample helper, with tests.
  • Documents the results publishing policy in AGENTS.md: runs are committed as they came out; the only allowed cleanup is one resume pass for self-inflicted failures (e.g. 429s caused by --concurrency).
  • Regenerates web/benchmark_summary.json and all leaderboard/efficiency charts.

Detection coordinate format

All five models use detection_coordinate_format: xyxy_normalized_0_to_1000, confirmed by a 20-image local format probe on each model before the full runs (the format matching Qwen's documented bbox_2d-style absolute output scored near zero for these API models, while 0-1000-normalized xyxy parsed and scored consistently). Pricing per model comes from the OpenRouter model pages (openrouter.ai/qwen/qwen3.6-plus, qwen3.6-flash, qwen3.5-plus-20260420, qwen3.5-flash-02-23, qwen3.6-27b).

Run provenance

  • All low-effort runs and the qwen-3.6-flash high run are single-pass.
  • qwen-3.5-plus-20260420 high: one resume pass over 25 rate-limit (429) errors caused by our own concurrent request rate.
  • qwen-3.6-27b high: parallel run with one resume pass over empty-content samples; 1 empty remains and is counted as wrong, consistent with existing leaderboard entries (e.g. qwen-3.5-27b high has 7 empties).
  • No existing model's committed results were modified.

Known follow-ups (not in this PR)

  • A uniform retry-failures pass across all committed reasoning runs (partially prepared locally) if we decide to change the methodology.
  • Incomplete samples in some committed non-reasoning runs (e.g. kimi-k2.6 detection has 18 empties).

Test plan

  • ruff check and ruff format --check pass on changed files
  • Full pytest suite passes on the branch (209 tests)
  • web/benchmark_summary.json regenerated with the detection dataset; five new models present at both efforts, no unrelated models included

SkalskiP and others added 3 commits August 21, 2026 13:18
… concurrent runner.

Adds qwen-3.6-plus, qwen-3.6-flash, qwen-3.5-plus-20260420,
qwen-3.5-flash-02-23, and qwen-3.6-27b via OpenRouter, with all six
low-effort task runs plus reasoning at effort high for each. The runner
gains a --concurrency option, --resume-file now also re-runs
empty-content samples, and AGENTS.md documents the publish-as-run
results policy.

Co-authored-by: Cursor <cursoragent@cursor.com>
Model keys and result filenames keep the snapshot identifiers; only the
rendered leaderboard and web summary labels change.

Co-authored-by: Cursor <cursoragent@cursor.com>
Keeps both qwen-3.8-27b (from #28) and the five models in this PR,
then rebuilds the summary and charts so they include the full set.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant