Add Qwen 3.6 Plus/Flash, 3.5 Plus/Flash snapshots, and Qwen3.6 27B benchmark results - #29
Open
SkalskiP wants to merge 3 commits into
Open
Add Qwen 3.6 Plus/Flash, 3.5 Plus/Flash snapshots, and Qwen3.6 27B benchmark results#29SkalskiP wants to merge 3 commits into
SkalskiP wants to merge 3 commits into
Conversation
… concurrent runner. Adds qwen-3.6-plus, qwen-3.6-flash, qwen-3.5-plus-20260420, qwen-3.5-flash-02-23, and qwen-3.6-27b via OpenRouter, with all six low-effort task runs plus reasoning at effort high for each. The runner gains a --concurrency option, --resume-file now also re-runs empty-content samples, and AGENTS.md documents the publish-as-run results policy. Co-authored-by: Cursor <cursoragent@cursor.com>
Model keys and result filenames keep the snapshot identifiers; only the rendered leaderboard and web summary labels change. Co-authored-by: Cursor <cursoragent@cursor.com>
Keeps both qwen-3.8-27b (from #28) and the five models in this PR, then rebuilds the summary and charts so they include the full set. Co-authored-by: Cursor <cursoragent@cursor.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
qwen-3.6-plus,qwen-3.6-flash,qwen-3.5-plus-20260420,qwen-3.5-flash-02-23, andqwen-3.6-27b, each with full-dataset runs for all six tasks at effort low plus reasoning at effort high (35 result files).--concurrencyoption tovlm-exam run(thread pool, order-preserving, thread-local image-encode cache in the OpenRouter provider) and extends--resume-fileto also re-run empty-content samples via the newis_incomplete_samplehelper, with tests.--concurrency).web/benchmark_summary.jsonand all leaderboard/efficiency charts.Detection coordinate format
All five models use
detection_coordinate_format: xyxy_normalized_0_to_1000, confirmed by a 20-image local format probe on each model before the full runs (the format matching Qwen's documentedbbox_2d-style absolute output scored near zero for these API models, while 0-1000-normalized xyxy parsed and scored consistently). Pricing per model comes from the OpenRouter model pages (openrouter.ai/qwen/qwen3.6-plus, qwen3.6-flash, qwen3.5-plus-20260420, qwen3.5-flash-02-23, qwen3.6-27b).Run provenance
qwen-3.6-flashhigh run are single-pass.qwen-3.5-plus-20260420high: one resume pass over 25 rate-limit (429) errors caused by our own concurrent request rate.qwen-3.6-27bhigh: parallel run with one resume pass over empty-content samples; 1 empty remains and is counted as wrong, consistent with existing leaderboard entries (e.g.qwen-3.5-27bhigh has 7 empties).Known follow-ups (not in this PR)
kimi-k2.6detection has 18 empties).Test plan
ruff checkandruff format --checkpass on changed filespytestsuite passes on the branch (209 tests)web/benchmark_summary.jsonregenerated with the detection dataset; five new models present at both efforts, no unrelated models included