Repository Issue Activity (beta)

#1679 MathVista post_check: non-numeric answers can never be scored correct (str(res) typo, also in qspatial)

#1690 olmOCRBench: `evaluate()` returns `None`, so the score never reaches `status.json`

#1689 `dump()` silently drops predictions that begin with a URL and exceed 2079 characters

#1661 OpenAI-compat: GPT4V api_base is the full /v1/chat/completions URL, not /v1

#1675 DOCLING / IDEFICS2 / Mantis / XGenMM fail to import on transformers 5.x: AutoModelForVision2Seq was removed

#1665 Structured extra_records output is discarded for the entire dataset if any single sample fails

#1652 Interop idea: a VLMEvalKit-openeval-adapter for portable eval results

#1590 [Bug] Thyme: sandbox image fed in assistant turn + KV-cache reused across re-encoded turns corrupts multi-round vision context

#323 [Help Wanted] Supporting the `chat_inner` API for existing VLMs.

#1632 Add Pluto-AI-Labs/Apollo-VL-Edge-3B to Leaderboard

#1614 有些基准没有放数据url

#1621 Qwen2.5-VL-7B模型，对评测集LogicVista，从推理结果中提取答案错误，导致误判

#1617 MMMU Open-ended问题无法正确评测

#1606 VLM2Bench counting metric silently degenerates to exact match (image_seq_len always defaults to 2)

#1615 OpenVLM: one-decimal Overall makes the board un-checkable, and hides that 8 boards' #1 leads are under 1.2 sigma

#1549 RFC: Ascend NPU device support

#1600 LiveMMBench空链接

#1456 【OmniDocBench】Recommend updating the benchmark version

#1598 请问flyai-vl模型的评测什么时候可以支持？

#1589 [Bug] run.py: LOCAL_RANK defaults to 1 instead of 0

#1563 LongVideoBench `use_subtitle` logic seems inverted

#1582 Adding [BenchCAD]: would you accept execution-based image→CadQuery-code tasks (heavier deps)?

#1580 [Bug] CharXiv_descriptive_val DATASET_MD5 appears to be incorrect after #1555

#1567 MMBench_v11 md5 failed

#1516 About MMBench Evaluation