Repository Issue Activity (beta)

#1541 VideoMME score is substantially higher than the reported LLaVA-OneVision-7B result with 32 frames

#1284 MMMU test dataset can be evaluated locally now

#1505 mathvision: _fix_sqrt corrupts indexed roots (\sqrt[3]{8} → \sqrt{[}3]{8}), silently failing the latex2sympy equivalence path

#1519 PyAV packet fallback retains the entire decoded video before frame sampling

#1534 vllm_generate crashes on video tasks for zai-org/GLM-4.1V-9B-Thinking

#1530 vllm_generate ignores trust_remote_code, breaks all custom-code video models

#1509 mathvision: find_math_answer deletes the \text command name but keeps its braces — \boxed{\text{Sunday}} is scored wrong against gold Sunday

#1511 mathvision: find_math_answer keeps the FIRST \\boxed answer when a model retracts it and boxes a corrected one

#1507 mathvision: _fix_fracs is gated on "sqrt" — \frac12-style shorthand never normalizes, silently failing is_equal for sqrt-free answers

#1300 qwen3_5是文本图像多模态模型，但目前只能测评文本

#1432 Decord causes segfaults on teardown

#1215 [Installation] lmms-eval --tasks list returns empty list after pip install

#1240 [Bug] NCCL warning on error path: destroy_process_group() not called

#925 MVBench dataset average result

#1481 OpenAI-compat: --model openai with base_url /v1; leave azure_openai off

#1219 'Qwen3_VL' object has no attribute 'fps'

#1254 📋 Documentation Enhancement Suggestion

#1483 Verify evals on Papers with Code

#1404 [Bug] chat `vllm` model drops video timestamps (to_openai_messages), degrading Qwen3-VL/Qwen3.5 on motion benchmarks

#1476 transform data before feeding into the model

#1393 `qwen2_5_vl` video metadata fix in #1269 does not affect `second_per_grid_ts`; effective workaround is to pass fps, but fps > 2 can break temporal position ids

#1436 [Bug] MMMU open-ended scoring flattens candidate lists before eval_open

#1319 OpenAI chat adapter drops text-task ctx, causing lm-eval-harness task mismatch

#1434 [Bug] BLINK answer extraction misparses verbose responses ("The correct answer is (B)" → "T"), penalizing small/verbose models

#1397 [Bug] SGLang batch reuses the first request's generation kwargs