Repository Issue Activity (beta)

#3915 [Task Proposal] Add JailbreakBench (JBB-Behaviors) — adversarial-prompt robustness eval

#3771 Task proposal: prompt_defense_eval — OWASP LLM02 output-handling benchmark (33 samples / regex-scored)

#3905 Dataset idea: Helium benchmarks

#3866 [Proposal] New Task: Turkish Clinical Source Support (clinical_safety group)

#3900 Recent main merges introduce pre-existing lint violations that block PR CI

#3895 InfiniteBench tasks fail on a clean install

#3881 Request cache key ignores generation_kwargs -> silently reuses cached instances across different sampling parameters

#3103 KorMedMCQA Implementation does not match the original paper

#1754 New Task Request: LegalBench

#3877 Version string uses .dev0 suffix which conflicts with PEP 440 release tagging

#3846 Unable to load XNLI, XCOPA, PAWS-X, and XQuAD without specifying namespace

#1013 [New Task] COLLIE

#2412 `AnthropicChat` fails when "until" is not provided explicitly in "generation_kwargs" or contains whitespace-only options

#3805 RULER hotpot dataset not available

#2690 Strip the input for the three tasks: FDA, SWDE, and SQuAD_completion.

#3365 promptsource:* does not work

#3859 SCROLLS tasks fail to load with recent datasets (dataset scripts no longer supported)

#1083 Add ZeroScrolls Benchmark

#1883 Add Regression Testing

#3831 Paired significance test for comparing two runs (model_comparator.py applies an unpaired z-test to paired data)

#3854 New task proposal: ArabicCulturalQA (cross-dialectal Arabic MCQ + OEQ)

#1554 New Task Request: InflectionAI's Physics GRE

#3027 Add Support Conditional Generation Models like Mistral3

#3849 [Task Proposal] Add SAS-Bench — Fine-Grained Short Answer Scoring (arXiv:2505.07247)

#3212 Squadv2 PostProcessing Issues with unanswerable questions