Repository Issue Activity (beta)

#3216 fix(benchmarks): replace bare n_shots/n_problems asserts with explicit validation in remaining benchmarks

#3245 ToolCorrectnessMetric reports type mismatches for identical calls in exact-match mode

#3218 fix(scorer): quasi_contains_score silently mis-scores a bare string target

#3210 Scorer.truth_identification_score returns percentages above 100 when predictions/targets contain duplicated indices

#3202 get_turns_in_sliding_window silently yields empty windows when window_size <= 0

#3235 MMLU reuses the first task's few-shot examples across subjects

#3230 New feature: Named Reference Attribution metric — catch citations to the wrong named table, section, or footnote

#3228 MIPROv2's sync execute() calls InstructionProposer.propose() twice, silently doubling LLM cost

#3110 Semantics question: what is a test case's canonical verdict when a judge-based metric is re-run and straddles the threshold?

#3134 log_hyperparameters documented decorator forms raise TypeError

#3220 fix(metrics): whitespace-only actual_output/expected_output pass the empty-param guard

#3214 MLLMImage validates the local flag against its URL with a bare assert that vanishes under python -O

#3212 Winogrande benchmark validates n_shots/n_problems with bare asserts and accepts values that crash evaluate()

#3208 HumanEval benchmark validates n/k with a bare assert and accepts invalid values

#3206 Scorer.pass_at_k silently returns wrong scores for degenerate inputs

#3204 CI lint is red on main: `.scripts/release.py` is not black-formatted, blocking every PR

#3200 ArenaTestCase crashes with an opaque IndexError when an arena has fewer than two contestants

#3198 PatternMatchMetric always treats pattern as regex; literal string matching is awkward

#2903 Return inside finally block suppresses exceptions in async evaluation loop

#2904 LOCAL_EMBEDDING_API_KEY enum value is defined as a tuple instead of a string

#2982 Six benchmarks use 'is not' instead of '!=' for length comparison in batch_predict

#3192 fix(metrics): conversational param guard claims 'cannot be empty' but only rejects None

#3190 fix(scorer): replace assert-based argument validation with explicit exceptions

#3188 fix(scorer): missing optional deps crash with bare NameError instead of a clear error

#3141 ToolCorrectnessMetric under-scores duplicate tool calls in default (non-exact) mode