Repository Issue Activity (beta)

#322 Community task: Helium open benchmarks

#320 📝 Integration Proposal: CAJAL — Scientific Paper Agent for BigCode

#318 Question: is there room for long horizon "spec tension" tests around complex code tasks?

#304 Unable to execute the MultiPL-E task for C++

#315 Wrong Hugging Face link for Spider dataset in bigcode-evaluation-harness/docs/README.md line 434

#311 Improve pass@1 Score on Humaneval

#313 Support configurability of FIM tokens on SantaCoder

#224 Multiple-E Go test file name suffix does not contain _test.go

#240 Some questions about APPS

#308 testing Humaneval of qwen-2.5-7B-coder-instruct

#307 how to add new model?

#306 When executing languages such as JS and GO in Multiple-E, the generated results suddenly end

#266 What is `fine-tuning` in task submission?

#303 Code-Llama-7B-Python 4 Bit Error on HumanEval

#300 is there a benchmark page on the benchmark results evaluated using bigcode-evaluation-harness

#131 'HumanEval' object has no attribute 'dataset'

#192 Potentially extra slow inference when using LoRA adapter

#271 Evaluating a Model with a Local Dataset in an Offline Environment

#289 HumanEval-X Go evaluation raises error

#297 Select device: GPU for model eval

#290 HumanEval-X generation appears to not time out called subproceses

#288 Adding an assistant_model Argument for Speculative Decoding

#283 Docker image for multiple evalulation broken

#287 Could you share a completed file of generations_mbppplus.json

#285 mbpp pass@1 is lower than mbppplus pass@1 on starcoder2?