Skip to content

Pull requests: huggingface/lighteval

New pull request New
77 Open 639 Closed
77 Open 639 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

squad_v2: include unanswerable questions in evaluation
#1185 opened Mar 9, 2026 by Matteovanypersele Loading…
1
Update vllm version requirement to 0.17.0
#1183 opened Mar 9, 2026 by NathanHB Loading…
1
Fail fast on non-retriable LiteLLM status codes
#1182 opened Mar 8, 2026 by yangbaechu Loading…
Redact model config credentials in saved and returned results
#1181 opened Mar 8, 2026 by yangbaechu Loading…
fix(normalizations): guard against index out of range in LogProbToken…
#1180 opened Mar 6, 2026 by inakiLakunza Loading…
Korean completed and Basque fixed
#1179 opened Mar 6, 2026 by inakiLakunza Loading…
[LiteLLM] Add cross-provider reasoning_effort support (first step) + token budget fixes
#1178 opened Mar 5, 2026 by dyurchenko98 Loading…
Fix LiteLLM split iteration in greedy_until to avoid duplicate API requests
#1177 opened Mar 5, 2026 by dyurchenko98 Loading…
Fix corpus reference orientation for chrF/chrF++/TER metrics
#1176 opened Mar 5, 2026 by dyurchenko98 Loading…
1
FIX : handle empty choices in Doc.get_golds() to prevent IndexError
#1174 opened Feb 23, 2026 by nandeanie Loading…
1
1
Fix: pass through custom_tasks and enable multilingual in eval command
#1172 opened Feb 19, 2026 by dzautner Loading…
2 tasks done
Fix IndexError in LogProbTokenNorm when choices_tokens is shorter than choices_logprob
#1171 opened Feb 18, 2026 by worksbyfriday Loading…
Add jfinqa: Japanese Financial Numerical Reasoning QA
#1169 opened Feb 17, 2026 by ajtgjmdjp Loading…
2 of 3 tasks
1
fix: restore task list display logic
#1166 opened Feb 10, 2026 by s1eeping-king Loading…
fix: Transformers Model no template cast stop_sequences to list
#1165 opened Feb 7, 2026 by mrsndmn Loading…
Fix TypeError in aa_omniscience_prompt
#1161 opened Jan 22, 2026 by pjavanrood Loading…
1
Fix split loading error in bigbench
#1159 opened Jan 22, 2026 by pjavanrood Loading…
1
Fix CoQA metric and support multi-doc loading
#1157 opened Jan 22, 2026 by pjavanrood • Draft
1
Fix RecursionError in imdb_contrastset_prompt
#1155 opened Jan 22, 2026 by pjavanrood Loading…
1
Fix legal_summarization keys and SummaC metric
#1153 opened Jan 22, 2026 by pjavanrood • Draft
1
Fix non-existent evaluation splits in lextreme
#1151 opened Jan 22, 2026 by pjavanrood Loading…
1
Fix evaluation split config in lsat_qa
#1149 opened Jan 22, 2026 by pjavanrood Loading…
1
Improve NarrativeQA metrics and prompt structure
#1147 opened Jan 22, 2026 by pjavanrood Loading…
1
Fix schema validation in olympiad_bench Doc.specific
#1145 opened Jan 22, 2026 by pjavanrood Loading…
1
Fix key mismatch and context access in PubMedQA
#1143 opened Jan 22, 2026 by pjavanrood Loading…
1
ProTip! What’s not been updated in a month: updated:<2026-02-09.