Welcome to CHOMPS 2025 Shared Task — SHROOM-CAP, the Shared-task on Hallucinations and Related Observable Overgeneration Mistakes in Crosslingual Analyses of Publications
Welcome to the official shared task website for SHROOM-CAP, a CHOMPS 2025 shared task!
SHROOM-CAP stands for “Shared-task on Hallucinations and Related Observable Overgeneration Mistakes in Crosslingual Analyses of Publications”. SHROOM-CAP will invite participants to detect hallucination in the outputs of LLMs in a scientific context. This shared task extends our previous iteration, SHROOM, with a few key changes:
- We focus on LLM outputs for scientific domain;
- We’re looking at crosslingual setting with both high-resource languages such as, English, Spanish, French, Hindi and suprisal low-level languages;
- Participants will have to detect if hallucination occurs or not.
The information on this website is subject to change. We will send announcements for any major update on the Google group mailing list.
What is SHROOM-CAP?
The task consists of detecting presence of scientific hallucinations. Participants are asked to determine if a given scientific text produced by LLMs constitute hallucinations. The task is held in a cross-lingual setting, i.e., we provide data in multiple mixed languages produced by a variety of public-weights LLMs.´
In practice, we provide an LLM output (as a string of characters, a list of tokens, and a list of logits), and participants have to predict if the LLM output string contains a hallucination (binary classification).
Participants are free to use any approach they deem appropriate, including using external resources, and work on any subset of languages they are interested in.
How will participants be evaluated?
Participants will be evaluated for performing binary classification to identify cases of scientific hallucinations. This will be done using via macro-F1 score for two criterions: (i) Factual Mistakes and (ii) Fluency Mistakes
Rankings and submissions will be done separately per language.
Participant info
To participate, the participants need to register via https://forms.gle/hWR9jwTBjZQmFKAE7. This form will enable us add the participants on the google group for further communication.
Data
Below are links to access the data already released, as well as provisional expected release dates for future splits. Do note that release dates are subject to change.
| Dataset split | Access | Description |
|---|---|---|
| Dev Set | download (dev1) | Contains languages: en, hi, es, fr |
| Sample Testing data | download (test1) | Contains sample format of test set |
Important dates
This information is subject to change.
- Starter Release – July 28
- Training Phase July 28 – October 5, 2025
- Testing Phase October 5 – October 15, 2025
- Paper Submission Deadline October 25, 2025
- Notification of Acceptance November 3, 2025
- Camera-ready Due November 11, 2025
- Proceedings Due December 1, 2025
- CHOMPS workshop: 23/24 December 2025 (co-located with AACL 2025)
Organizers of the shared task
- Aman Sinha, Université de Lorraine, France
- Raúl Vázquez, University of Helsinki, Finland
- Timothee Mickus, University of Helsinki, Finland
- Federica Gamba, Charles University, Prague
- Laura Zanella, Independent Researcher
- Sai Asrith Devisetti, IIIT Hyderabad, India
- Secret CHOMPERS to be revealed later….
Looking for something else?
The websites for all the iterations of the shared task are available here: