Shared Task: Biomedical Translation Task
This task aims to evaluate systems on the translation of documents from the biomedical domain.
The test data will consist of biomedical abstracts and summaries of proposals for animal experiments.
This year, the biomedical translation task will address the following language pairs:
- English-Chinese and Chinese-English (en/zh, zh/en)
- English-French and French-English (en/fr, fr/en)
- English-German and German-English (en/de, de/en)
- English-Italian and Italian-English (en/it, it/en)
- English-Portuguese and Portuguese-English (en/pt, pt/en)
- English-Russian and Russian-English (en/ru, ru/en)
- English-Spanish and Spanish-English (en/es, es/en)
We ask the participants to not download the Medline database by themselves in order to retrieve training data.
Submissions that are derived from a model that was trained on the whole PubMed will be not considererd in the evaluation.
Participants can rely on training (and development) data from various sources, for instance:
- The Biomedical Translation repository includes links to
parallel corpora of scientific publications (en/pt, en/es, en/fr, en/de, en/zh, en/it, en/ru), among others
- The UFAL Medical Corpus (formerly HimLCorpus) includes
medical text from various sources for many language pairs (en/es, en/de, en/fr, en/ro)
HimL test sets can be used as the development sets for some language pairs (en/es, en/de, en/fr, en/ro)
- The Khresmoi development data can be used for some language pairs (en/es, en/de, en/fr).
- The UNCorpus contains training data for some languages (en/es, en/fr, en/zh)
- The MeSpEn corpus contains many parallel documents of en/es
- The Scielo full text corpus for en, es and pt
- The Brazilian Thesis and Dissertations for en/pt
Participants are also free to use out-of-domain data.
Evaluation will be carried out both automatically and manually.
Automatic evaluation will make use of standard machine translation metrics, such as BLEU.
Native speakers of each of the languages will manually check the quality of the translation for a small sample of the submissions.
If necessary, we also expect participants to support us in the manual evaluation (accordingly to the number of submissions).
We plan to release test sets for the following language pairs and sources:
- Scientific abstracts:
- Chinese/English (both directions)
- French/English (both directions)
- German/English (both directions)
- Italian/English (both directions)
- Portuguese/English (both directions)
- Russian/English (both directions)
- Spanish/English (both directions)
- Clinical case reports:
- Ontology Concepts:
- Clinical Terminology:
Test Sets and Submission formats
The formats for the clinical case reports, ontology concepts, and clinical terminology sub-task are available in the ClinSpEn Web site.
For the test set of Medline abstracts, the format will be plain text files.
The format will be the following:
DOC_ID SENT_ID SENTENCE_TEXT
The three values are separated by a TAB character:
- DOC_ID: sequential one, e.g. doc1, doc8, not the original PMID in Medline
- SENT_ID: a sequential number from 1 to n
- SENTENCE_TEXT: the sentence text to be translated by the participants
doc1 1 sentence_1
doc2 2 sentence_2
doc2 3 sentence_3
doc2 4 sentence_4
doc2 5 sentence_5
doc2 n sentence_n
doc4 1 sentence_1
doc4 2 sentence_2
The format for the submission will be the same, such as in the example below.
The participants should follow the same order of the sentences as in the original test set file.
doc1 1 translated_sentence_1
doc2 2 translated_sentence_2
doc2 3 translated_sentence_3
doc2 4 translated_sentence_4
doc2 5 translated_sentence_5
doc2 n translated_sentence_n
doc4 1 translated_sentence_1
doc4 2 translated_sentence_2
Please notice that, following general WMT policy explicitly enforced in other tasks, we will release all participants'
submissions after this year's edition of the task to promote further studies.
For all participants, before 4th August: Prepare an abstract of your system (it may be a half/one-page brief description, or already full system description paper)
and upload it to the paper submission system.
Only primary systems of teams that submit an abstract paper will be included in the human evaluation.
It is possible to download test sets and submit the translation either using OCELoT or our submission system.
Our testsets are also included in the General MT Task.
Please check instructions below.
- Register your team at OCELoT.
- Send an email with your name, affiliation, OCELoT username to email@example.com to get your team activated
(it is not possible to submit translations before team validation).
- Download testsets for the Biomedical task and sub-tasks.
- You may want to use XML wrapping and unwrapping scripts.
- Translate testsets.
- Upload your submissions to the OCELoT. Each team is allowed at most 3 submissions per language pair.
Scores in the system do not reflect actual system performance, they are mainly for validation purposes.
WMT Biomedical Submission System
Please register your team using this form.
You will receive a mail with the confirmation of your registration.
The link for submission site will be informed in this mail.
Please register your team as soon as possible.
The Medline test files are available in the WMT'22 biomedical task Google Drive folder.
For the test sets for the "Clinical case reports", "Ontology Concepts", and "Clinical Terminology" sub-tasks, please refere to OCELoT above.
The format for the submission files should include the original test file name preceded by the team identifier
(as registered in the form above) and the run number, following this example for the abstracts:
- The submission file for run 1 of the "ABC" team for the Medline scientific abstracts from English into Spanish should be called
- The submission file for run 3 of the "ABC" team for the Medline scientific abstracts from Spanish into English should be called
Each team will be allowed to submit up to 3 runs per test set.
Initial results for the biomedical task are available here.
|Release of training data for shared tasks||February/March, 2022|
|Release of test data||21st July, 2022|
|Results submission deadline||29th July, 2022|
|Paper submission deadline||7th September, 2022|
|Paper notification||9th October, 2022|
|Camera-ready version due||16th October, 2022|
|Conference EMNLP||7th - 8th December, 2022|
All deadlines are in AoE (Anywhere on Earth).
Rachel Bawden (University of Edinburgh, UK)
Giorgio Maria Di Nunzio (University of Padua, Italy)
Darryl Johan Estrada (Barcelona Supercomputing Center, Spain)
Eulàlia Farré-Maduell (Barcelona Supercomputing Center, Spain)
Cristian Grozea (Fraunhofer Institute, Germany)
Antonio Jimeno Yepes (University of Melbourne, Australia)
Salvador Lima-López (Barcelona Supercomputing Center, Spain)
Martin Krallinger (Barcelona Supercomputing Center, Spain)
Aurélie Névéol (Université Paris Saclay, CNRS, LISN, France)
Mariana Neves (German Federal Institute for Risk Assessment, Germany)
Roland Roller (DFKI, Germany)
Amy Siu (Beuth University of Applied Sciences, Germany)
Philippe Thomas (DFKI, Germany)
Federica Vezzani (University of Padua, Italy)
Maika Vicente Navarro, Maika Spanish Translator, Melbourne, Australia
Dina Wiemann (Novartis, Switzerland)
Lana Yeganova (NCBI/NLM/NIH, USA)
The ClinSpen sub-track was supported by Encargo of Plan TL (SEAD) to BSC and BIOMATDB project of
European Union’s Horizon Europe Coordination & Support Action under Grant Agreement No 101058779 and AI4PROFHEALTH project (PID2020-119266RA-I00).
Please contact us in the mail firstname.lastname@example.org.
Please join our discussion forum.