Shared Task: Biomedical Translation Task
This task aims to evaluate systems on the translation of documents from the biomedical domain.
The test data will consist of biomedical abstracts and terminologies.
This year, the biomedical translation task will address the following language pairs:
- English-Basque (en/eu) NEW
- English-Chinese and Chinese-English (en/zh, zh/en)
- English-French and French-English (en/fr, fr/en)
- English-German and German-English (en/de, de/en)
- English-Italian and Italian-English (en/it, it/en) NEW
- English-Portuguese and Portuguese-English (en/pt, pt/en)
- English-Russian and Russian-English (en/ru, ru/en) NEW
- English-Spanish and Spanish-English (en/es, es/en)
Participants can rely on training (and development) data from various sources, for instance:
- The Biomedical Translation repository includes links to
parallel corpora of scientific publications (en/pt, en/es, en/fr, en/de, en/zh, en/it, en/ru), among others. NEW
- The UFAL Medical Corpus (formerly HimLCorpus) includes
medical text from various sources for many language pairs (en/es, en/de, en/fr, en/ro).
HimL test sets can be used as the development sets for some language pairs (en/es, en/de, en/fr, en/ro).
- The Khresmoi development data can be used for some language pairs (en/es, en/de, en/fr).
- The UNCorpus contains training data for some languages (en/es, en/fr, en/zh).
- The MeSpEn corpus contains many parallel documents of en/es.
- The Scielo full text corpus for en, es and pt.
- The Brazilian Thesis and Dissertations for en/pt.
- The ICD-10 codes translation train/dev datasets for en/eu. NEW
- The medical domain monolingual corpora includes hospital notes, medical domain wikipedia articles and medical dictionaries (eu). NEW
- Out-of-domain monolingual corpora for eu. NEW
- Out-of-domain parallel corpora (en/eu). NEW
Participants are also free to use out-of-domain data.
- BERTeus, pre-trained model for Basque
Evaluation will be carried out both automatically and manually.
Automatic evaluation will make use of standard machine translation metrics, such as BLEU.
For the evaluation of terminology we will measure two accuracy scores, by relying on strict and partial matches between ground truth and predictions.
Native speakers of each of the languages will manually check the quality of the translation for a small sample of the submissions.
If necessary, we also expect participants to support us in the manual evaluation (accordingly to the number of submissions).
We plan to release test sets for the following language pairs and sources:
- Scientific abstracts:
- English to Basque
- Chinese/English (both directions)
- French/English (both directions)
- German/English (both directions)
- Italian/English (both directions)
- Portuguese/English (both directions)
- Spanish/English (both directions)
- Translation of terms from biomedical terminologies:
Test Sets and Submission formats
Terms from biomedical terminologies:
For the translation of terms, the format of the test set will be one term per line, as in the example below:
Traumatic rupture of left ulnar collateral ligament, subsequent encounter
posterior dislocation of left hip, sequela
Anterior subluxation of right humerus
other subluxation of left foot, subsequent encounter
Hypothermia of newborn
Absence and agenesis of lacrimal apparatus
poisoning by unspecified primarily systemic and hematological agent, assault, initial encounter
Disorders of glycoprotein metabolism
The format for the submission will be the same, such as in the example below. The participants should follow the same order of the terms as in the original test set file.
ezkerreko kubitu aldeko lotailu albokideko etendura traumatikoa, segidako harremana
ezkerreko aldakako atzeko lokadura, sekuela
eskuineko besahezurraren aurreko subluxazio
ezkerreko oineko beste subluxazio batzuk, segidako harremana
malko-aparatuaren absentzia eta agenesia
nagusiki sistemikoa eta hematologikoa den agente zehaztugabeak eragindako pozoidura, erasoa, hasierako harremana
glukoproteinen metabolismoaren nahasmenduak
For the test set of Medline abstracts, the format will be plain text files.
The format will be the following:
DOC_ID SENT_ID SENTENCE_TEXT
The three values are separated by a TAB character:
- DOC_ID: sequential one, e.g. doc1, doc8, not the original PMID in Medline
- SENT_ID: a sequential number from 1 to n
- SENTENCE_TEXT: the sentence text to be translated by the participants
doc1 1 sentence_1
doc2 2 sentence_2
doc2 3 sentence_3
doc2 4 sentence_4
doc2 5 sentence_5
doc2 n sentence_n
doc4 1 sentence_1
doc4 2 sentence_2
The format for the submission will be the same, such as in the example below.
The participants should
doc1 1 translated_sentence_1
doc2 2 translated_sentence_2
doc2 3 translated_sentence_3
doc2 4 translated_sentence_4
doc2 5 translated_sentence_5
doc2 n translated_sentence_n
doc4 1 translated_sentence_1
doc4 2 translated_sentence_2
Please notice that, following general WMT policy explicitly enforced in other tasks, we will release all participants'
submissions after this year's edition of the task to promote further studies.
Please register your team using this form.
You will receive a mail with the confirmation of your registration.
The link for submission site will be informed in this mail Please register your team as soon as possible.
The test files will be available in the WMT'20 biomedical task Google Drive folder ("test_sets" folder).
The format for the submission files should include the original test file name preceded by the team identifier
(as registered in the form above) and the run number, following this example for the abstracts:
A similar format should be followed for the terminology sub-task.
However, there is no need to identifiy the languages, since this task only addressed English to Basque:
- The submission file for run 1 of the "ABC" team for the Medline abstracts from English to Spanish should be called
- The submission file for run 3 of the "ABC" team for the Medline abstracts from Spanish to English should be called
- The submission file for run 1 of the "ABC" team for the abstracts from English to Basque should be called
- The submission file for run 1 of the "ABC" team for the terminology dataset should be called "ABC_run1_terms.txt".
Each team will be allowed to submit up to 3 runs per test set.
Initial results for the biomedical task are available here.
Further, the "submissions" folder in the WMT'20 biomedical task Google Drive contais all submission files from all participants.
The keys for the test sets are available in the WMT'20 biomedical task Google Drive folder ("gold_sets" folder).
The following folders and files are included:
- "gold_test_files" folder: 14 "medline_*.txt" files, one file for the Basque abstract translation and one gold file for the terminology task, i.e., the reference sentences for all test sets.
- "docid_mapping_files" folder: Eight "*_mapping.txt" files for each Medline language pair for mapping document sequentials to PubMed PMIDs.
- "alignment_files" folder: Eight MEDLINE "*_align_validation.tsv" files for each language pair with the manual validation of the automatic alignment. The order of the languages in the alignment (column 2 in the TAB-separated file) is foreign language vs. (<=>) English.
- wmt20bio_align.py: Python script to produce the submission and gold files before carrying out an evaluation.
|Release of training data||March, 2020|
|Release of test data||June 29, 2020|
|Results submission deadline|July 6, 2020 July 9, 2020
|Paper submission deadline|July 15, 2020 August 15, 2020
|Paper notification|August 17, 2020 September 29, 2020
|Camera-ready version due|August 31, 2020 October 10, 2020
|Conference (virtual)||November 16-20, 2020|
Rachel Bawden (University of Edinburgh, UK)
Kevin B. Cohen (University of Colorado, USA)
Giorgio Maria Di Nunzio (University of Padua, Italy)
Cristian Grozea (Fraunhofer Institute, Germany)
Iñigo Jauregi (University of Technology Sydney, Australia)
Antonio Jimeno Yepes (IBM Research Australia)
Nancy Mah (Charite, Germany)
David Martinez (IBM Research Australia)
Aurélie Névéol (LIMSI, CNRS, France)
Mariana Neves (German Federal Institute for Risk Assessment, Germany)
Maite Oronoz (University of the Basque Country)
Olatz Perez de Viñaspre (University of the Basque Country)
Roland Roller (DFKI, Germany)
Amy Siu (Beuth University of Applied Sciences, Germany)
Philippe Thomas (DFKI, Germany)
Federica Vezzani (University of Padua, Italy)
Maika Vicente Navarro, Maika Spanish Translator, Melbourne, Australia
Dina Wiemann (Novartis, Switzerland)
Lana Yeganova (NCBI/NLM/NIH, USA)
Please contact us in the mail firstname.lastname@example.org.
Please also joing our discussion forum.