Unknown Words

Even with large training corpora, unknown words in the test data will appear.

Unknown Words is the main subject of 12 publications.

Topics in LinguisticProblems

Publications

Habash, Nizar (2008): Four Techniques for Online Handling of Out-of-Vocabulary Words in Arabic-English Statistical Machine Translation, Proceedings of ACL-08: HLT, Short Papers

Habash (2008) points out that these come in different types, which require different solutions: unknown names need to be transliterated, unknown morphological variants need to be matched to known ones, and spelling errors need to be corrected. Another case are abbreviation, which may be expanded with a corpus-driven method

Karakos, Damianos and Eisner, Jason and Khudanpur, Sanjeev and Dreyer, Markus (2008): Machine Translation System Combination using ITG-based Alignments, Proceedings of ACL-08: HLT, Short Papers

mentioned in Unknown Words and System Combination

(Karakos et al., 2008).

Benchmarks

Discussion

New Publications

Servan, Christophe and Dymetman, Marc (2015): Adaptation par enrichissement terminologique en traduction automatique statistique fondée sur la génération et le filtrage de bi-segments virtuels, 22ème Conférence sur le Traitement Automatique des Langues Naturelles mentioned in Domain Adaptation, Context Features and Unknown Words
add
@inproceedings{servan:hal-01157850,
author = {Servan, Christophe and Dymetman, Marc},
title = {{Adaptation par enrichissement terminologique en traduction automatique statistique fond{\'e}e sur la g{\'e}n{\'e}ration et le filtrage de bi-segments virtuels}},
url = {https://hal.archives-ouvertes.fr/hal-01157850},
booktitle = {{22{\`e}me Conf{\'e}rence sur le Traitement Automatique des Langues Naturelles}},
address = {Caen, France},
organization = {{ATALA}},
month = {Jun},
keywords = {Traduction Automatique Statistique ; G{\'e}n{\'e}ration Automatique de Texte ; contexte phrastique ; terminologie bilingue},
pdf = {https://hal.archives-ouvertes.fr/hal-01157850/file/terminology-final-version.pdf},
hal_id = {hal-01157850},
hal_version = {v1},
year = 2015
}
Servan and Dymetman (2015)
Tsvetkov, Yulia and Dyer, Chris (2015): Lexicon Stratification for Translating Out-of-Vocabulary Words, Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
add
@InProceedings{tsvetkov-dyer:2015:ACL-IJCNLP,
author = {Tsvetkov, Yulia and Dyer, Chris},
title = {Lexicon Stratification for Translating Out-of-Vocabulary Words},
booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
month = {July},
address = {Beijing, China},
publisher = {Association for Computational Linguistics},
pages = {125--131},
url = {http://www.aclweb.org/anthology/P15-2021},
year = 2015
}
Tsvetkov and Dyer (2015)
Singla, Karan and Sachdeva, Kunal and Bangalore, Srinivas and Sharma, Dipti Misra and Yadav, Diksha (2014): Reducing the Impact of Data Sparsity in Statistical Machine Translation, Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation
add
@InProceedings{singla-EtAl:2014:SSST-8,
author = {Singla, Karan and Sachdeva, Kunal and Bangalore, Srinivas and Sharma, Dipti Misra and Yadav, Diksha},
title = {Reducing the Impact of Data Sparsity in Statistical Machine Translation},
booktitle = {Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation},
month = {October},
address = {Doha, Qatar},
publisher = {Association for Computational Linguistics},
pages = {51--56},
url = {http://www.aclweb.org/anthology/W14-4006},
year = 2014
}
Singla et al. (2014)
Mark Fishel and Rico Sennrich (2014): Handling technical OOVs in SMT, Proceedings of 17th Annual conference of the European Association for Machine Translation
add
@inproceedings{eamt-2014-Fishel,
author = {Mark Fishel and Rico Sennrich},
title = {Handling technical OOVs in SMT},
booktitle = {Proceedings of 17th Annual conference of the European Association for Machine Translation},
pages = {159-162},
url = {http://www.mt-archive.info/10/EAMT-2014-Fishel.pdf},
location = {Dubrovnik, Croatia},
year = 2014
}
Fishel and Sennrich (2014)
Razmara, Majid and Siahbani, Maryam and Haffari, Reza and Sarkar, Anoop (2013): Graph Propagation for Paraphrasing Out-of-Vocabulary Words in Statistical Machine Translation, Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
add
@InProceedings{razmara-EtAl:2013:ACL2013,
author = {Razmara, Majid and Siahbani, Maryam and Haffari, Reza and Sarkar, Anoop},
title = {Graph Propagation for Paraphrasing Out-of-Vocabulary Words in Statistical Machine Translation},
booktitle = {Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
month = {August},
address = {Sofia, Bulgaria},
publisher = {Association for Computational Linguistics},
pages = {1105--1115},
url = {http://www.aclweb.org/anthology/P13-1109},
year = 2013
}
Razmara et al. (2013)
Chung-chi Huang and Ho-ching Yen and Shih-ting Huang and Jason Chang (2010): Using Sublexical Translations to Handle the OOV Problem in MT, Proceedings of the Ninth Conference of the Association for Machine Translation in the Americas
add
@inproceedings{AMTA-2010-Huang,
author = {Chung-chi Huang and Ho-ching Yen and Shih-ting Huang and Jason Chang},
title = {Using Sublexical Translations to Handle the {OOV} Problem in {MT}},
url = {http://www.mt-archive.info/AMTA-2010-Huang.pdf},
booktitle = {Proceedings of the Ninth Conference of the Association for Machine Translation in the Americas},
location = {Denver, Colorado},
year = 2010
}
Huang et al. (2010)
P. Banerjee and S. K. Naskar and J. Roturier and A. Way and J. van Genabith (2012): Domain Adaptation in SMT of User-Generated Forum Content Guided by OOV Word Reduction: Normalization and/or Supplementary Data?, Proceedings of th 16th International Conference of the European Association for Machine Translation (EAMT)
add
@inproceedings{EAMT-2012-Banerjee,
author = {P. Banerjee and S. K. Naskar and J. Roturier and A. Way and J. van Genabith},
title = {Domain Adaptation in SMT of User-Generated Forum Content Guided by {OOV} Word Reduction: Normalization and/or Supplementary Data?},
url = {http://www.mt-archive.info/EAMT-2012-Banerjee},
pages = {169-176},
booktitle = {Proceedings of th 16th International Conference of the European Association for Machine Translation (EAMT)},
location = {Trento, Italy},
editor = {Mauro Cettolo and Marcello Federico and Lucia Specia and Andy Way},
year = 2012
}
Banerjee et al. (2012)
Nizar Habash and Hayden Metsky (2008): Automatic Learning of Morphological Variations for Handling Out-of-Vocabulary Terms in Urdu-English MT, Proceedings of the Eighth Conference of the Association for Machine Translation in the Americas (AMTA)
add
@inproceedings{amta08:Habash,
author = {Nizar Habash and Hayden Metsky},
title = {Automatic Learning of Morphological Variations for Handling Out-of-Vocabulary Terms in {U}rdu-{E}nglish {MT}},
url = {http://www.mt-archive.info/AMTA-2008-Habash.pdf},
googlescholar = {13194293164029515226},
pages = {107--116},
booktitle = {Proceedings of the Eighth Conference of the Association for Machine Translation in the Americas (AMTA)},
location = {Waikiki, Hawaii},
year = 2008
}
Habash and Metsky (2008)
Zhang, Yuejie and Wang, Yang and Xue, Xiangyang (2009): English-Chinese Bi-Directional OOV Translation based on Web Mining and Supervised Learning, Proceedings of the ACL-IJCNLP 2009 Conference Short Papers
add
@InProceedings{zhang-wang-xue:2009:Short,
author = {Zhang, Yuejie and Wang, Yang and Xue, Xiangyang},
title = {English-Chinese Bi-Directional OOV Translation based on Web Mining and Supervised Learning},
booktitle = {Proceedings of the ACL-IJCNLP 2009 Conference Short Papers},
month = {August},
address = {Suntec, Singapore},
publisher = {Association for Computational Linguistics},
pages = {129--132},
url = {http://www.aclweb.org/anthology/P/P09/P09-2033},
year = 2009
}
Zhang et al. (2009)
Mirkin, Shachar and Specia, Lucia and Cancedda, Nicola and Dagan, Ido and Dymetman, Marc and Szpektor, Idan (2009): Source-Language Entailment Modeling for Translating Unknown Terms, Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP
add
@InProceedings{mirkin-EtAl:2009:ACLIJCNLP,
author = {Mirkin, Shachar and Specia, Lucia and Cancedda, Nicola and Dagan, Ido and Dymetman, Marc and Szpektor, Idan},
title = {Source-Language Entailment Modeling for Translating Unknown Terms},
booktitle = {Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP},
month = {August},
address = {Suntec, Singapore},
publisher = {Association for Computational Linguistics},
pages = {791--799},
url = {http://www.aclweb.org/anthology/P/P09/P09-1089},
year = 2009
}
Mirkin et al. (2009)

SMT Research Survey Wiki

A Comprehensive Survey of Statistical Machine Translation Research Publications

Search Descriptions

Unknown Words

Publications

Benchmarks

Discussion

Related Topics

New Publications