Coverage

A common source of error in neural machine translation is dropping or double-translating words in the input sentence. Explicit models of word coverage address this problem.

Coverage is the main subject of 11 publications. 8 are discussed here.

Publications

Chen et al. (2016); Liu et al. (2016) add supervised word alignment information (obtained with traditional statistical word alignment methods) to training. They augment the objective function to also optimize matching of the attention mechanism to the given alignments. To better model coverage, Tu et al. (2016) add coverage states for each input word by either (a) summing up attention values, scaled by a fertility value predicted from the input word in context, or (b) learning a coverage update function as a feed-forward neural network layer. This coverage state is added as additional conditioning context for the prediction of the attention state. Feng et al. (2016) condition the prediction of the attention state also on the previous context state and also introduce a coverage state (initialized with the sum of input source embeddings) that aims to subtract covered words at each step. Similarly, Meng et al. (2016) separate hidden states that keep track of source coverage and hidden states that keep track of produced output. Cohn et al. (2016) add a number of biases to model coverage, fertility, and alignment inspired by traditional statistical machine translation models. They condition the prediction of the attention state on absolute word positions, the attention state of the previous output word in a limited window, and coverage (added attention state values) over a limited window. They also add a fertility model and add coverage in the training objective. Alkhouli et al. (2016) propose to integrate an alignment model that is similar to word-based statistical machine translation into a basic sequence-to-sequence translation model. This model is trained externally with traditional word alignment methods and informs predictions about which input word to translate next and bases the lexical translation decision on that word. Alkhouli and Ney (2017) combine such a alignment model with the more traditional attention model, showing improvements.

Benchmarks

Discussion

Related Topics

New Publications

• Yang, Zonghan and Cheng, Yong and Liu, Yang and Sun, Maosong (2019): Reducing Word Omission Errors in Neural Machine Translation: A Contrastive Learning Approach, Proceedings of the 57th Conference of the Association for Computational Linguistics

  DELETEIntroduction   ResearchGroups   AvailableSoftware   SpeechTranslation   SignLanguage   ComputerAidedTranslation   OtherApproaches   Deployment Data   ParallelCorpora   ComparableCorpora   Dictionaries   CorpusCleaning   SentenceAlignment   Truecasing   WordSegmentation   SpellingCorrection   SparseData   PivotLanguages   DomainAdaptation WordBasedModels   IBMModels   Symmetrization   WordAlignment     WordAlignmentBasedOnCoocurrence     WordAlignmentWithLinguisticAnnotation     WordAlignmentForSyntaxBasedModels     DiscriminativeWordAlignment     WordAlignmentCombination     WordAlignmentEvaluation     AlignmentOfSubsententialUnits   DictionariesFromComparableCorpora   LexicalChoice   MTWithoutWordAlignment PhraseBasedModels   PhraseBasedModelTraining   PhraseBasedModelEM   PhraseBasedModelRefinement   ContextFeatures   Smoothing   Paraphrasing   PruningLargeModels   SuffixArrays   ReorderingModels   PhraseBasedVsEBMT SyntaxBasedModels   InversionTransductionGrammars   HierarchicalModels   StringToTree   TreeToString   TreeToTree   BinarizingSCFG   GenerativeSyntaxModels   DependencyStructure   DeepSyntax   ParseQuality   ParallelTreebanks   SyntacticCoherence LinguisticProblems   UnknownWords   Morphology   GeneratingRichMorphology   FactoredTranslationModels   Compounds   TranslatingTenseCaseMarkers   TranslatingChunksAndClauses   DocumentLevelAspects   Transliteration     TransliterationWithFSM     TransliterationWithOtherMethods     ForwardTransliteration     TransliterationTrainingData     IntegratingTransliteration   SyntacticPrereordering   POSChunkPrereordering   SyntacticReranking   IntegratingSyntacticFeatures LanguageModels   NGramLanguageModels   TargetedLanguageModels   MorphologicalLanguageModels   VeryLargeLanguageModels Decoding   StackDecoding   ReorderingConstraints   WordBasedDecoding   FiniteStateMachines   DecodingComplexity   SyntaxModelDecoding Evaluation   EvaluationCampaigns   ManualMetrics   TaskBasedMetrics   EditRateMetrics   NGramMetrics   SyntaxBasedMetrics   CorrelationOfMetrics   TrainedMetrics   TranslationDifficulty   StatisticalSignificance MachineLearning   WordGraphs   ParameterTuning   Reranking   LargeScaleDiscriminativeTraining   MinimumBayesRisk   ConfidenceMeasures   SystemCombination   IncrementalUpdating NeuralNetworkModels   NeuralLanguageModels   AttentionModel   Training   Inference   Coverage   Vocabulary   Embeddings   MultilingualWordEmbeddings   MonolingualData   Adaptation   LinguisticAnnotation   MultilingualMultimodalMultitask   AlternativeArchitectures   AnalysisAndVisualization   NeuralComponentsInStatisticalMachineTranslation add

  

  @inproceedings{yang-etal-2019-reducing,
  author = {Yang, Zonghan and Cheng, Yong and Liu, Yang and Sun, Maosong},
  title = {Reducing Word Omission Errors in Neural Machine Translation: A Contrastive Learning Approach},
  booktitle = {Proceedings of the 57th Conference of the Association for Computational Linguistics},
  month = {jul},
  address = {Florence, Italy},
  publisher = {Association for Computational Linguistics},
  url = {https://www.aclweb.org/anthology/P19-1623},
  pages = {6191--6196},
  year = 2019
  }
  Yang et al. (2019)
• Li, Yanyang and Xiao, Tong and Li, Yinqiao and Wang, Qiang and Xu, Changming and Zhu, Jingbo (2018): A Simple and Effective Approach to Coverage-Aware Neural Machine Translation, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)

  DELETEIntroduction   ResearchGroups   AvailableSoftware   SpeechTranslation   SignLanguage   ComputerAidedTranslation   OtherApproaches   Deployment Data   ParallelCorpora   ComparableCorpora   Dictionaries   CorpusCleaning   SentenceAlignment   Truecasing   WordSegmentation   SpellingCorrection   SparseData   PivotLanguages   DomainAdaptation WordBasedModels   IBMModels   Symmetrization   WordAlignment     WordAlignmentBasedOnCoocurrence     WordAlignmentWithLinguisticAnnotation     WordAlignmentForSyntaxBasedModels     DiscriminativeWordAlignment     WordAlignmentCombination     WordAlignmentEvaluation     AlignmentOfSubsententialUnits   DictionariesFromComparableCorpora   LexicalChoice   MTWithoutWordAlignment PhraseBasedModels   PhraseBasedModelTraining   PhraseBasedModelEM   PhraseBasedModelRefinement   ContextFeatures   Smoothing   Paraphrasing   PruningLargeModels   SuffixArrays   ReorderingModels   PhraseBasedVsEBMT SyntaxBasedModels   InversionTransductionGrammars   HierarchicalModels   StringToTree   TreeToString   TreeToTree   BinarizingSCFG   GenerativeSyntaxModels   DependencyStructure   DeepSyntax   ParseQuality   ParallelTreebanks   SyntacticCoherence LinguisticProblems   UnknownWords   Morphology   GeneratingRichMorphology   FactoredTranslationModels   Compounds   TranslatingTenseCaseMarkers   TranslatingChunksAndClauses   DocumentLevelAspects   Transliteration     TransliterationWithFSM     TransliterationWithOtherMethods     ForwardTransliteration     TransliterationTrainingData     IntegratingTransliteration   SyntacticPrereordering   POSChunkPrereordering   SyntacticReranking   IntegratingSyntacticFeatures LanguageModels   NGramLanguageModels   TargetedLanguageModels   MorphologicalLanguageModels   VeryLargeLanguageModels Decoding   StackDecoding   ReorderingConstraints   WordBasedDecoding   FiniteStateMachines   DecodingComplexity   SyntaxModelDecoding Evaluation   EvaluationCampaigns   ManualMetrics   TaskBasedMetrics   EditRateMetrics   NGramMetrics   SyntaxBasedMetrics   CorrelationOfMetrics   TrainedMetrics   TranslationDifficulty   StatisticalSignificance MachineLearning   WordGraphs   ParameterTuning   Reranking   LargeScaleDiscriminativeTraining   MinimumBayesRisk   ConfidenceMeasures   SystemCombination   IncrementalUpdating NeuralNetworkModels   NeuralLanguageModels   AttentionModel   Training   Inference   Coverage   Vocabulary   Embeddings   MultilingualWordEmbeddings   MonolingualData   Adaptation   LinguisticAnnotation   MultilingualMultimodalMultitask   AlternativeArchitectures   AnalysisAndVisualization   NeuralComponentsInStatisticalMachineTranslation add

  

  @InProceedings{P18-2047,
  author = {Li, Yanyang and Xiao, Tong and Li, Yinqiao and Wang, Qiang and Xu, Changming and Zhu, Jingbo},
  title = {A Simple and Effective Approach to Coverage-Aware Neural Machine Translation},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  publisher = {Association for Computational Linguistics},
  pages = {292--297},
  location = {Melbourne, Australia},
  url = {http://aclweb.org/anthology/P18-2047},
  year = 2018
  }
  Li et al. (2018)

Supervised Attention

• Mi, Haitao and Wang, Zhiguo and Ittycheriah, Abe (2016): Supervised Attentions for Neural Machine Translation, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing

  DELETEIntroduction   ResearchGroups   AvailableSoftware   SpeechTranslation   SignLanguage   ComputerAidedTranslation   OtherApproaches   Deployment Data   ParallelCorpora   ComparableCorpora   Dictionaries   CorpusCleaning   SentenceAlignment   Truecasing   WordSegmentation   SpellingCorrection   SparseData   PivotLanguages   DomainAdaptation WordBasedModels   IBMModels   Symmetrization   WordAlignment     WordAlignmentBasedOnCoocurrence     WordAlignmentWithLinguisticAnnotation     WordAlignmentForSyntaxBasedModels     DiscriminativeWordAlignment     WordAlignmentCombination     WordAlignmentEvaluation     AlignmentOfSubsententialUnits   DictionariesFromComparableCorpora   LexicalChoice   MTWithoutWordAlignment PhraseBasedModels   PhraseBasedModelTraining   PhraseBasedModelEM   PhraseBasedModelRefinement   ContextFeatures   Smoothing   Paraphrasing   PruningLargeModels   SuffixArrays   ReorderingModels   PhraseBasedVsEBMT SyntaxBasedModels   InversionTransductionGrammars   HierarchicalModels   StringToTree   TreeToString   TreeToTree   BinarizingSCFG   GenerativeSyntaxModels   DependencyStructure   DeepSyntax   ParseQuality   ParallelTreebanks   SyntacticCoherence LinguisticProblems   UnknownWords   Morphology   GeneratingRichMorphology   FactoredTranslationModels   Compounds   TranslatingTenseCaseMarkers   TranslatingChunksAndClauses   DocumentLevelAspects   Transliteration     TransliterationWithFSM     TransliterationWithOtherMethods     ForwardTransliteration     TransliterationTrainingData     IntegratingTransliteration   SyntacticPrereordering   POSChunkPrereordering   SyntacticReranking   IntegratingSyntacticFeatures LanguageModels   NGramLanguageModels   TargetedLanguageModels   MorphologicalLanguageModels   VeryLargeLanguageModels Decoding   StackDecoding   ReorderingConstraints   WordBasedDecoding   FiniteStateMachines   DecodingComplexity   SyntaxModelDecoding Evaluation   EvaluationCampaigns   ManualMetrics   TaskBasedMetrics   EditRateMetrics   NGramMetrics   SyntaxBasedMetrics   CorrelationOfMetrics   TrainedMetrics   TranslationDifficulty   StatisticalSignificance MachineLearning   WordGraphs   ParameterTuning   Reranking   LargeScaleDiscriminativeTraining   MinimumBayesRisk   ConfidenceMeasures   SystemCombination   IncrementalUpdating NeuralNetworkModels   NeuralLanguageModels   AttentionModel   Training   Inference   Coverage   Vocabulary   Embeddings   MultilingualWordEmbeddings   MonolingualData   Adaptation   LinguisticAnnotation   MultilingualMultimodalMultitask   AlternativeArchitectures   AnalysisAndVisualization   NeuralComponentsInStatisticalMachineTranslation add

  

  @InProceedings{mi-wang-ittycheriah:2016:EMNLP2016,
  author = {Mi, Haitao and Wang, Zhiguo and Ittycheriah, Abe},
  title = {Supervised Attentions for Neural Machine Translation},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  month = {November},
  address = {Austin, Texas},
  publisher = {Association for Computational Linguistics},
  pages = {2283--2288},
  url = {https://aclweb.org/anthology/D16-1249},
  year = 2016
  }
  Mi et al. (2016)