Decoding : Challenge Problem 2

Decoding is process of taking input that looks like this:

honorables sénateurs , que se est - il passé ici , mardi dernier ?

...And turning into output that looks like this:

honourable senators , what happened here on Tuesday ?

In order to decode we need a probability model over pairs of English and French sentences. You did most of the work of creating such a model in excercise 1. In this assignment, we will give you a fixed model consisting of a (phrase-based) translation model and a language model. Your challenge is to find the most probable translation.

Getting Started

If you don't already have a clone of the repository from exercise 1, you can obtain a version by running the command:

git clone git://github.com/alopez/dreamt.git

Under the new decode directory, We have provided you with a very simple decoder written in Python (v.2.6-2.7; older versions will not work!) The decoder translates monotonically — that is, without reordering the English phrases — and by default it also uses very strict pruning limits, which you can vary on the command line. There is also a data directory containing a translation model, a language model, and a set of input sentences to translate. Run the decoder using this command:

decode > output

This loads the models and decodes the input sentences, storing the result in output. You can see the translations simply by looking at the file. To calculate their true model score, run the command:

grade < output

This command computes the probability of the output sentences according to the model. It works by summing over all possible ways that the model could have generated the English from the French. In general this is intractable, but because the phrase dictionary is fixed and sparse, the specific instances here can be computed in a few minutes. It is still easier to do this exactly than it is to find the optimal translation. In fact, if you look at the grade script you may get some hints about how to do the assignment!

Improving the search algorithm in the decoder — for instance by enabling it to search over permutations of English phrases — should permit you to find more probable translations of the input French sentences than the ones found by the baseline system. This exercise differs from exercise 1, in that there is no hidden evaluation measure. The grade program will tell you the probability of your output.

The Challenge

Your task for this assignment is to find the English sentence with the highest possible probability. Formally, this means your goal is to solve the problem: \( \mathop{\arg\,\max}\limits_e~ p(f|e) \times p(e) \), where \(f\) is a French sentence and \(e\) is an English sentence. In the model we have provided you, \( p(e) = p(e_1|START) \times \prod_{j=2}^J p(e_j|e_{j-1}e_{j-2}) \times p(END|e_J,e_{J-1}) \) and \( p(f|e) = p(segmentation) \times p(reordering) \times p(phrase~translation) \). We will make the simplifying assumption that segmentation and reordering probabilities are uniform across all sentences, and hence constant. This results in a model whose probability density function does not sum to one. But from a practical perspective, it slightly simplifies the implementation without substantially harming empirical accuracy. This means that you only need consider the product of the phrase translation probabilities \( p(f|e) = \prod_{\langle i,i',j,j' \rangle \in a} p(f_{\langle i,i' \rangle}|e_{\langle j,j' \rangle}) \) where \( \langle i,i' \rangle \) and \( \langle j,j' \rangle \) index phrases in \(f\) and \(e\), respectively. Unfortunately, even with all of these simplifications, finding the most probable English sentence is completely intractable! To compute it exactly, for each English sentence you would need to compute \( p(f|e) \) as a sum over all possible alignments with the French sentence: \( p(f|e) = \sum_a p(f,a|e) \). A nearly universal approximation is to instead search for the English string together with a single alignment, \(\mathop{\arg\,\max}\limits_{e,a}~ p(f,a|e) \times p(e) \). This is the approach taken by the monotone baseline decoder.

Since this involves multiplying together many small probabilities, it is helpful to work in logspace to avoid numerical underflow. We instead solve for \(e,a\) that maximizes: \( \log p(f,a|e) + \log p(e) = \log p(e_1|START) + \sum_{j=2}^J \log p(e_j|e_{j-1}e_{j-2}) + \log p(END|e_J) + \sum_{\langle i,i',j,j' \rangle \in a} \log p(f_{\langle i,i' \rangle}|e_{\langle j,j' \rangle}) \). The baseline decoder already works with log probabilities, so it is not necessary for you to perform any additional conversion; you can simply work with the sum of the scores that the model provides for you. Note that since probabilities are always less than or equal to one, their equivalent values in logspace will always be negative or zero, respectively (you may notice that grade sometimes reports positive translation model scores; this is because the sum it computes does not include the large negative constant associated with the log probabilities of segmentation and reordering). Hence your translations will always have negative scores, and you will be looking for the one with the smallest absolute value. In other words, we have transformed the problem of finding the most probable translation into a problem of finding the shortest path through a large graph of possible outputs.

Under the phrase-based model we've given you, the goal is to find a phrase segmentation, translation of each resulting phrase, and permutation of those phrases such that the product of the phrase translation probabilities and the language model score of the resulting sentence is as high as possible. Arbitrary permutations of the English phrases are allowed, provided that the phrase translations are one-to-one and exactly cover the input sentence. Even with all of the simplifications we have made, this problem is still NP-Complete, so we recommend that you solve it using an approximate method like stack decoding, discussed in Chapter 6 of the textbook. You can trade efficiency for search effectiveness by implementing histogram pruning or threshold pruning, or by playing around with reordering limits as described in the textbook. Or, you might consider implementing other approaches to solving the search problem:

Several techniques used for the IBM Models (which have very similar search problems as phrase-based models) could also be adapted:

But the sky's the limit! There are many, many ways to try to solve the decoding problem, and you can try anything you want