A Manually-editing Tool of Translation Templates Based on Automatic Evaluation

INTRODUCTION

Expansion of template base is very essential for a transfer-based MT system to improve its translation quality (Yamada et al., 2002). At the same time, translation templates are often applied to many language processing tasks, such as question classification (Peng and Kai-Hui, 2011), information retrieval (Bedi and Chawla, 2010; Rajan and Rajagopalan, 2008) and summary (Mala and Geetha, 2008). Translation templates are often written by knowledge engineers. At the same time, they can also be acquired from bilingual sentence pairs (Zhang et al., 2010). For example, error-driven learning method is also applied to get translation templates from bilingual corpus. But with new translation templates added into template base, the output quality in MT system cannot improve consistently because of redundance and conflicts between templates. Although redundance and conflicts can be eliminated in form, they still exist at a deep layer, when the template base is applied and different templates work together. There are lots of noises in template base. Recent progress in natural language processing such as part-of-speech tagging, syntactic parsing, translation model, automatic translation evaluation, has made it real to develop the tool of checking translation templates semi-automatically now. In order to obtain template base with the best performance, many methods have been presented recently to evaluate translation templates. The frequency has been applied to evaluate translation templates (Menezes and Richardson, 2001). If the frequency of a translation template is low, it will occur in little language phenomenon and it should be deleted from template base. But this method only improves the efficiency of MT system rather than the translation quality. Lavoie uses log-likelihood ratio to evaluate translation templates (Lavoie et al., 2001). Another current topic of machine translation is automatic evaluation of MT quality (Yasuda et al., 2001). These methods aim to replace subjective evaluation in order to speed up the development cycle of MT systems (Akiba et al., 2001). However, they can be utilized not only as developers’ aids but also for tuning MT systems semi-automatically and automatically (Su et al., 1992). With the development of translation assessment, bleu score is presented to facilitate automatic MT evaluation (Papineni et al., 2002). Nist score is also applied to automatic MT evaluation (Doddington, 2002). Automatic evaluation metric is used as feedback to evaluate translation templates and those that contribute adversely to translation quality will be deleted (Imamura et al., 2003). An improved evaluation method is proposed to score translation templates (Chun-Xiang et al., 2009).

In this study, a manually-editing tool is developed in order to facilitate MT knowledge engineers to check and edit translation templates. In this tool, the template’s contribution to the translation of training corpus is computed under 2 g bleu metric. After ten MT knowledge engineers spend 100 h to edit translation templates in this tool, the translation quality of open test corpus is improved.

BLEU EVALUATION SCORE

The bleu score measures the similarity between MT results and reference translation results made by human beings. This similarity is measured by N-gram precision scores. Several kinds of N-grams can be used in bleu evaluation score. One gram precision score indicates the adequacy of word translation and longer N-gram precision scores indicate the fluency of sentence translation. The bleu score is calculated from the product of N-gram precision scores. So this measure combines adequacy and fluency.

Papineni et al. (2002) calculates their modified precision score p_n, for each N-gram length by summing over the matches for every hypothesis sentence S in translations of test corpus C. Because the computation of bleu evaluation score is based on precision and because recall is difficult to formulate over multiple reference translations, a brevity penalty is introduced to compensate for the possibility of proposing high precision hypothesis translations which are too short. So, the brevity penalty BP is calculated in Eq. 1.

(1)

Here, c is the length of the corpus of hypothesis translations and r is the effective length of reference corpus.

Thus, the computation of bleu evaluation score is shown as Eq. 2 describes.

(2)

Here, w_n is the uniform weight. In this paper, the translation assessment score of training corpus is computed under 2 g bleu metric because its speed is very fast. We use N = 2 and uniform weights w_n = 1/N. The bleu score can range from 0 to 1, where higher scores indicate closer matches to the reference translations and where a score of 1 is assigned to a hypothesis translation which exactly matches one of the reference translations.

EDIT TRANSLATION TEMPLATES MANUALLY BASED ON BLEU EVALUATION SCORE

In order to evaluate a translation template, Chinese-English bilingual sentence pairs are selected from target field. Chinese sentences are collected as training corpus which is denoted as T={CS₁, CS₂, …, CS_n}. English sentences are collected as reference translation corpus which is denoted as R={ES₁, ES₂, …, ES_n}. Training corpus is corresponded with reference translation corpus sentence by sentence. ScoreTable is used to record current evaluation score of training corpus’s machine translations. ScoreTable contains translation evaluation score of every Chinese sentence in training corpus T. So, ScoreTable is corresponded with training corpus T sentence by sentence. In template base which is denoted as TB, there are lots of translation templates to be evaluated. The architecture of manually-editing tool is shown in Fig. 1.

Template optimization module is used to evaluate every translation template in TB. The evaluation score will be shown in manually-editing interface, according to which MT knowledge engineers can decide whether this translation template will improve the translation quality of training corpus T or not. When the size of training corpus T is very large, we consider that training corpus T can cover enough more language phenomenon. The process of evaluating translation templates is reasonable which will provide more information to help MT knowledge engineers finishing this edition task. In evaluation of translation templates, MT engine will be used to translate Chinese sentences in training corpus T.

The process of editing translation templates semi-automatically in this tool is shown as follows:

Fig. 1:	Architecture for manually-editing tool

(3)

In this manually-editing tool, we allow MT knowledge engineers to decide whether r should be removed from template base TB or not. Although sum_blue_score is positive for a translation template, MT knowledge engineers think that this translation template can cover some language phenomenon and it will be kept in template base. After they correct condition part and translation part of r according to their language knowledge, r will be kept in template base TB. Although sum_blue_score is negative for a translation template, MT knowledge engineers think that syntax of this translation template is incorrect. So, r should be removed from template base. The manually-editing tool will provide more choices to select and correct translation templates for improving the quality of MT output.

EXPERIMENT

From Fig. 1, we can see that different manually-editing tools of translation templates for different transfer-based MT systems will be gotten, if we change the MT engine. Here, we use MTS2005 as MT engine in this study. MTS2005 is a transfer-based MT system which is developed by MOE-MS Key Laboratory of Natural Language Processing and Speech in Harbin Institute of Technology. There are 5480 translation templates in its template base TB. Sixty thousands of Chinese-English bilingual sentence pairs are selected from traveling fields. Chinese sentences are collected as training corpus T and English sentences are collected as reference translation corpus R. Training corpus T is corresponded with reference translation corpus R sentence by sentence.

Fig. 2:	Interface of manually-editing tool

Visual C++6.0 is used to develop the manually-editing tool of translation templates and its interface. The interface of this manually-editing tool is shown in Fig. 2.

In its interface, we can find Chinese sentence, parsing tree of Chinese sentence, its English translation and the template which is used in the process of translation. At the same time, we can get the lexicon information of the selected Chinese word in Chinese parsing tree.

For example, in the case of the following Chinese fake-parsing results, the process of editing translation templates is shown as follows:

•	Chinese fake-parsing results: NP[BMP [40/m /q] / usde /ng] VBA[ /p /r AP[/a /usdf VP [VO[BVP[ /vg /d /vq]/ng]/vq]]]○/wj
•	Chinese sentence: 40 ○
•	English translation: High temperature of 40°C make everybody so hot that breathing heavily better than the breath come
•	Applied template: #NP 0:BMP+1:Cate=usde+2:Cate=ng->(2,0,1,2)+2:*+I: of+I:_Plu(0,Num=0)

There are evaluation button, adding button, deleting button and editing button in this interface. MT knowledge engineers can click the evaluation button to score the applied template r. If r is deleted from template base, MT knowledge engineers should click the deleting button. Then r will be removed from template base. If r is kept in template base, MT knowledge engineers should click the adding button. If r is corrected, MT knowledge engineers should edit r and click the editing button. The old version of r will be removed from template base and the new version of r will be added into template base.

MT knowledge engineers use 7000 Chinese sentences with fake-parsing results to check translation templates in this manually-editing tool. Ten MT knowledge engineers spend 100 h to finish this task. After MT knowledge engineers check and edit these translation templates semi-automatically in this tool, the size of template base becomes 4015 which is denoted as TB_edition.

We also select another 2100 Chinese-English bilingual sentence pairs from traveling field for open test purpose. Chinese sentences are collected as open test corpus and English sentences are collected as reference translations of open test corpus. Two groups of experiments are conducted. In experiment 1, TB is used to translate open test corpus in MTS2005. In Experiment 2, TB_edition is used to translate open test corpus in MTS2005.

Table 1:	The performance for two groups of experiments

The translations are evaluated under 3-gram bleu metric and experimental results are described in Table 1.

As shown in Table 1, we can find that the performance of TB_edition is better than that of TB. The translation assessment score of open test corpus improves 3.85% under 3-gram bleu metric.