Moses-support Digest, Vol 120, Issue 30

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Build error (Dingyuan Wang)
2. Re: Cumulative BLEU scores (Nat Gillin)


----------------------------------------------------------------------

Message: 1
Date: Thu, 27 Oct 2016 08:21:15 +0800
From: Dingyuan Wang <abcdoyle888@gmail.com>
Subject: Re: [Moses-support] Build error
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAFt8H76bMuoSgGYSwJD9ECRTbT53E3tcuPENSeQCv841cEccZw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

It compiles well. Thanks.

2016?10?26? ??10:54?"Hieu Hoang" <hieuhoang@gmail.com>???

> fixed.
>
> https://github.com/moses-smt/mosesdecoder/commit/80ae436db17
> 7bfadc2db9e1d078f392acfde844c
>
> it compiles but I didn't test whether it runs ok on this new debian.
> Please let me know if you encounter problems
>
> On 26/10/2016 12:26, Dingyuan Wang wrote:
>
>> Get a stable one, edit /etc/apt/sources.list to change all 'jessie' to
>> 'testing', then apt update && apt dist-upgrade.
>>
>> 2016-10-26 19:09, Hieu Hoang:
>>
>>> i'm not familiar with Debian. How do you install the 'testing'
>>> configuration? The default gcc on there seems to be 4.9.2
>>>
>>>
>>> On 26/10/2016 10:22, Dingyuan Wang wrote:
>>>
>>>> Hi,
>>>>
>>>> Debian "testing" latest, gcc 6.2.0
>>>>
>>>> I do want to test out the moses2.
>>>>
>>>> 2016-10-26 17:17, Hieu Hoang:
>>>>
>>>>> hiya
>>>>>
>>>>> can you tell me which debian version and gcc version you are using.
>>>>>
>>>>> The compile error comes from Moses2. You can disable compilation of
>>>>> Moses2 by adding this argument to the bjam command:
>>>>>
>>>>> bjam .... --no-xmlrpc-c
>>>>>
>>>>>
>>>>> On 26/10/2016 10:10, Dingyuan Wang wrote:
>>>>>
>>>>>> Dear All,
>>>>>>
>>>>>> I recently pulled the git repo and got a build error with moses2 on a
>>>>>> Debian testing machine.
>>>>>>
>>>>>> Build command is
>>>>>>
>>>>>> ./bjam --with-cmph=/usr/local -j6 -a
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Moses-support mailing list
>>>>>> Moses-support@mit.edu
>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>
>>>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161026/e5523ed4/attachment-0001.html

------------------------------

Message: 2
Date: Thu, 27 Oct 2016 09:47:56 +0800
From: Nat Gillin <nat.gillin@gmail.com>
Subject: Re: [Moses-support] Cumulative BLEU scores
To: Philipp Koehn <phi@jhu.edu>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAD2EOZibX5ZAyom2hRmVa2Vh1UtUh+_2ofAjHnNg+RFvN8CsWg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Dear Phi and Moses community,

@Phi, thanks for confirming the note on the 2nd row of BLEU up to the 4th
order being our "normally-regarded" BLEU.

>From looking through some sets, it seems like the cumulative BLEU is some
how linearly declining while the individual BLEU is decaying at some
exponential rate. Possibly this is because the higher order ngrams are
rarer to match than the unigrams/bigrams.

So are we artificially inflating the numbers by overcounting the repeats
(e.g. unigrams appears in bigram) when we solely consider the cumulative
BLEU, esp. when individually the ngram matches are exponentially decaying?


Any input from some statistics pro?

Regards,
Nat

On Thu, Oct 27, 2016 at 3:46 AM, Philipp Koehn <phi@jhu.edu> wrote:

> Hi,
>
> I think you are right - the first set of numbers are the n-gram precisions
> for each order of n-gram.
> The second set are numbers that you get if you take the geometric mean of
> the n-gram precisions.
> Hence, the number under 4-gram is the BLEU score.
>
> The BLEU score is traditionally computed for 1-4 grams, the original BLEU
> paper discusses this.
> There was the expectation that if machine translation gets better, we
> should use higher-order BLEU,
> but we never did.
>
> -phi
>
>
>
>
> On Wed, Oct 26, 2016 at 12:44 AM, Nat Gillin <nat.gillin@gmail.com> wrote:
>
>> Dear Moses community,
>>
>> Ah, I found out what the cumulative means. The cumulative scores are the
>> usual BLEU scores that we report because it includes the order of ngrams
>> before the order that is desired.
>>
>> The only odd numbers from the mteval-v13a.pl are the individual BLEU
>> scores. Is it right that the individual BLEU scores are the bp * weights *
>> modified_precision for each order of ngram? Are there corresponding papers
>> that investigates these numbers?
>>
>> Regards,
>> Nat
>>
>> On Tue, Oct 25, 2016 at 12:02 PM, Nat Gillin <nat.gillin@gmail.com>
>> wrote:
>>
>>> Dear Moses community,
>>>
>>> To make the question clearer:
>>>
>>> The question is why does the cumulative score add the brevity penalty
>>> before taking the exponent at every order of ngram but the individual score
>>> only takes the brevity penalty into account at
>>> https://github.com/moses-smt/mosesdecoder/blob/master/scr
>>> ipts/generic/mteval-v13a.pl#L874
>>>
>>> Any pointers to the papers describing the cumulative score would be nice
>>> =)
>>>
>>> Thanks in advance again,
>>> Nat
>>>
>>> On Tue, Oct 25, 2016 at 11:58 AM, Nat Gillin <nat.gillin@gmail.com>
>>> wrote:
>>>
>>>> Dear Moses Community,
>>>>
>>>> When using mteval-13a.pl, we note that the output looks like this:
>>>>
>>>> length ratio: 1.07303974221267 (1998/1862), penalty (log): 0
>>>>
>>>> NIST score = 5.0564 BLEU score = 0.2318 for system "Google"
>>>>
>>>>
>>>> # ------------------------------------------------------------
>>>> ------------
>>>>
>>>>
>>>> Individual N-gram scoring
>>>>
>>>> 1-gram 2-gram 3-gram 4-gram 5-gram 6-gram 7-gram
>>>> 8-gram 9-gram
>>>>
>>>> ------ ------ ------ ------ ------ ------ ------
>>>> ------ ------
>>>>
>>>> NIST: 4.4488 0.5554 0.0477 0.0045 0.0000 0.0000 0.0000
>>>> 0.0000 0.0000 "Google"
>>>>
>>>>
>>>> BLEU: 0.5415 0.2972 0.1752 0.1025 0.0626 0.0354 0.0193
>>>> 0.0085 0.0017 "Google"
>>>>
>>>>
>>>> # ------------------------------------------------------------
>>>> ------------
>>>>
>>>> Cumulative N-gram scoring
>>>>
>>>> 1-gram 2-gram 3-gram 4-gram 5-gram 6-gram 7-gram
>>>> 8-gram 9-gram
>>>>
>>>> ------ ------ ------ ------ ------ ------ ------
>>>> ------ ------
>>>>
>>>> NIST: 4.4488 5.0043 5.0520 5.0564 5.0564 5.0564 5.0564
>>>> 5.0564 5.0564 "Google"
>>>>
>>>>
>>>> BLEU: 0.5415 0.4012 0.3044 0.2318 0.1784 0.1362 0.1031
>>>> 0.0754 0.0493 "Google"
>>>>
>>>> And at https://github.com/moses-smt/mosesdecoder/blob/master/scr
>>>> ipts/generic/mteval-v13a.pl#L823, it tries to calculate the cumulative
>>>> score by accumulate the individual ngram precisions and at each order of
>>>> ngram add to it and do a normalization before calculating the cumulative
>>>> score for each order of nrgram.
>>>>
>>>> The question is why does it add the brevity penalty? (i.e. $len_score)
>>>>
>>>> Also, is this score discussed in any paper?
>>>>
>>>> Thanks in advance for the clarifications!
>>>>
>>>> Regards,
>>>> Nat
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161026/8f2c4730/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 120, Issue 30
**********************************************

Tweet