Moses-support Digest, Vol 109, Issue 66

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: different versions of moses yielding different
translations (Barry Haddow)


----------------------------------------------------------------------

Message: 1
Date: Thu, 26 Nov 2015 11:16:01 +0000
From: Barry Haddow <bhaddow@inf.ed.ac.uk>
Subject: Re: [Moses-support] different versions of moses yielding
different translations
To: Vito Mandorino <vito.mandorino@linguacustodia.com>, moses-support
<moses-support@mit.edu>
Message-ID: <5656E9F1.4000803@inf.ed.ac.uk>
Content-Type: text/plain; charset="utf-8"

Hi Vito

The tcmalloc message is normal.

Are you absolutely sure you are using the same model (and same pre- and
post-processing)? A difference of 5 or 14 bleu should be quite visible
in the output. What do the outputs look like?

cheers - Barry

On 26/11/15 09:58, Vito Mandorino wrote:
> Hi Barry,
>
> actually with OnDisk table there is virtually no difference (0.2
> average difference no matter if re-tuning has been done or not).
> With compact Phrase-table however the difference is larger. The latest
> test this morning yields a loss of 14 Bleu score points without
> re-tuning. I don't know which could be the cause.
> Sometimes there is this message on loading the phrase-tables
> tcmalloc: large alloc 1149427712 bytes == 0x28a54000 @
>
> After re-tuning however the difference in BLEU score gets smaller even
> with compact phrase-table.
>
> Best regards,
> Vito
>
> 2015-11-25 21:23 GMT+01:00 Barry Haddow <bhaddow@inf.ed.ac.uk
> <mailto:bhaddow@inf.ed.ac.uk>>:
>
> Hi Vito
>
> The 0.2 difference is after retuning? That's normal then.
>
> But a difference of 5 bleu without retuning suggests a bug. Did
> you say that this only happens with PhraseDictionaryMultiModel?
>
> cheers - Barry
>
>
> On 25/11/15 13:53, Vito Mandorino wrote:
>> Thank you. In our tests it seems that with the OnDisk table the
>> quality is basically the same between the two versions of Moses
>> (average 0.2 difference in score Bleu) but for the
>> CompactPhraseTable the difference is larger (2 points Bleu loss
>> in average after re-tuning with the new version of Moses, and
>> more than 5 points Bleu in average without re-tuning).
>> Do you think a better quality would be obtained by running a
>> complete re-training of the model with the new version of Moses?
>>
>>
>> Best regards,
>> Vito
>>
>> 2015-11-24 16:31 GMT+01:00 Hieu Hoang <hieuhoang@gmail.com
>> <mailto:hieuhoang@gmail.com>>:
>>
>> There was a change in the underlying datastructure for
>> stacks, it changed from std::set (ordered) to
>> boost::unordered_set.
>> https://github.com/moses-smt/mosesdecoder/commit/6b182ee5e987a5b2823aea7eaaa7ef0457c6a30d
>> This got some speed gains
>>
>> 1 5 10 15 20 25 30 35
>> 56 real4m57.795s real1m19.005s real0m51.636s
>> real0m49.624s real0m49.869s real0m52.475s
>> real0m53.806s real 0m54.957s
>> 13/10 baseline user4m41.255s user5m45.086s user6m34.053s
>> user8m12.430s user8m10.667s user8m16.486s
>> user8m10.592s user 8m13.859s
>>
>> sys0m16.514s sys0m35.494s sys0m54.513s sys1m10.643s
>> sys1m18.449s sys1m21.738s sys1m23.133s sys 1m25.048s
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 57 real4m41.148s real1m16.002s real0m50.747s
>> real0m48.711s real0m49.130s real0m51.473s
>> real0m53.141s real 0m54.513s
>> (56) + unordered set stack user4m23.968s user5m30.356s
>> user6m26.167s user7m39.286s user7m56.229s user7m52.669s
>> user7m56.978s user 7m56.216s
>>
>> sys0m17.231s sys0m35.063s sys0m54.081s sys1m10.137s
>> sys1m17.194s sys1m22.912s sys1m25.948s sys 1m26.247s
>>
>>
>> However, the hypotheses are now added to the stack in a
>> different order so there will be slight differences in results
>>
>>
>> On 24/11/2015 13:53, Vito Mandorino wrote:
>>> Hi,
>>>
>>> in some of our tests a recent version of Moses (pulled from
>>> github last week) and an older one do not give the same
>>> translations on the same source segment (with the same
>>> moses.ini).
>>> Here is the 5-best list for the translation of 'test' with
>>> the last week version:
>>>
>>> 0 ||| test ||| LexicalReordering0= -1.1969 0 0 0 0 0
>>> Distortion0= 0 LM0= -51.1788 WordPenalty0= -1
>>> PhrasePenalty0= 1 PhraseDictionaryMultiModel0= -3.03811
>>> -2.5834 -2.08503 -1.83075 ||| -1.27754
>>> 0 ||| testing ||| LexicalReordering0= 0 0 0 0 0 0
>>> Distortion0= 0 LM0= -35.1495 WordPenalty0= -1
>>> PhrasePenalty0= 1 PhraseDictionaryMultiModel0= -5.21045
>>> -5.04877 -4.71131 -4.66382 ||| -1.70337
>>> 0 ||| funds ||| LexicalReordering0= -3.1355 0 0 0 0 0
>>> Distortion0= 0 LM0= -11.3753 WordPenalty0= -1
>>> PhrasePenalty0= 1 PhraseDictionaryMultiModel0= -10.8209
>>> -10.6835 -5.14555 -5.73388 ||| -1.77009
>>> 0 ||| known as a ||| LexicalReordering0= -3.1355 0 0 0 0 0
>>> Distortion0= 0 LM0= -58.8877 WordPenalty0= -3
>>> PhrasePenalty0= 1 PhraseDictionaryMultiModel0= -4.42285
>>> -11.9339 -5.14555 -18.0392 ||| -1.89152
>>> 0 ||| as a ||| LexicalReordering0= -3.1355 0 0 0 0 0
>>> Distortion0= 0 LM0= -35.5353 WordPenalty0= -2
>>> PhrasePenalty0= 1 PhraseDictionaryMultiModel0= -9.34698
>>> -11.9339 -5.14555 -9.14874 ||| -1.89159
>>>
>>> and with the older version of Moses:
>>>
>>> 0 ||| funds ||| LexicalReordering0= -3.1355 0 0 0 0 0
>>> Distortion0= 0 LM0= -11.3753 WordPenalty0= -1
>>> PhrasePenalty0= 1 PhraseDictionaryMultiModel0= -2.52548
>>> -2.52544 -2.45544 -2.48609 ||| -0.815668
>>> 0 ||| as a ||| LexicalReordering0= -3.1355 0 0 0 0 0
>>> Distortion0= 0 LM0= -35.5353 WordPenalty0= -2
>>> PhrasePenalty0= 1 PhraseDictionaryMultiModel0= -2.52464
>>> -2.52565 -2.45544 -2.5244 ||| -0.953799
>>> 0 ||| as ||| LexicalReordering0= -3.1355 0 0 0 0 0
>>> Distortion0= 0 LM0= -34.1633 WordPenalty0= -1
>>> PhrasePenalty0= 1 PhraseDictionaryMultiModel0= -2.5256
>>> -2.52565 -2.45544 -2.48609 ||| -1.07254
>>> 0 ||| known as a ||| LexicalReordering0= -3.1355 0 0 0 0 0
>>> Distortion0= 0 LM0= -58.8877 WordPenalty0= -3
>>> PhrasePenalty0= 1 PhraseDictionaryMultiModel0= -2.38597
>>> -2.52565 -2.45544 -2.52573 ||| -1.07536
>>> 0 ||| is known as a ||| LexicalReordering0= -3.1355 0 0 0 0
>>> 0 Distortion0= 0 LM0= -80.8518 WordPenalty0= -4
>>> PhrasePenalty0= 1 PhraseDictionaryMultiModel0= -2.37158
>>> -2.52565 -2.45544 -2.52573 ||| -1.18753
>>>
>>> This looks very strange. The only difference is in the
>>> phrase-table scores. Do you have any idea of what is going
>>> on? The only possibility which come to mind is maybe a
>>> different handling of the PhraseDictionaryMultiModel feature.
>>> The moses.ini is in attachment.
>>>
>>> Best regards,
>>>
>>> Vito
>>>
>>>
>>>
>>> --
>>> *M**. Vito MANDORINO -- Chief Scientist*
>>>
>>> Description : Description : lingua_custodia_final full logo
>>>
>>> */The Translation Trustee/*
>>>
>>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>>>
>>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
>>> <tel:%2B33%206%2084%2065%2068%2089>*
>>>
>>> *Email :****<mailto:massinissa.ahmim@linguacustodia.com>vito.mandorino@linguacustodia.com
>>> <mailto:vito.mandorino@linguacustodia.com>***
>>>
>>> *Website :****www.linguacustodia.com
>>> <http://www.linguacustodia.com> -
>>> www.thetranslationtrustee.com
>>> <http://www.thetranslationtrustee.com/>*
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> --
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>>
>>
>>
>> --
>> *M**. Vito MANDORINO -- Chief Scientist*
>>
>> Description : Description : lingua_custodia_final full logo
>>
>> */The Translation Trustee/*
>>
>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>>
>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
>> <tel:%2B33%206%2084%2065%2068%2089>*
>>
>> *Email :****<mailto:massinissa.ahmim@linguacustodia.com>vito.mandorino@linguacustodia.com
>> <mailto:vito.mandorino@linguacustodia.com>***
>>
>> *Website :****www.linguacustodia.com
>> <http://www.linguacustodia.com/> - www.thetranslationtrustee.com
>> <http://www.thetranslationtrustee.com/>*
>>
>>
>>
>> This body part will be downloaded on demand.
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
>
>
> --
> *M**. Vito MANDORINO -- Chief Scientist*
>
> Description : Description : lingua_custodia_final full logo
>
> */The Translation Trustee/*
>
> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>
> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89*
>
> *Email :****vito.mandorino@linguacustodia.com
> <mailto:massinissa.ahmim@linguacustodia.com>***
>
> *Website :****www.linguacustodia.com
> <http://www.linguacustodia.com/> - www.thetranslationtrustee.com
> <http://www.thetranslationtrustee.com/>*
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151126/991bc249/attachment.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 4421 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20151126/991bc249/attachment.jpe
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151126/991bc249/attachment.pl

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 109, Issue 66
**********************************************

0 Response to "Moses-support Digest, Vol 109, Issue 66"

Post a Comment