Moses-support Digest, Vol 119, Issue 47

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: differences between moses and moses2 output (Vito Mandorino)
2. Re: differences between moses and moses2 output (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Thu, 29 Sep 2016 16:19:49 +0200
From: Vito Mandorino <vito.mandorino@linguacustodia.com>
Subject: Re: [Moses-support] differences between moses and moses2
output
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CA+8mSmG6Mrisgg3_rB4Y5vxm+oSuaL6aAZBk3m3fQNdwTyVBLQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Ok thank you, I'll check that. Do you know how to perform the integration?
Juxtaposing the 4 phrase-table scores and the 6 reordering scores before
calling CreateProbingPT2 would be enough?

(I have used the CreateProbingPT2 binary and not CreateProbingPT so far)

Vito

2016-09-29 16:07 GMT+02:00 Hieu Hoang <hieuhoang@gmail.com>:

> you will get another big speedup fromm integrating the lexro into the pt
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 29 September 2016 at 15:03, Vito Mandorino <vito.mandorino@
> linguacustodia.com> wrote:
>
>> Yes the model includes a lexicalised reordering model but is not
>> integrated into the probingPT. The size of the LM is 1.8G.
>>
>> 2016-09-29 15:59 GMT+02:00 Hieu Hoang <hieuhoang@gmail.com>:
>>
>>> ps. how big is your LM?
>>>
>>> Hieu Hoang
>>> http://www.hoang.co.uk/hieu
>>>
>>> On 29 September 2016 at 14:58, Hieu Hoang <hieuhoang@gmail.com> wrote:
>>>
>>>> great, thanks. Do you use the lexicalised reordering model, and is it
>>>> integrated into the phrase-table in Moses2?
>>>>
>>>> There is latency in communicating with the server. As Moses2 is much
>>>> faster now, the client can't feed it fast enough. You should see that
>>>> moses2 command line will max out the CPU, whereas the server won't. I'm
>>>> thinking of extending the server to processing multiple sentences at a time
>>>> to speed it up
>>>>
>>>> Hieu Hoang
>>>> http://www.hoang.co.uk/hieu
>>>>
>>>> On 29 September 2016 at 14:49, Vito Mandorino <
>>>> vito.mandorino@linguacustodia.com> wrote:
>>>>
>>>>> Yes, here are some data:
>>>>>
>>>>> Average source sentence length: 29 tokens
>>>>> Phrase-table size, probingPT: 11G
>>>>> Phrase-table size, compact phrase-table: 2.1G
>>>>>
>>>>> Translation time Moses2 with 32 threads: 1m36.511s
>>>>> Translation time Moses with 32 threads: 6m14.248s
>>>>> Translation time Moses2 with 32 threads in server mode: 16m30.137s
>>>>> Translation time Moses with 32 threads in server mode: 62m33.208s
>>>>>
>>>>> Ram consumption during decoding: 4G for Moses2, 5G for Moses
>>>>>
>>>>> So Moses2 is 4 times faster, and 3 times faster in server mode.
>>>>>
>>>>> Do you know why in server mode the speed is so much slower with
>>>>> respect to batch mode, for both Moses and Moses2?
>>>>>
>>>>> Best regards,
>>>>> Vito
>>>>>
>>>>> 2016-09-28 18:52 GMT+02:00 Hieu Hoang <hieuhoang@gmail.com>:
>>>>>
>>>>>> cool. do you have any indications of speed, especially when using
>>>>>> multiple threads? model sizes and average input sentence length are also
>>>>>> relevant.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *M**. Vito MANDORINO -- Chief Scientist*
>>>>>
>>>>>
>>>>> [image: Description : Description : lingua_custodia_final full logo]
>>>>>
>>>>> *The Translation Trustee*
>>>>>
>>>>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>>>>>
>>>>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
>>>>> <%2B33%206%2084%2065%2068%2089>*
>>>>>
>>>>> *Email :* *vito.mandorino@linguacustodia.com
>>>>> <massinissa.ahmim@linguacustodia.com>*
>>>>>
>>>>> *Website :*
>>>>> *www.linguacustodia.finance <http://www.linguacustodia.com/>*
>>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> *M**. Vito MANDORINO -- Chief Scientist*
>>
>>
>> [image: Description : Description : lingua_custodia_final full logo]
>>
>> *The Translation Trustee*
>>
>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>>
>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
>> <%2B33%206%2084%2065%2068%2089>*
>>
>> *Email :* *vito.mandorino@linguacustodia.com
>> <massinissa.ahmim@linguacustodia.com>*
>>
>> *Website :*
>> *www.linguacustodia.finance <http://www.linguacustodia.com/>*
>>
>
>


--
*M**. Vito MANDORINO -- Chief Scientist*


[image: Description : Description : lingua_custodia_final full logo]

*The Translation Trustee*

*1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*

*Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89*

*Email :* *vito.mandorino@linguacustodia.com
<massinissa.ahmim@linguacustodia.com>*

*Website :*
*www.linguacustodia.finance <http://www.linguacustodia.com/>*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160929/d9831a4e/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 4421 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20160929/d9831a4e/attachment-0001.jpg

------------------------------

Message: 2
Date: Thu, 29 Sep 2016 15:25:46 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] differences between moses and moses2
output
To: Vito Mandorino <vito.mandorino@linguacustodia.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbg2dpBv+h=5bbG05vrK=vPdVwCsVT=49azcJ6Pn8+BeYA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

use the script:
scripts/generic/binarize4moses2.perl
It takes as input the (text) phrase-table and the (text) lexro model. It
will give you the probing pt which contains the info for both.

To use this script, Moses MUST be compiled with the flag --with-cmph. Also,
the program in contrib/sigtest-filter MUST have been successfully compiled.



Hieu Hoang
http://www.hoang.co.uk/hieu

On 29 September 2016 at 15:19, Vito Mandorino <
vito.mandorino@linguacustodia.com> wrote:

> Ok thank you, I'll check that. Do you know how to perform the integration?
> Juxtaposing the 4 phrase-table scores and the 6 reordering scores before
> calling CreateProbingPT2 would be enough?
>
> (I have used the CreateProbingPT2 binary and not CreateProbingPT so far)
>
> Vito
>
> 2016-09-29 16:07 GMT+02:00 Hieu Hoang <hieuhoang@gmail.com>:
>
>> you will get another big speedup fromm integrating the lexro into the pt
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 29 September 2016 at 15:03, Vito Mandorino <
>> vito.mandorino@linguacustodia.com> wrote:
>>
>>> Yes the model includes a lexicalised reordering model but is not
>>> integrated into the probingPT. The size of the LM is 1.8G.
>>>
>>> 2016-09-29 15:59 GMT+02:00 Hieu Hoang <hieuhoang@gmail.com>:
>>>
>>>> ps. how big is your LM?
>>>>
>>>> Hieu Hoang
>>>> http://www.hoang.co.uk/hieu
>>>>
>>>> On 29 September 2016 at 14:58, Hieu Hoang <hieuhoang@gmail.com> wrote:
>>>>
>>>>> great, thanks. Do you use the lexicalised reordering model, and is it
>>>>> integrated into the phrase-table in Moses2?
>>>>>
>>>>> There is latency in communicating with the server. As Moses2 is much
>>>>> faster now, the client can't feed it fast enough. You should see that
>>>>> moses2 command line will max out the CPU, whereas the server won't. I'm
>>>>> thinking of extending the server to processing multiple sentences at a time
>>>>> to speed it up
>>>>>
>>>>> Hieu Hoang
>>>>> http://www.hoang.co.uk/hieu
>>>>>
>>>>> On 29 September 2016 at 14:49, Vito Mandorino <
>>>>> vito.mandorino@linguacustodia.com> wrote:
>>>>>
>>>>>> Yes, here are some data:
>>>>>>
>>>>>> Average source sentence length: 29 tokens
>>>>>> Phrase-table size, probingPT: 11G
>>>>>> Phrase-table size, compact phrase-table: 2.1G
>>>>>>
>>>>>> Translation time Moses2 with 32 threads: 1m36.511s
>>>>>> Translation time Moses with 32 threads: 6m14.248s
>>>>>> Translation time Moses2 with 32 threads in server mode: 16m30.137s
>>>>>> Translation time Moses with 32 threads in server mode: 62m33.208s
>>>>>>
>>>>>> Ram consumption during decoding: 4G for Moses2, 5G for Moses
>>>>>>
>>>>>> So Moses2 is 4 times faster, and 3 times faster in server mode.
>>>>>>
>>>>>> Do you know why in server mode the speed is so much slower with
>>>>>> respect to batch mode, for both Moses and Moses2?
>>>>>>
>>>>>> Best regards,
>>>>>> Vito
>>>>>>
>>>>>> 2016-09-28 18:52 GMT+02:00 Hieu Hoang <hieuhoang@gmail.com>:
>>>>>>
>>>>>>> cool. do you have any indications of speed, especially when using
>>>>>>> multiple threads? model sizes and average input sentence length are also
>>>>>>> relevant.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *M**. Vito MANDORINO -- Chief Scientist*
>>>>>>
>>>>>>
>>>>>> [image: Description : Description : lingua_custodia_final full logo]
>>>>>>
>>>>>> *The Translation Trustee*
>>>>>>
>>>>>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>>>>>>
>>>>>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
>>>>>> <%2B33%206%2084%2065%2068%2089>*
>>>>>>
>>>>>> *Email :* *vito.mandorino@linguacustodia.com
>>>>>> <massinissa.ahmim@linguacustodia.com>*
>>>>>>
>>>>>> *Website :*
>>>>>> *www.linguacustodia.finance <http://www.linguacustodia.com/>*
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> *M**. Vito MANDORINO -- Chief Scientist*
>>>
>>>
>>> [image: Description : Description : lingua_custodia_final full logo]
>>>
>>> *The Translation Trustee*
>>>
>>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>>>
>>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
>>> <%2B33%206%2084%2065%2068%2089>*
>>>
>>> *Email :* *vito.mandorino@linguacustodia.com
>>> <massinissa.ahmim@linguacustodia.com>*
>>>
>>> *Website :*
>>> *www.linguacustodia.finance <http://www.linguacustodia.com/>*
>>>
>>
>>
>
>
> --
> *M**. Vito MANDORINO -- Chief Scientist*
>
>
> [image: Description : Description : lingua_custodia_final full logo]
>
> *The Translation Trustee*
>
> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>
> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
> <%2B33%206%2084%2065%2068%2089>*
>
> *Email :* *vito.mandorino@linguacustodia.com
> <massinissa.ahmim@linguacustodia.com>*
>
> *Website :*
> *www.linguacustodia.finance <http://www.linguacustodia.com/>*
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160929/f7790c5a/attachment.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 4421 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20160929/f7790c5a/attachment.jpg

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 119, Issue 47
**********************************************

Tweet