Moses-support Digest, Vol 120, Issue 4

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: differences between moses and moses2 output (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Mon, 3 Oct 2016 16:24:02 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] differences between moses and moses2
output
To: Vito Mandorino <vito.mandorino@linguacustodia.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbhA4v4VacY3Cu8sF+WertY0z411rJMASJc54jnOdU4S4w@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

if guessed you ran binarize4moses2.perl so please give me the phrase-table
and reordering model, and the exact command you ran

Hieu Hoang
http://www.hoang.co.uk/hieu

On 3 October 2016 at 15:20, Vito Mandorino <
vito.mandorino@linguacustodia.com> wrote:

> I have managed to replicate the issue on a smaller corpus. Do you need the
> training corpus, the tables (phrase- and reordering-), or all of them?
>
> Vito
>
> 2016-09-30 13:30 GMT+02:00 Hieu Hoang <hieuhoang@gmail.com>:
>
>> wow, that looks like a seriously problem.
>>
>> I've not seen this before. If you can make the data file available for
>> download, it would be much appreciated
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 30 September 2016 at 09:11, Vito Mandorino <
>> vito.mandorino@linguacustodia.com> wrote:
>>
>>> I tried the following command:
>>>
>>> perl /home/Moses/mosesdecoder/scripts/generic/binarize4moses2.perl
>>> --phrase-table=/home/vito/phrase-table.sorted
>>> --lex-ro=/home/vito/reordering-table.sorted
>>> --output-dir=/home/vito/integrated_phrase-reordering/ --num-lex-scores=6
>>>
>>> but it gets stuck with the following message:
>>>
>>> Reading phrase table finished, writing remaining files to disk.
>>> terminate called after throwing an instance of
>>> 'util::ProbingSizeException'
>>> what(): ./util/probing_hash_table.hh:150 in
>>> util::ProbingHashTable<EntryT, HashT, EqualT, ModT>::Entry*
>>> util::ProbingHashTable<EntryT, HashT, EqualT, ModT>::Insert(const T&) [with
>>> T = Moses2::Entry; EntryT = Moses2::Entry; HashT = boost::hash<long
>>> unsigned int>; EqualT = std::equal_to<long unsigned int>; ModT =
>>> util::DivMod; util::ProbingHashTable<EntryT, HashT, EqualT,
>>> ModT>::MutableIterator = Moses2::Entry*; util::ProbingHashTable<EntryT,
>>> HashT, EqualT, ModT>::Entry = Moses2::Entry] threw ProbingSizeException
>>> because `++entries_ >= buckets_'.
>>> Hash table with 1 buckets is full.
>>>
>>>
>>>
>>>
>>> 2016-09-29 16:25 GMT+02:00 Hieu Hoang <hieuhoang@gmail.com>:
>>>
>>>> use the script:
>>>> scripts/generic/binarize4moses2.perl
>>>> It takes as input the (text) phrase-table and the (text) lexro model.
>>>> It will give you the probing pt which contains the info for both.
>>>>
>>>> To use this script, Moses MUST be compiled with the flag --with-cmph.
>>>> Also, the program in contrib/sigtest-filter MUST have been successfully
>>>> compiled.
>>>>
>>>>
>>>>
>>>> Hieu Hoang
>>>> http://www.hoang.co.uk/hieu
>>>>
>>>> On 29 September 2016 at 15:19, Vito Mandorino <
>>>> vito.mandorino@linguacustodia.com> wrote:
>>>>
>>>>> Ok thank you, I'll check that. Do you know how to perform the
>>>>> integration? Juxtaposing the 4 phrase-table scores and the 6 reordering
>>>>> scores before calling CreateProbingPT2 would be enough?
>>>>>
>>>>> (I have used the CreateProbingPT2 binary and not CreateProbingPT so
>>>>> far)
>>>>>
>>>>> Vito
>>>>>
>>>>> 2016-09-29 16:07 GMT+02:00 Hieu Hoang <hieuhoang@gmail.com>:
>>>>>
>>>>>> you will get another big speedup fromm integrating the lexro into the
>>>>>> pt
>>>>>>
>>>>>> Hieu Hoang
>>>>>> http://www.hoang.co.uk/hieu
>>>>>>
>>>>>> On 29 September 2016 at 15:03, Vito Mandorino <
>>>>>> vito.mandorino@linguacustodia.com> wrote:
>>>>>>
>>>>>>> Yes the model includes a lexicalised reordering model but is not
>>>>>>> integrated into the probingPT. The size of the LM is 1.8G.
>>>>>>>
>>>>>>> 2016-09-29 15:59 GMT+02:00 Hieu Hoang <hieuhoang@gmail.com>:
>>>>>>>
>>>>>>>> ps. how big is your LM?
>>>>>>>>
>>>>>>>> Hieu Hoang
>>>>>>>> http://www.hoang.co.uk/hieu
>>>>>>>>
>>>>>>>> On 29 September 2016 at 14:58, Hieu Hoang <hieuhoang@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> great, thanks. Do you use the lexicalised reordering model, and is
>>>>>>>>> it integrated into the phrase-table in Moses2?
>>>>>>>>>
>>>>>>>>> There is latency in communicating with the server. As Moses2 is
>>>>>>>>> much faster now, the client can't feed it fast enough. You should see that
>>>>>>>>> moses2 command line will max out the CPU, whereas the server won't. I'm
>>>>>>>>> thinking of extending the server to processing multiple sentences at a time
>>>>>>>>> to speed it up
>>>>>>>>>
>>>>>>>>> Hieu Hoang
>>>>>>>>> http://www.hoang.co.uk/hieu
>>>>>>>>>
>>>>>>>>> On 29 September 2016 at 14:49, Vito Mandorino <
>>>>>>>>> vito.mandorino@linguacustodia.com> wrote:
>>>>>>>>>
>>>>>>>>>> Yes, here are some data:
>>>>>>>>>>
>>>>>>>>>> Average source sentence length: 29 tokens
>>>>>>>>>> Phrase-table size, probingPT: 11G
>>>>>>>>>> Phrase-table size, compact phrase-table: 2.1G
>>>>>>>>>>
>>>>>>>>>> Translation time Moses2 with 32 threads: 1m36.511s
>>>>>>>>>> Translation time Moses with 32 threads: 6m14.248s
>>>>>>>>>> Translation time Moses2 with 32 threads in server mode: 16m30.137s
>>>>>>>>>> Translation time Moses with 32 threads in server mode: 62m33.208s
>>>>>>>>>>
>>>>>>>>>> Ram consumption during decoding: 4G for Moses2, 5G for Moses
>>>>>>>>>>
>>>>>>>>>> So Moses2 is 4 times faster, and 3 times faster in server mode.
>>>>>>>>>>
>>>>>>>>>> Do you know why in server mode the speed is so much slower with
>>>>>>>>>> respect to batch mode, for both Moses and Moses2?
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>> Vito
>>>>>>>>>>
>>>>>>>>>> 2016-09-28 18:52 GMT+02:00 Hieu Hoang <hieuhoang@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> cool. do you have any indications of speed, especially when
>>>>>>>>>>> using multiple threads? model sizes and average input sentence length are
>>>>>>>>>>> also relevant.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *M**. Vito MANDORINO -- Chief Scientist*
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [image: Description : Description : lingua_custodia_final full
>>>>>>>>>> logo]
>>>>>>>>>>
>>>>>>>>>> *The Translation Trustee*
>>>>>>>>>>
>>>>>>>>>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>>>>>>>>>>
>>>>>>>>>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
>>>>>>>>>> <%2B33%206%2084%2065%2068%2089>*
>>>>>>>>>>
>>>>>>>>>> *Email :* *vito.mandorino@linguacustodia.com
>>>>>>>>>> <massinissa.ahmim@linguacustodia.com>*
>>>>>>>>>>
>>>>>>>>>> *Website :*
>>>>>>>>>> *www.linguacustodia.finance <http://www.linguacustodia.com/>*
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *M**. Vito MANDORINO -- Chief Scientist*
>>>>>>>
>>>>>>>
>>>>>>> [image: Description : Description : lingua_custodia_final full logo]
>>>>>>>
>>>>>>> *The Translation Trustee*
>>>>>>>
>>>>>>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>>>>>>>
>>>>>>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
>>>>>>> <%2B33%206%2084%2065%2068%2089>*
>>>>>>>
>>>>>>> *Email :* *vito.mandorino@linguacustodia.com
>>>>>>> <massinissa.ahmim@linguacustodia.com>*
>>>>>>>
>>>>>>> *Website :*
>>>>>>> *www.linguacustodia.finance <http://www.linguacustodia.com/>*
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *M**. Vito MANDORINO -- Chief Scientist*
>>>>>
>>>>>
>>>>> [image: Description : Description : lingua_custodia_final full logo]
>>>>>
>>>>> *The Translation Trustee*
>>>>>
>>>>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>>>>>
>>>>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
>>>>> <%2B33%206%2084%2065%2068%2089>*
>>>>>
>>>>> *Email :* *vito.mandorino@linguacustodia.com
>>>>> <massinissa.ahmim@linguacustodia.com>*
>>>>>
>>>>> *Website :*
>>>>> *www.linguacustodia.finance <http://www.linguacustodia.com/>*
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> *M**. Vito MANDORINO -- Chief Scientist*
>>>
>>>
>>> [image: Description : Description : lingua_custodia_final full logo]
>>>
>>> *The Translation Trustee*
>>>
>>> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>>>
>>> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
>>> <%2B33%206%2084%2065%2068%2089>*
>>>
>>> *Email :* *vito.mandorino@linguacustodia.com
>>> <massinissa.ahmim@linguacustodia.com>*
>>>
>>> *Website :*
>>> *www.linguacustodia.finance <http://www.linguacustodia.com/>*
>>>
>>
>>
>
>
> --
> *M**. Vito MANDORINO -- Chief Scientist*
>
>
> [image: Description : Description : lingua_custodia_final full logo]
>
> *The Translation Trustee*
>
> *1, Place Charles de Gaulle, **78180 Montigny-le-Bretonneux*
>
> *Tel : +33 1 30 44 04 23 Mobile : +33 6 84 65 68 89
> <%2B33%206%2084%2065%2068%2089>*
>
> *Email :* *vito.mandorino@linguacustodia.com
> <massinissa.ahmim@linguacustodia.com>*
>
> *Website :*
> *www.linguacustodia.finance <http://www.linguacustodia.com/>*
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161003/b7b2fd1c/attachment.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 4421 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20161003/b7b2fd1c/attachment.jpg

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 120, Issue 4
*********************************************

Tweet