Moses-support Digest, Vol 89, Issue 58

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Loading a binary rule table (Massinissa Ahmim)


----------------------------------------------------------------------

Message: 1
Date: Sat, 22 Mar 2014 12:24:17 +0100
From: Massinissa Ahmim <massinissa.ahmim@linguacustodia.com>
Subject: Re: [Moses-support] Loading a binary rule table
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CANN0mWasSkHf75d=XCbRTwdJCx1kERiu3_YiOwNKLx3Y16=+sA@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi Hieu,

It's actually KenLM I've just updated my moses.ini changing the 0 to 8 but
I'm still getting the same error

Regards

Massinissa


2014-03-22 11:40 GMT+01:00 Hieu Hoang <Hieu.Hoang@ed.ac.uk>:

> did you binarize the LM and what did you binarize it with? KenLM or IRSTLM?
>
> at the moment, it's set to SRILM, which doesn't have a binary format
>
>
>
> On 22 March 2014 10:10, Massinissa Ahmim <
> massinissa.ahmim@linguacustodia.com> wrote:
>
>> Hi Hieu,
>>
>> I used the same binarisation command as the one stated here-above
>>
>> I get this error when I try to translate something like : echo 'ceci est
>> un test' | /mosesdecoder/bin/moses_chart -f moses.ini
>>
>> Here is my moses.ini :
>>
>> #########################
>> ### MOSES CONFIG FILE ###
>> #########################
>>
>> # input factors
>> [input-factors]
>> 0
>>
>> # mapping steps
>> [mapping]
>> 0 T 0
>> 1 T 1
>>
>> # translation tables: table type (hierarchical(0), textual (0), binary
>> (1)), source-factors, target-factors, number of scores, file
>> # OLD FORMAT is still handled for back-compatibility
>> # OLD FORMAT translation tables: source-factors, target-factors, number
>> of scores, file
>> # OLD FORMAT a binary table type (1) is assumed
>> [ttable-file]
>> 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table
>> 6 0 0 1
>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>
>> # no generation models, no generation-file section
>>
>> # language models: type(srilm/irstlm), factors, order, file
>> [lmodel-file]
>> 0 0 5 /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>
>>
>> # limit on how many phrase translations e for each phrase f are loaded
>> # 0 = all elements loaded
>> [ttable-limit]
>> 20
>>
>>
>> # language model weights
>> [weight-l]
>> 0.5000
>>
>>
>> # translation model weights
>> [weight-t]
>> 0.20
>> 0.20
>> 0.20
>> 0.20
>> 0.20
>> 1.0
>>
>> # no generation models, no weight-generation section
>>
>> # word penalty
>> [weight-w]
>> -1
>>
>> [cube-pruning-pop-limit]
>> 1000
>>
>> [non-terminals]
>> X
>>
>> [search-algorithm]
>> 3
>>
>> [inputtype]
>> 3
>>
>> [max-chart-span]
>> 20
>> 1000
>>
>>
>>
>> 2014-03-21 20:49 GMT+01:00 Hieu Hoang <Hieu.Hoang@ed.ac.uk>:
>>
>> i've never seen that error before. Did you change anything else, other
>>> than rebinarizing the phrase-table? What is the exact command you ran when
>>> you got this error? Can I have a look at your moses.ini file
>>>
>>>
>>> On 21 March 2014 18:32, Massinissa Ahmim <
>>> massinissa.ahmim@linguacustodia.com> wrote:
>>>
>>>> Thank you Hieu,
>>>>
>>>> I've binarised my rule-table again and it seems to be successfully
>>>> loaded but I'm getting this message now :
>>>>
>>>> Reading /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table
>>>>
>>>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>>>>
>>>> ****************************************************************************************************
>>>> Exception: <fd:3>: Is a directory from zlib
>>>>
>>>> What can I do?
>>>>
>>>> Thanks !
>>>>
>>>> Regards
>>>>
>>>> Massinissa
>>>>
>>>>
>>>> 2014-03-21 11:18 GMT+01:00 Hieu Hoang <Hieu.Hoang@ed.ac.uk>:
>>>>
>>>> everything seems to be ok. Are you sure the server you are running on
>>>>> has access to the directory
>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>>>
>>>>> Also, you should consider updating your version of Moses to the
>>>>> current github version, or version 2.1.1
>>>>>
>>>>>
>>>>> On 20 March 2014 11:20, Massinissa Ahmim <
>>>>> massinissa.ahmim@linguacustodia.com> wrote:
>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> I've managed to train a hierarchical model using the following
>>>>>> command :
>>>>>>
>>>>>> nohup /mosesdecoder/scripts/training/train-model.perl --hierarchical
>>>>>> --glue-grammar --score-options="--GoodTuring" -root-dir
>>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT -corpus
>>>>>> /disque2/Preparation/backoff/PSCT.tok.uni.low -f fr -e en -lm
>>>>>> 0:5:/disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>>>>> -external-bin-dir /root/external-bin-dir/ -mgiza -mgiza-cpus 30 >&
>>>>>> training.out &
>>>>>>
>>>>>> The resulting model works great but it is very slow.
>>>>>>
>>>>>> So I used CreatOnDiskPt to binarise the rule table as follows :
>>>>>>
>>>>>> /home/Moses/mosesdecoder/bin/CreateOnDiskPt 1 1 5 20 2
>>>>>> model/rule-table.gz rules-table
>>>>>>
>>>>>> This outputs the following files
>>>>>>
>>>>>> -rw-r--r-- 1 root root 85 20 mars 11:35 Misc.dat
>>>>>> -rw-r--r-- 1 root root 781509322 20 mars 11:35 Source.dat
>>>>>> -rw-r--r-- 1 root root 1758588137 20 mars 11:35 TargetColl.dat
>>>>>> -rw-r--r-- 1 root root 1555353459 20 mars 11:35 TargetInd.dat
>>>>>> -rw-r--r-- 1 root root 400714 20 mars 11:35 Vocab.dat
>>>>>>
>>>>>> and I've updated my moses.ini as follows :
>>>>>>
>>>>>> 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>>>> 6 0 0 1
>>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>>>>>
>>>>>> But as I try to use it, I'm getting this :
>>>>>>
>>>>>> Defined parameters (per moses.ini or switch):
>>>>>> config: moses.ini
>>>>>> cube-pruning-pop-limit: 1000
>>>>>> input-factors: 0
>>>>>> inputtype: 3
>>>>>> lmodel-file: 0 0 5
>>>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>>>>> mapping: 0 T 0 1 T 1
>>>>>> max-chart-span: 20 1000
>>>>>> non-terminals: X
>>>>>> search-algorithm: 3
>>>>>> ttable-file: 6 0 0 1
>>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table 6 0 0 1
>>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>>>>> ttable-limit: 20
>>>>>> weight-l: 0.5000
>>>>>> weight-t: 0.20 0.20 0.20 0.20 0.20 1.0
>>>>>> weight-w: -1
>>>>>> /mosesdecoder/bin
>>>>>> ScoreProducer: WordPenalty start: 0 end: 1
>>>>>> ScoreProducer: !UnknownWordPenalty start: 1 end: 2
>>>>>> Loading lexical distortion models...have 0 models
>>>>>> Start loading LanguageModel
>>>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm : [0.000] seconds
>>>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm: line
>>>>>> 80492: reached EOF before \end\
>>>>>> ScoreProducer: LM start: 2 end: 3
>>>>>> Finished loading LanguageModels : [0.056] seconds
>>>>>> Using uniform ttable-limit of 20 for all translation tables.
>>>>>> Start loading PhraseTable
>>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table : [0.056]
>>>>>> seconds
>>>>>> filePath: /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>>>> ScoreProducer: PhraseModel start: 3 end: 4
>>>>>> Start loading PhraseTable
>>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar :
>>>>>> [0.056] seconds
>>>>>> filePath:
>>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>>>>> ScoreProducer: PhraseModel:2 start: 4 end: 5
>>>>>> Finished loading phrase tables : [0.056] seconds
>>>>>> max-chart-span: 20
>>>>>> max-chart-span: 1000
>>>>>> Start loading phrase table from
>>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table : [0.056]
>>>>>> seconds
>>>>>> Can't read /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>>>>
>>>>>> So I wondering if I did something wrong with my training
>>>>>> command/binarisation or with the parameters in the moses.ini
>>>>>>
>>>>>>
>>>>>> Many thanks
>>>>>>
>>>>>> Regads
>>>>>>
>>>>>> MA
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> [image: Description : Description : lingua_custodia_final full logo]
>>>>>>
>>>>>> *The Translation Trustee*
>>>>>>
>>>>>> *1, Place Charles de Gaulle*
>>>>>>
>>>>>> *78180 Montigny-le-Bretonneux*
>>>>>>
>>>>>> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>>>>>>
>>>>>> *Email :* *massinissa.ahmim@linguacustodia.com
>>>>>> <massinissa.ahmim@linguacustodia.com>*
>>>>>>
>>>>>> *Website :* *www.linguacustodia.com
>>>>>> <http://www.linguacustodia.com/> - www.thetranslationtrustee.com
>>>>>> <http://www.thetranslationtrustee.com>*
>>>>>>
>>>>>> ? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.
>>>>>>
>>>>>> Please do not print this email unless it is absolutely necessary. Spread
>>>>>> environmental awareness.
>>>>>>
>>>>>> _______________________________________________
>>>>>> Moses-support mailing list
>>>>>> Moses-support@mit.edu
>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Hieu Hoang
>>>>> Research Associate
>>>>> University of Edinburgh
>>>>> http://www.hoang.co.uk/hieu
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> [image: Description : Description : lingua_custodia_final full logo]
>>>>
>>>> *The Translation Trustee*
>>>>
>>>> *1, Place Charles de Gaulle*
>>>>
>>>> *78180 Montigny-le-Bretonneux*
>>>>
>>>> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>>>>
>>>> *Email :* *massinissa.ahmim@linguacustodia.com
>>>> <massinissa.ahmim@linguacustodia.com>*
>>>>
>>>> *Website :* *www.linguacustodia.com
>>>> <http://www.linguacustodia.com/> - www.thetranslationtrustee.com
>>>> <http://www.thetranslationtrustee.com>*
>>>>
>>>> ? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.
>>>>
>>>> Please do not print this email unless it is absolutely necessary. Spread
>>>> environmental awareness.
>>>>
>>>
>>>
>>>
>>> --
>>> Hieu Hoang
>>> Research Associate
>>> University of Edinburgh
>>> http://www.hoang.co.uk/hieu
>>>
>>>
>>
>>
>> --
>>
>> [image: Description : Description : lingua_custodia_final full logo]
>>
>> *The Translation Trustee*
>>
>> *1, Place Charles de Gaulle*
>>
>> *78180 Montigny-le-Bretonneux*
>>
>> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>>
>> *Email :* *massinissa.ahmim@linguacustodia.com
>> <massinissa.ahmim@linguacustodia.com>*
>>
>> *Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> -
>> www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*
>>
>> ? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.
>>
>> Please do not print this email unless it is absolutely necessary. Spread
>> environmental awareness.
>>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>


--

[image: Description : Description : lingua_custodia_final full logo]

*The Translation Trustee*

*1, Place Charles de Gaulle*

*78180 Montigny-le-Bretonneux*

*Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*

*Email :* *massinissa.ahmim@linguacustodia.com
<massinissa.ahmim@linguacustodia.com>*

*Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> -
www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*

? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.

Please do not print this email unless it is absolutely necessary. Spread
environmental awareness.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140322/f1031e62/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 4421 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140322/f1031e62/attachment.jpg

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 89, Issue 58
*********************************************

Tweet