Moses-support Digest, Vol 89, Issue 57

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Loading a binary rule table (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Sat, 22 Mar 2014 10:40:11 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Loading a binary rule table
To: Massinissa Ahmim <massinissa.ahmim@linguacustodia.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAEKMkbi2_gG_kCNqsxQa4_p9-gQJvpZhzNCKsSowGc9K1EBMOQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

did you binarize the LM and what did you binarize it with? KenLM or IRSTLM?

at the moment, it's set to SRILM, which doesn't have a binary format



On 22 March 2014 10:10, Massinissa Ahmim <
massinissa.ahmim@linguacustodia.com> wrote:

> Hi Hieu,
>
> I used the same binarisation command as the one stated here-above
>
> I get this error when I try to translate something like : echo 'ceci est
> un test' | /mosesdecoder/bin/moses_chart -f moses.ini
>
> Here is my moses.ini :
>
> #########################
> ### MOSES CONFIG FILE ###
> #########################
>
> # input factors
> [input-factors]
> 0
>
> # mapping steps
> [mapping]
> 0 T 0
> 1 T 1
>
> # translation tables: table type (hierarchical(0), textual (0), binary
> (1)), source-factors, target-factors, number of scores, file
> # OLD FORMAT is still handled for back-compatibility
> # OLD FORMAT translation tables: source-factors, target-factors, number of
> scores, file
> # OLD FORMAT a binary table type (1) is assumed
> [ttable-file]
> 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table
> 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>
> # no generation models, no generation-file section
>
> # language models: type(srilm/irstlm), factors, order, file
> [lmodel-file]
> 0 0 5 /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>
>
> # limit on how many phrase translations e for each phrase f are loaded
> # 0 = all elements loaded
> [ttable-limit]
> 20
>
>
> # language model weights
> [weight-l]
> 0.5000
>
>
> # translation model weights
> [weight-t]
> 0.20
> 0.20
> 0.20
> 0.20
> 0.20
> 1.0
>
> # no generation models, no weight-generation section
>
> # word penalty
> [weight-w]
> -1
>
> [cube-pruning-pop-limit]
> 1000
>
> [non-terminals]
> X
>
> [search-algorithm]
> 3
>
> [inputtype]
> 3
>
> [max-chart-span]
> 20
> 1000
>
>
>
> 2014-03-21 20:49 GMT+01:00 Hieu Hoang <Hieu.Hoang@ed.ac.uk>:
>
> i've never seen that error before. Did you change anything else, other
>> than rebinarizing the phrase-table? What is the exact command you ran when
>> you got this error? Can I have a look at your moses.ini file
>>
>>
>> On 21 March 2014 18:32, Massinissa Ahmim <
>> massinissa.ahmim@linguacustodia.com> wrote:
>>
>>> Thank you Hieu,
>>>
>>> I've binarised my rule-table again and it seems to be successfully
>>> loaded but I'm getting this message now :
>>>
>>> Reading /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table
>>>
>>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>>>
>>> ****************************************************************************************************
>>> Exception: <fd:3>: Is a directory from zlib
>>>
>>> What can I do?
>>>
>>> Thanks !
>>>
>>> Regards
>>>
>>> Massinissa
>>>
>>>
>>> 2014-03-21 11:18 GMT+01:00 Hieu Hoang <Hieu.Hoang@ed.ac.uk>:
>>>
>>> everything seems to be ok. Are you sure the server you are running on
>>>> has access to the directory
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>>
>>>> Also, you should consider updating your version of Moses to the
>>>> current github version, or version 2.1.1
>>>>
>>>>
>>>> On 20 March 2014 11:20, Massinissa Ahmim <
>>>> massinissa.ahmim@linguacustodia.com> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I've managed to train a hierarchical model using the following command
>>>>> :
>>>>>
>>>>> nohup /mosesdecoder/scripts/training/train-model.perl --hierarchical
>>>>> --glue-grammar --score-options="--GoodTuring" -root-dir
>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT -corpus
>>>>> /disque2/Preparation/backoff/PSCT.tok.uni.low -f fr -e en -lm
>>>>> 0:5:/disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>>>> -external-bin-dir /root/external-bin-dir/ -mgiza -mgiza-cpus 30 >&
>>>>> training.out &
>>>>>
>>>>> The resulting model works great but it is very slow.
>>>>>
>>>>> So I used CreatOnDiskPt to binarise the rule table as follows :
>>>>>
>>>>> /home/Moses/mosesdecoder/bin/CreateOnDiskPt 1 1 5 20 2
>>>>> model/rule-table.gz rules-table
>>>>>
>>>>> This outputs the following files
>>>>>
>>>>> -rw-r--r-- 1 root root 85 20 mars 11:35 Misc.dat
>>>>> -rw-r--r-- 1 root root 781509322 20 mars 11:35 Source.dat
>>>>> -rw-r--r-- 1 root root 1758588137 20 mars 11:35 TargetColl.dat
>>>>> -rw-r--r-- 1 root root 1555353459 20 mars 11:35 TargetInd.dat
>>>>> -rw-r--r-- 1 root root 400714 20 mars 11:35 Vocab.dat
>>>>>
>>>>> and I've updated my moses.ini as follows :
>>>>>
>>>>> 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>>> 6 0 0 1
>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>>>>
>>>>> But as I try to use it, I'm getting this :
>>>>>
>>>>> Defined parameters (per moses.ini or switch):
>>>>> config: moses.ini
>>>>> cube-pruning-pop-limit: 1000
>>>>> input-factors: 0
>>>>> inputtype: 3
>>>>> lmodel-file: 0 0 5
>>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>>>> mapping: 0 T 0 1 T 1
>>>>> max-chart-span: 20 1000
>>>>> non-terminals: X
>>>>> search-algorithm: 3
>>>>> ttable-file: 6 0 0 1
>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table 6 0 0 1
>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>>>> ttable-limit: 20
>>>>> weight-l: 0.5000
>>>>> weight-t: 0.20 0.20 0.20 0.20 0.20 1.0
>>>>> weight-w: -1
>>>>> /mosesdecoder/bin
>>>>> ScoreProducer: WordPenalty start: 0 end: 1
>>>>> ScoreProducer: !UnknownWordPenalty start: 1 end: 2
>>>>> Loading lexical distortion models...have 0 models
>>>>> Start loading LanguageModel
>>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm : [0.000] seconds
>>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm: line 80492:
>>>>> reached EOF before \end\
>>>>> ScoreProducer: LM start: 2 end: 3
>>>>> Finished loading LanguageModels : [0.056] seconds
>>>>> Using uniform ttable-limit of 20 for all translation tables.
>>>>> Start loading PhraseTable
>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table : [0.056]
>>>>> seconds
>>>>> filePath: /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>>> ScoreProducer: PhraseModel start: 3 end: 4
>>>>> Start loading PhraseTable
>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar :
>>>>> [0.056] seconds
>>>>> filePath:
>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>>>> ScoreProducer: PhraseModel:2 start: 4 end: 5
>>>>> Finished loading phrase tables : [0.056] seconds
>>>>> max-chart-span: 20
>>>>> max-chart-span: 1000
>>>>> Start loading phrase table from
>>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table : [0.056]
>>>>> seconds
>>>>> Can't read /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>>>
>>>>> So I wondering if I did something wrong with my training
>>>>> command/binarisation or with the parameters in the moses.ini
>>>>>
>>>>>
>>>>> Many thanks
>>>>>
>>>>> Regads
>>>>>
>>>>> MA
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> [image: Description : Description : lingua_custodia_final full logo]
>>>>>
>>>>> *The Translation Trustee*
>>>>>
>>>>> *1, Place Charles de Gaulle*
>>>>>
>>>>> *78180 Montigny-le-Bretonneux*
>>>>>
>>>>> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>>>>>
>>>>> *Email :* *massinissa.ahmim@linguacustodia.com
>>>>> <massinissa.ahmim@linguacustodia.com>*
>>>>>
>>>>> *Website :* *www.linguacustodia.com
>>>>> <http://www.linguacustodia.com/> - www.thetranslationtrustee.com
>>>>> <http://www.thetranslationtrustee.com>*
>>>>>
>>>>> ? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.
>>>>>
>>>>> Please do not print this email unless it is absolutely necessary. Spread
>>>>> environmental awareness.
>>>>>
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Hieu Hoang
>>>> Research Associate
>>>> University of Edinburgh
>>>> http://www.hoang.co.uk/hieu
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> [image: Description : Description : lingua_custodia_final full logo]
>>>
>>> *The Translation Trustee*
>>>
>>> *1, Place Charles de Gaulle*
>>>
>>> *78180 Montigny-le-Bretonneux*
>>>
>>> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>>>
>>> *Email :* *massinissa.ahmim@linguacustodia.com
>>> <massinissa.ahmim@linguacustodia.com>*
>>>
>>> *Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> -
>>> www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*
>>>
>>> ? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.
>>>
>>> Please do not print this email unless it is absolutely necessary. Spread
>>> environmental awareness.
>>>
>>
>>
>>
>> --
>> Hieu Hoang
>> Research Associate
>> University of Edinburgh
>> http://www.hoang.co.uk/hieu
>>
>>
>
>
> --
>
> [image: Description : Description : lingua_custodia_final full logo]
>
> *The Translation Trustee*
>
> *1, Place Charles de Gaulle*
>
> *78180 Montigny-le-Bretonneux*
>
> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>
> *Email :* *massinissa.ahmim@linguacustodia.com
> <massinissa.ahmim@linguacustodia.com>*
>
> *Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> -
> www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*
>
> ? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.
>
> Please do not print this email unless it is absolutely necessary. Spread
> environmental awareness.
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140322/f1b2fa70/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 4421 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140322/f1b2fa70/attachment.jpg

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 89, Issue 57
*********************************************

Tweet