Moses-support Digest, Vol 89, Issue 65

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Loading a binary rule table (Massinissa Ahmim)


----------------------------------------------------------------------

Message: 1
Date: Mon, 24 Mar 2014 15:51:06 +0100
From: Massinissa Ahmim <massinissa.ahmim@linguacustodia.com>
Subject: Re: [Moses-support] Loading a binary rule table
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CANN0mWb46N7RFK_-gR+cQ=cCK+q5Vo8XVK8CDgzELj34nwg2GQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

It works ! Thank you Hieu !


2014-03-24 15:28 GMT+01:00 Hieu Hoang <Hieu.Hoang@ed.ac.uk>:

> it should be
> 6 0 0 *5* /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
> not
> 6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>
> The glue rule entry shouldn't be changed
>
>
>
> On 24 March 2014 13:38, Massinissa Ahmim <
> massinissa.ahmim@linguacustodia.com> wrote:
>
>> Thank you Hieu and Kenneth by rule table seems to be loaded now but I'm
>> still getting an error
>>
>> ScoreProducer: PhraseModel:2 start: 4 end: 5
>> Finished loading phrase tables : [0.039] seconds
>> max-chart-span: 20
>> max-chart-span: 1000
>> Start loading phrase table from
>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar :
>> [0.039] seconds
>> Start loading text SCFG phrase table. Moses format : [0.039] seconds
>> Reading
>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>
>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>>
>> ****************************************************************************************************
>> Finished loading phrase tables : [0.040] seconds
>> IO from STDOUT/STDIN
>> Created input-output object : [0.040] seconds
>> Check m_dbWrapper.GetMisc("NumScores") == weight.size() failed in
>> moses/src/RuleTable/PhraseDictionaryOnDisk.cpp:63
>> End. : [0.236] seconds
>> user 0.032
>> sys 0.016
>> VmPeak: 69020 kB
>> VmRSS: 26980 kB
>>
>>
>> How can I fixe it ?
>>
>> Many thanks
>>
>> Regards
>>
>>
>> 2014-03-22 19:40 GMT+01:00 Hieu Hoang <Hieu.Hoang@ed.ac.uk>:
>>
>> ah yes, change 6 to 2, but only for the rule-table, not glue rule
>>>
>>>
>>> On 22 March 2014 18:35, Kenneth Heafield <moses@kheafield.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> The error message said
>>>>
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table
>>>>
>>>> is a directory. But IIRC phrase table 6 is "SCFG" and expects a file.
>>>> If you have binarized the phrase table specify a different magic number
>>>> in the moses.ini.
>>>>
>>>> Nothing to do with the language model but, for the record,
>>>> SRILM does
>>>> have a binary format. Just not a memory mapped binary format.
>>>>
>>>> Kenneth
>>>>
>>>> On 03/22/14 04:24, Massinissa Ahmim wrote:
>>>> > Hi Hieu,
>>>> >
>>>> > It's actually KenLM I've just updated my moses.ini changing the 0 to 8
>>>> > but I'm still getting the same error
>>>> >
>>>> > Regards
>>>> >
>>>> > Massinissa
>>>> >
>>>> >
>>>> > 2014-03-22 11:40 GMT+01:00 Hieu Hoang <Hieu.Hoang@ed.ac.uk
>>>> > <mailto:Hieu.Hoang@ed.ac.uk>>:
>>>> >
>>>> > did you binarize the LM and what did you binarize it with? KenLM
>>>> or
>>>> > IRSTLM?
>>>> >
>>>> > at the moment, it's set to SRILM, which doesn't have a binary
>>>> format
>>>> >
>>>> >
>>>> >
>>>> > On 22 March 2014 10:10, Massinissa Ahmim
>>>> > <massinissa.ahmim@linguacustodia.com
>>>> > <mailto:massinissa.ahmim@linguacustodia.com>> wrote:
>>>> >
>>>> > Hi Hieu,
>>>> >
>>>> > I used the same binarisation command as the one stated
>>>> here-above
>>>> >
>>>> > I get this error when I try to translate something like : echo
>>>> > 'ceci est un test' | /mosesdecoder/bin/moses_chart -f
>>>> moses.ini
>>>> >
>>>> > Here is my moses.ini :
>>>> >
>>>> > #########################
>>>> > ### MOSES CONFIG FILE ###
>>>> > #########################
>>>> >
>>>> > # input factors
>>>> > [input-factors]
>>>> > 0
>>>> >
>>>> > # mapping steps
>>>> > [mapping]
>>>> > 0 T 0
>>>> > 1 T 1
>>>> >
>>>> > # translation tables: table type (hierarchical(0), textual
>>>> (0),
>>>> > binary (1)), source-factors, target-factors, number of
>>>> scores, file
>>>> > # OLD FORMAT is still handled for back-compatibility
>>>> > # OLD FORMAT translation tables: source-factors,
>>>> target-factors,
>>>> > number of scores, file
>>>> > # OLD FORMAT a binary table type (1) is assumed
>>>> > [ttable-file]
>>>> > 6 0 0 1
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table
>>>> > 6 0 0 1
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>>> >
>>>> > # no generation models, no generation-file section
>>>> >
>>>> > # language models: type(srilm/irstlm), factors, order, file
>>>> > [lmodel-file]
>>>> > 0 0 5
>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>>> >
>>>> >
>>>> > # limit on how many phrase translations e for each phrase f
>>>> are
>>>> > loaded
>>>> > # 0 = all elements loaded
>>>> > [ttable-limit]
>>>> > 20
>>>> >
>>>> >
>>>> > # language model weights
>>>> > [weight-l]
>>>> > 0.5000
>>>> >
>>>> >
>>>> > # translation model weights
>>>> > [weight-t]
>>>> > 0.20
>>>> > 0.20
>>>> > 0.20
>>>> > 0.20
>>>> > 0.20
>>>> > 1.0
>>>> >
>>>> > # no generation models, no weight-generation section
>>>> >
>>>> > # word penalty
>>>> > [weight-w]
>>>> > -1
>>>> >
>>>> > [cube-pruning-pop-limit]
>>>> > 1000
>>>> >
>>>> > [non-terminals]
>>>> > X
>>>> >
>>>> > [search-algorithm]
>>>> > 3
>>>> >
>>>> > [inputtype]
>>>> > 3
>>>> >
>>>> > [max-chart-span]
>>>> > 20
>>>> > 1000
>>>> >
>>>> >
>>>> >
>>>> > 2014-03-21 20:49 GMT+01:00 Hieu Hoang <Hieu.Hoang@ed.ac.uk
>>>> > <mailto:Hieu.Hoang@ed.ac.uk>>:
>>>> >
>>>> > i've never seen that error before. Did you change anything
>>>> > else, other than rebinarizing the phrase-table? What is
>>>> the
>>>> > exact command you ran when you got this error? Can I have
>>>> a
>>>> > look at your moses.ini file
>>>> >
>>>> >
>>>> > On 21 March 2014 18:32, Massinissa Ahmim
>>>> > <massinissa.ahmim@linguacustodia.com
>>>> > <mailto:massinissa.ahmim@linguacustodia.com>> wrote:
>>>> >
>>>> > Thank you Hieu,
>>>> >
>>>> > I've binarised my rule-table again and it seems to be
>>>> > successfully loaded but I'm getting this message now :
>>>> >
>>>> > Reading
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table
>>>> >
>>>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>>>> >
>>>> ****************************************************************************************************
>>>> > Exception: <fd:3>: Is a directory from zlib
>>>> >
>>>> > What can I do?
>>>> >
>>>> > Thanks !
>>>> >
>>>> > Regards
>>>> >
>>>> > Massinissa
>>>> >
>>>> >
>>>> > 2014-03-21 11:18 GMT+01:00 Hieu Hoang
>>>> > <Hieu.Hoang@ed.ac.uk <mailto:Hieu.Hoang@ed.ac.uk>>:
>>>> >
>>>> > everything seems to be ok. Are you sure the server
>>>> > you are running on has access to the directory
>>>> >
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>> >
>>>> > Also, you should consider updating your version of
>>>> > Moses to the current github version, or version
>>>> 2.1.1
>>>> >
>>>> >
>>>> > On 20 March 2014 11:20, Massinissa Ahmim
>>>> > <massinissa.ahmim@linguacustodia.com
>>>> > <mailto:massinissa.ahmim@linguacustodia.com>>
>>>> wrote:
>>>> >
>>>> > Dear all,
>>>> >
>>>> > I've managed to train a hierarchical model
>>>> using
>>>> > the following command :
>>>> >
>>>> > nohup
>>>> >
>>>> /mosesdecoder/scripts/training/train-model.perl
>>>> > --hierarchical --glue-grammar
>>>> > --score-options="--GoodTuring" -root-dir
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT
>>>> > -corpus
>>>> > /disque2/Preparation/backoff/PSCT.tok.uni.low
>>>> > -f fr -e en -lm
>>>> >
>>>> 0:5:/disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>>> > -external-bin-dir /root/external-bin-dir/
>>>> -mgiza
>>>> > -mgiza-cpus 30 >& training.out &
>>>> >
>>>> > The resulting model works great but it is very
>>>> > slow.
>>>> >
>>>> > So I used CreatOnDiskPt to binarise the rule
>>>> > table as follows :
>>>> >
>>>> > /home/Moses/mosesdecoder/bin/CreateOnDiskPt 1
>>>> 1
>>>> > 5 20 2 model/rule-table.gz rules-table
>>>> >
>>>> > This outputs the following files
>>>> >
>>>> > -rw-r--r-- 1 root root 85 20 mars
>>>> 11:35
>>>> > Misc.dat
>>>> > -rw-r--r-- 1 root root 781509322 20 mars
>>>> 11:35
>>>> > Source.dat
>>>> > -rw-r--r-- 1 root root 1758588137 20 mars
>>>> 11:35
>>>> > TargetColl.dat
>>>> > -rw-r--r-- 1 root root 1555353459 20 mars
>>>> 11:35
>>>> > TargetInd.dat
>>>> > -rw-r--r-- 1 root root 400714 20 mars
>>>> 11:35
>>>> > Vocab.dat
>>>> >
>>>> > and I've updated my moses.ini as follows :
>>>> >
>>>> > 6 0 0 1
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>> > 6 0 0 1
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>>> >
>>>> > But as I try to use it, I'm getting this :
>>>> >
>>>> > Defined parameters (per moses.ini or switch):
>>>> > config: moses.ini
>>>> > cube-pruning-pop-limit: 1000
>>>> > input-factors: 0
>>>> > inputtype: 3
>>>> > lmodel-file: 0 0 5
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>>> > mapping: 0 T 0 1 T 1
>>>> > max-chart-span: 20 1000
>>>> > non-terminals: X
>>>> > search-algorithm: 3
>>>> > ttable-file: 6 0 0 1
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>> > 6 0 0 1
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>>> > ttable-limit: 20
>>>> > weight-l: 0.5000
>>>> > weight-t: 0.20 0.20 0.20 0.20 0.20 1.0
>>>> > weight-w: -1
>>>> > /mosesdecoder/bin
>>>> > ScoreProducer: WordPenalty start: 0 end: 1
>>>> > ScoreProducer: !UnknownWordPenalty start: 1
>>>> end: 2
>>>> > Loading lexical distortion models...have 0
>>>> models
>>>> > Start loading LanguageModel
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>>> > : [0.000] seconds
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm:
>>>> > line 80492: reached EOF before \end\
>>>> > ScoreProducer: LM start: 2 end: 3
>>>> > Finished loading LanguageModels : [0.056]
>>>> seconds
>>>> > Using uniform ttable-limit of 20 for all
>>>> > translation tables.
>>>> > Start loading PhraseTable
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>> > : [0.056] seconds
>>>> > filePath:
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>> > ScoreProducer: PhraseModel start: 3 end: 4
>>>> > Start loading PhraseTable
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>>> > : [0.056] seconds
>>>> > filePath:
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>>> > ScoreProducer: PhraseModel:2 start: 4 end: 5
>>>> > Finished loading phrase tables : [0.056]
>>>> seconds
>>>> > max-chart-span: 20
>>>> > max-chart-span: 1000
>>>> > Start loading phrase table from
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>> > : [0.056] seconds
>>>> > Can't read
>>>> >
>>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>>> >
>>>> > So I wondering if I did something wrong with
>>>> my
>>>> > training command/binarisation or with the
>>>> > parameters in the moses.ini
>>>> >
>>>> >
>>>> > Many thanks
>>>> >
>>>> > Regads
>>>> >
>>>> > MA
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> >
>>>> > Description : Description :
>>>> > lingua_custodia_final full logo
>>>> >
>>>> > */The Translation Trustee/*
>>>> >
>>>> > *1, Place Charles de Gaulle*
>>>> >
>>>> > *78180 Montigny-le-Bretonneux*
>>>> >
>>>> > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61
>>>> 44
>>>> > 40 84*
>>>> >
>>>> > *Email :**
>>>> > **massinissa.ahmim@linguacustodia.com
>>>> > <mailto:massinissa.ahmim@linguacustodia.com
>>>> >***
>>>> >
>>>> > *Website :** **www.linguacustodia.com
>>>> > <http://www.linguacustodia.com/> -
>>>> > www.thetranslationtrustee.com
>>>> > <http://www.thetranslationtrustee.com>*
>>>> >
>>>> > ? Pensez ? l'environnement, n'imprimez ce
>>>> > courriel que si n?cessaire.
>>>> >
>>>> > Please do not print this email unless it is
>>>> > absolutely necessary. Spread environmental
>>>> > awareness.
>>>> >
>>>> >
>>>> >
>>>> _______________________________________________
>>>> > Moses-support mailing list
>>>> > Moses-support@mit.edu <mailto:
>>>> Moses-support@mit.edu>
>>>> >
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Hieu Hoang
>>>> > Research Associate
>>>> > University of Edinburgh
>>>> > http://www.hoang.co.uk/hieu
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> >
>>>> > Description : Description : lingua_custodia_final
>>>> full logo
>>>> >
>>>> > */The Translation Trustee/*
>>>> >
>>>> > *1, Place Charles de Gaulle*
>>>> >
>>>> > *78180 Montigny-le-Bretonneux*
>>>> >
>>>> > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84
>>>> *
>>>> >
>>>> > *Email :** **massinissa.ahmim@linguacustodia.com
>>>> > <mailto:massinissa.ahmim@linguacustodia.com>***
>>>> >
>>>> > *Website :** **www.linguacustodia.com
>>>> > <http://www.linguacustodia.com/> -
>>>> > www.thetranslationtrustee.com
>>>> > <http://www.thetranslationtrustee.com>*
>>>> >
>>>> > ? Pensez ? l'environnement, n'imprimez ce courriel que
>>>> > si n?cessaire.
>>>> >
>>>> > Please do not print this email unless it is absolutely
>>>> > necessary. Spread environmental awareness.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Hieu Hoang
>>>> > Research Associate
>>>> > University of Edinburgh
>>>> > http://www.hoang.co.uk/hieu
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> >
>>>> > Description : Description : lingua_custodia_final full logo
>>>> >
>>>> > */The Translation Trustee/*
>>>> >
>>>> > *1, Place Charles de Gaulle*
>>>> >
>>>> > *78180 Montigny-le-Bretonneux*
>>>> >
>>>> > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>>>> >
>>>> > *Email :** **massinissa.ahmim@linguacustodia.com
>>>> > <mailto:massinissa.ahmim@linguacustodia.com>***
>>>> >
>>>> > *Website :** **www.linguacustodia.com
>>>> > <http://www.linguacustodia.com/> -
>>>> > www.thetranslationtrustee.com
>>>> > <http://www.thetranslationtrustee.com>*
>>>> >
>>>> > ? Pensez ? l'environnement, n'imprimez ce courriel que si
>>>> > n?cessaire.
>>>> >
>>>> > Please do not print this email unless it is absolutely
>>>> > necessary. Spread environmental awareness.
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Hieu Hoang
>>>> > Research Associate
>>>> > University of Edinburgh
>>>> > http://www.hoang.co.uk/hieu
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> >
>>>> > Description : Description : lingua_custodia_final full logo
>>>> >
>>>> > */The Translation Trustee/*
>>>> >
>>>> > *1, Place Charles de Gaulle*
>>>> >
>>>> > *78180 Montigny-le-Bretonneux*
>>>> >
>>>> > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>>>> >
>>>> > *Email :** **massinissa.ahmim@linguacustodia.com
>>>> > <mailto:massinissa.ahmim@linguacustodia.com>***
>>>> >
>>>> > *Website :** **www.linguacustodia.com
>>>> > <http://www.linguacustodia.com/> - www.thetranslationtrustee.com
>>>> > <http://www.thetranslationtrustee.com>*
>>>> >
>>>> > ? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.
>>>> >
>>>> > Please do not print this email unless it is absolutely necessary.
>>>> Spread
>>>> > environmental awareness.
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Moses-support mailing list
>>>> > Moses-support@mit.edu
>>>> > http://mailman.mit.edu/mailman/listinfo/moses-support
>>>> >
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>
>>>
>>>
>>> --
>>> Hieu Hoang
>>> Research Associate
>>> University of Edinburgh
>>> http://www.hoang.co.uk/hieu
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>>
>> --
>>
>> [image: Description : Description : lingua_custodia_final full logo]
>>
>> *The Translation Trustee*
>>
>> *1, Place Charles de Gaulle*
>>
>> *78180 Montigny-le-Bretonneux*
>>
>> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>>
>> *Email :* *massinissa.ahmim@linguacustodia.com
>> <massinissa.ahmim@linguacustodia.com>*
>>
>> *Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> -
>> www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*
>>
>> ? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.
>>
>> Please do not print this email unless it is absolutely necessary. Spread
>> environmental awareness.
>>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>


--

[image: Description : Description : lingua_custodia_final full logo]

*The Translation Trustee*

*1, Place Charles de Gaulle*

*78180 Montigny-le-Bretonneux*

*Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*

*Email :* *massinissa.ahmim@linguacustodia.com
<massinissa.ahmim@linguacustodia.com>*

*Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> -
www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*

? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.

Please do not print this email unless it is absolutely necessary. Spread
environmental awareness.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140324/e947d9b5/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 4421 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140324/e947d9b5/attachment.jpg

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 89, Issue 65
*********************************************

Tweet