Moses-support Digest, Vol 89, Issue 64

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Loading a binary rule table (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Mon, 24 Mar 2014 14:28:04 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Loading a binary rule table
To: Massinissa Ahmim <massinissa.ahmim@linguacustodia.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbhXWeYzPxuBVW+fWHgWUWxpw++K1aq4cXWq=+piV8XVzA@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

it should be
6 0 0 *5* /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
not
6 0 0 1 /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table

The glue rule entry shouldn't be changed



On 24 March 2014 13:38, Massinissa Ahmim <
massinissa.ahmim@linguacustodia.com> wrote:

> Thank you Hieu and Kenneth by rule table seems to be loaded now but I'm
> still getting an error
>
> ScoreProducer: PhraseModel:2 start: 4 end: 5
> Finished loading phrase tables : [0.039] seconds
> max-chart-span: 20
> max-chart-span: 1000
> Start loading phrase table from
> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar :
> [0.039] seconds
> Start loading text SCFG phrase table. Moses format : [0.039] seconds
> Reading /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>
> ****************************************************************************************************
> Finished loading phrase tables : [0.040] seconds
> IO from STDOUT/STDIN
> Created input-output object : [0.040] seconds
> Check m_dbWrapper.GetMisc("NumScores") == weight.size() failed in
> moses/src/RuleTable/PhraseDictionaryOnDisk.cpp:63
> End. : [0.236] seconds
> user 0.032
> sys 0.016
> VmPeak: 69020 kB
> VmRSS: 26980 kB
>
>
> How can I fixe it ?
>
> Many thanks
>
> Regards
>
>
> 2014-03-22 19:40 GMT+01:00 Hieu Hoang <Hieu.Hoang@ed.ac.uk>:
>
> ah yes, change 6 to 2, but only for the rule-table, not glue rule
>>
>>
>> On 22 March 2014 18:35, Kenneth Heafield <moses@kheafield.com> wrote:
>>
>>> Hi,
>>>
>>> The error message said
>>>
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table
>>>
>>> is a directory. But IIRC phrase table 6 is "SCFG" and expects a file.
>>> If you have binarized the phrase table specify a different magic number
>>> in the moses.ini.
>>>
>>> Nothing to do with the language model but, for the record, SRILM
>>> does
>>> have a binary format. Just not a memory mapped binary format.
>>>
>>> Kenneth
>>>
>>> On 03/22/14 04:24, Massinissa Ahmim wrote:
>>> > Hi Hieu,
>>> >
>>> > It's actually KenLM I've just updated my moses.ini changing the 0 to 8
>>> > but I'm still getting the same error
>>> >
>>> > Regards
>>> >
>>> > Massinissa
>>> >
>>> >
>>> > 2014-03-22 11:40 GMT+01:00 Hieu Hoang <Hieu.Hoang@ed.ac.uk
>>> > <mailto:Hieu.Hoang@ed.ac.uk>>:
>>> >
>>> > did you binarize the LM and what did you binarize it with? KenLM or
>>> > IRSTLM?
>>> >
>>> > at the moment, it's set to SRILM, which doesn't have a binary
>>> format
>>> >
>>> >
>>> >
>>> > On 22 March 2014 10:10, Massinissa Ahmim
>>> > <massinissa.ahmim@linguacustodia.com
>>> > <mailto:massinissa.ahmim@linguacustodia.com>> wrote:
>>> >
>>> > Hi Hieu,
>>> >
>>> > I used the same binarisation command as the one stated
>>> here-above
>>> >
>>> > I get this error when I try to translate something like : echo
>>> > 'ceci est un test' | /mosesdecoder/bin/moses_chart -f moses.ini
>>> >
>>> > Here is my moses.ini :
>>> >
>>> > #########################
>>> > ### MOSES CONFIG FILE ###
>>> > #########################
>>> >
>>> > # input factors
>>> > [input-factors]
>>> > 0
>>> >
>>> > # mapping steps
>>> > [mapping]
>>> > 0 T 0
>>> > 1 T 1
>>> >
>>> > # translation tables: table type (hierarchical(0), textual (0),
>>> > binary (1)), source-factors, target-factors, number of scores,
>>> file
>>> > # OLD FORMAT is still handled for back-compatibility
>>> > # OLD FORMAT translation tables: source-factors,
>>> target-factors,
>>> > number of scores, file
>>> > # OLD FORMAT a binary table type (1) is assumed
>>> > [ttable-file]
>>> > 6 0 0 1
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table
>>> > 6 0 0 1
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>> >
>>> > # no generation models, no generation-file section
>>> >
>>> > # language models: type(srilm/irstlm), factors, order, file
>>> > [lmodel-file]
>>> > 0 0 5 /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>> >
>>> >
>>> > # limit on how many phrase translations e for each phrase f are
>>> > loaded
>>> > # 0 = all elements loaded
>>> > [ttable-limit]
>>> > 20
>>> >
>>> >
>>> > # language model weights
>>> > [weight-l]
>>> > 0.5000
>>> >
>>> >
>>> > # translation model weights
>>> > [weight-t]
>>> > 0.20
>>> > 0.20
>>> > 0.20
>>> > 0.20
>>> > 0.20
>>> > 1.0
>>> >
>>> > # no generation models, no weight-generation section
>>> >
>>> > # word penalty
>>> > [weight-w]
>>> > -1
>>> >
>>> > [cube-pruning-pop-limit]
>>> > 1000
>>> >
>>> > [non-terminals]
>>> > X
>>> >
>>> > [search-algorithm]
>>> > 3
>>> >
>>> > [inputtype]
>>> > 3
>>> >
>>> > [max-chart-span]
>>> > 20
>>> > 1000
>>> >
>>> >
>>> >
>>> > 2014-03-21 20:49 GMT+01:00 Hieu Hoang <Hieu.Hoang@ed.ac.uk
>>> > <mailto:Hieu.Hoang@ed.ac.uk>>:
>>> >
>>> > i've never seen that error before. Did you change anything
>>> > else, other than rebinarizing the phrase-table? What is the
>>> > exact command you ran when you got this error? Can I have a
>>> > look at your moses.ini file
>>> >
>>> >
>>> > On 21 March 2014 18:32, Massinissa Ahmim
>>> > <massinissa.ahmim@linguacustodia.com
>>> > <mailto:massinissa.ahmim@linguacustodia.com>> wrote:
>>> >
>>> > Thank you Hieu,
>>> >
>>> > I've binarised my rule-table again and it seems to be
>>> > successfully loaded but I'm getting this message now :
>>> >
>>> > Reading
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rules-table
>>> >
>>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>>> >
>>> ****************************************************************************************************
>>> > Exception: <fd:3>: Is a directory from zlib
>>> >
>>> > What can I do?
>>> >
>>> > Thanks !
>>> >
>>> > Regards
>>> >
>>> > Massinissa
>>> >
>>> >
>>> > 2014-03-21 11:18 GMT+01:00 Hieu Hoang
>>> > <Hieu.Hoang@ed.ac.uk <mailto:Hieu.Hoang@ed.ac.uk>>:
>>> >
>>> > everything seems to be ok. Are you sure the server
>>> > you are running on has access to the directory
>>> >
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>> >
>>> > Also, you should consider updating your version of
>>> > Moses to the current github version, or version
>>> 2.1.1
>>> >
>>> >
>>> > On 20 March 2014 11:20, Massinissa Ahmim
>>> > <massinissa.ahmim@linguacustodia.com
>>> > <mailto:massinissa.ahmim@linguacustodia.com>>
>>> wrote:
>>> >
>>> > Dear all,
>>> >
>>> > I've managed to train a hierarchical model
>>> using
>>> > the following command :
>>> >
>>> > nohup
>>> > /mosesdecoder/scripts/training/train-model.perl
>>> > --hierarchical --glue-grammar
>>> > --score-options="--GoodTuring" -root-dir
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT
>>> > -corpus
>>> > /disque2/Preparation/backoff/PSCT.tok.uni.low
>>> > -f fr -e en -lm
>>> >
>>> 0:5:/disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>> > -external-bin-dir /root/external-bin-dir/
>>> -mgiza
>>> > -mgiza-cpus 30 >& training.out &
>>> >
>>> > The resulting model works great but it is very
>>> > slow.
>>> >
>>> > So I used CreatOnDiskPt to binarise the rule
>>> > table as follows :
>>> >
>>> > /home/Moses/mosesdecoder/bin/CreateOnDiskPt 1 1
>>> > 5 20 2 model/rule-table.gz rules-table
>>> >
>>> > This outputs the following files
>>> >
>>> > -rw-r--r-- 1 root root 85 20 mars
>>> 11:35
>>> > Misc.dat
>>> > -rw-r--r-- 1 root root 781509322 20 mars
>>> 11:35
>>> > Source.dat
>>> > -rw-r--r-- 1 root root 1758588137 20 mars
>>> 11:35
>>> > TargetColl.dat
>>> > -rw-r--r-- 1 root root 1555353459 20 mars
>>> 11:35
>>> > TargetInd.dat
>>> > -rw-r--r-- 1 root root 400714 20 mars
>>> 11:35
>>> > Vocab.dat
>>> >
>>> > and I've updated my moses.ini as follows :
>>> >
>>> > 6 0 0 1
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>> > 6 0 0 1
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>> >
>>> > But as I try to use it, I'm getting this :
>>> >
>>> > Defined parameters (per moses.ini or switch):
>>> > config: moses.ini
>>> > cube-pruning-pop-limit: 1000
>>> > input-factors: 0
>>> > inputtype: 3
>>> > lmodel-file: 0 0 5
>>> >
>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>> > mapping: 0 T 0 1 T 1
>>> > max-chart-span: 20 1000
>>> > non-terminals: X
>>> > search-algorithm: 3
>>> > ttable-file: 6 0 0 1
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>> > 6 0 0 1
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>> > ttable-limit: 20
>>> > weight-l: 0.5000
>>> > weight-t: 0.20 0.20 0.20 0.20 0.20 1.0
>>> > weight-w: -1
>>> > /mosesdecoder/bin
>>> > ScoreProducer: WordPenalty start: 0 end: 1
>>> > ScoreProducer: !UnknownWordPenalty start: 1
>>> end: 2
>>> > Loading lexical distortion models...have 0
>>> models
>>> > Start loading LanguageModel
>>> >
>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm
>>> > : [0.000] seconds
>>> >
>>> /disque2/Preparation/syntactic/hierarchical/LM.sur.en.blm:
>>> > line 80492: reached EOF before \end\
>>> > ScoreProducer: LM start: 2 end: 3
>>> > Finished loading LanguageModels : [0.056]
>>> seconds
>>> > Using uniform ttable-limit of 20 for all
>>> > translation tables.
>>> > Start loading PhraseTable
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>> > : [0.056] seconds
>>> > filePath:
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>> > ScoreProducer: PhraseModel start: 3 end: 4
>>> > Start loading PhraseTable
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>> > : [0.056] seconds
>>> > filePath:
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/model/glue-grammar
>>> > ScoreProducer: PhraseModel:2 start: 4 end: 5
>>> > Finished loading phrase tables : [0.056]
>>> seconds
>>> > max-chart-span: 20
>>> > max-chart-span: 1000
>>> > Start loading phrase table from
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>> > : [0.056] seconds
>>> > Can't read
>>> >
>>> /disque2/Preparation/syntactic/hierarchical_PSCT/rule-table
>>> >
>>> > So I wondering if I did something wrong with my
>>> > training command/binarisation or with the
>>> > parameters in the moses.ini
>>> >
>>> >
>>> > Many thanks
>>> >
>>> > Regads
>>> >
>>> > MA
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Description : Description :
>>> > lingua_custodia_final full logo
>>> >
>>> > */The Translation Trustee/*
>>> >
>>> > *1, Place Charles de Gaulle*
>>> >
>>> > *78180 Montigny-le-Bretonneux*
>>> >
>>> > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61
>>> 44
>>> > 40 84*
>>> >
>>> > *Email :**
>>> > **massinissa.ahmim@linguacustodia.com
>>> > <mailto:massinissa.ahmim@linguacustodia.com
>>> >***
>>> >
>>> > *Website :** **www.linguacustodia.com
>>> > <http://www.linguacustodia.com/> -
>>> > www.thetranslationtrustee.com
>>> > <http://www.thetranslationtrustee.com>*
>>> >
>>> > ? Pensez ? l'environnement, n'imprimez ce
>>> > courriel que si n?cessaire.
>>> >
>>> > Please do not print this email unless it is
>>> > absolutely necessary. Spread environmental
>>> > awareness.
>>> >
>>> >
>>> > _______________________________________________
>>> > Moses-support mailing list
>>> > Moses-support@mit.edu <mailto:
>>> Moses-support@mit.edu>
>>> >
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Hieu Hoang
>>> > Research Associate
>>> > University of Edinburgh
>>> > http://www.hoang.co.uk/hieu
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Description : Description : lingua_custodia_final full
>>> logo
>>> >
>>> > */The Translation Trustee/*
>>> >
>>> > *1, Place Charles de Gaulle*
>>> >
>>> > *78180 Montigny-le-Bretonneux*
>>> >
>>> > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>>> >
>>> > *Email :** **massinissa.ahmim@linguacustodia.com
>>> > <mailto:massinissa.ahmim@linguacustodia.com>***
>>> >
>>> > *Website :** **www.linguacustodia.com
>>> > <http://www.linguacustodia.com/> -
>>> > www.thetranslationtrustee.com
>>> > <http://www.thetranslationtrustee.com>*
>>> >
>>> > ? Pensez ? l'environnement, n'imprimez ce courriel que
>>> > si n?cessaire.
>>> >
>>> > Please do not print this email unless it is absolutely
>>> > necessary. Spread environmental awareness.
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Hieu Hoang
>>> > Research Associate
>>> > University of Edinburgh
>>> > http://www.hoang.co.uk/hieu
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Description : Description : lingua_custodia_final full logo
>>> >
>>> > */The Translation Trustee/*
>>> >
>>> > *1, Place Charles de Gaulle*
>>> >
>>> > *78180 Montigny-le-Bretonneux*
>>> >
>>> > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>>> >
>>> > *Email :** **massinissa.ahmim@linguacustodia.com
>>> > <mailto:massinissa.ahmim@linguacustodia.com>***
>>> >
>>> > *Website :** **www.linguacustodia.com
>>> > <http://www.linguacustodia.com/> -
>>> > www.thetranslationtrustee.com
>>> > <http://www.thetranslationtrustee.com>*
>>> >
>>> > ? Pensez ? l'environnement, n'imprimez ce courriel que si
>>> > n?cessaire.
>>> >
>>> > Please do not print this email unless it is absolutely
>>> > necessary. Spread environmental awareness.
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > Hieu Hoang
>>> > Research Associate
>>> > University of Edinburgh
>>> > http://www.hoang.co.uk/hieu
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> >
>>> > Description : Description : lingua_custodia_final full logo
>>> >
>>> > */The Translation Trustee/*
>>> >
>>> > *1, Place Charles de Gaulle*
>>> >
>>> > *78180 Montigny-le-Bretonneux*
>>> >
>>> > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>>> >
>>> > *Email :** **massinissa.ahmim@linguacustodia.com
>>> > <mailto:massinissa.ahmim@linguacustodia.com>***
>>> >
>>> > *Website :** **www.linguacustodia.com
>>> > <http://www.linguacustodia.com/> - www.thetranslationtrustee.com
>>> > <http://www.thetranslationtrustee.com>*
>>> >
>>> > ? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.
>>> >
>>> > Please do not print this email unless it is absolutely necessary.
>>> Spread
>>> > environmental awareness.
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Moses-support mailing list
>>> > Moses-support@mit.edu
>>> > http://mailman.mit.edu/mailman/listinfo/moses-support
>>> >
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>
>>
>>
>> --
>> Hieu Hoang
>> Research Associate
>> University of Edinburgh
>> http://www.hoang.co.uk/hieu
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
>
> --
>
> [image: Description : Description : lingua_custodia_final full logo]
>
> *The Translation Trustee*
>
> *1, Place Charles de Gaulle*
>
> *78180 Montigny-le-Bretonneux*
>
> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>
> *Email :* *massinissa.ahmim@linguacustodia.com
> <massinissa.ahmim@linguacustodia.com>*
>
> *Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> -
> www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*
>
> ? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.
>
> Please do not print this email unless it is absolutely necessary. Spread
> environmental awareness.
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140324/12aeb9f2/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 4421 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140324/12aeb9f2/attachment.jpg

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 89, Issue 64
*********************************************

0 Response to "Moses-support Digest, Vol 89, Issue 64"

Post a Comment