Home » Uncategories » Moses-support Digest, Vol 92, Issue 48

Moses-support Digest, Vol 92, Issue 48

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Loading binary Phrasetable (Barry Haddow)
2. escape-special-characters (charmaine ponay)
3. phrase table first into compact or binary format.
(charmaine ponay)
4. Re: Loading binary Phrasetable (Massinissa Ahmim)


----------------------------------------------------------------------

Message: 1
Date: Thu, 26 Jun 2014 21:29:23 +0100
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Loading binary Phrasetable
To: Massinissa Ahmim <massinissa.ahmim@linguacustodia.com>,
"moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <53AC82A3.5060104@staffmail.ed.ac.uk>
Content-Type: text/plain; charset=windows-1252; format=flowed

Hi Massinissa

Are you loading the right ini file? The one in the debug message is
different from the one you show at the bottom of your mail. In
particular, the debug message shows that you are mixing
"TranslationModel0" and TranslationModel1" which is why Moses complains
about missing weights,

cheers - Barry

On 26/06/14 20:48, Massinissa Ahmim wrote:
> Dear all,
>
> I'm having some troubles loading the binary phrasetable using the
> latest version of moses.
>
> When querying my moses.ini using :
>
> echo 'le monde' |/home/mosesdecoder/bin/moses -f moses.ini
>
>
> I'm getting this error :
>
> Defined parameters (per moses.ini or switch):
> config: moses.ini
> distortion-limit: 6
> feature: UnknownWordPenalty WordPenalty PhrasePenalty
> PhraseDictionaryMemory name=TranslationModel0 table-limit=20
> path=/home/training/binarised-model/phrase-table input-factor=0
> output-factor=0 LexicalReordering name=LexicalReordering0
> num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0
> output-factor=0 path=/home/training/binarised-model/reordering-table
> Distortion KENLM lazyken=0 name=LM0 factor=1
> path=/home/Massi/UN.en-fr.en.lm.blm.en order=5
> input-factors: 0
> mapping: 0 T 0
> weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
> TranslationModel1= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3
> 0.3 0.3 Distortion0= 0.3 LM0= 0.5
> line=UnknownWordPenalty
> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
> line=WordPenalty
> FeatureFunction: WordPenalty0 start: 1 end: 1
> line=PhrasePenalty
> FeatureFunction: PhrasePenalty0 start: 2 end: 2
> line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20
> path=/home/training/binarised-model/phrase-table input-factor=0
> output-factor=0
> FeatureFunction: TranslationModel0 start: 3 end: 3
> Exception: moses/ScoreComponentCollection.cpp:242 in void
> Moses::ScoreComponentCollection::Assign(const Moses::FeatureFunction*,
> const std::vector<float>&) threw util::Exception'.
> Feature function TranslationModel0 specified 1 dense scores or
> weights. Actually has 0
>
> Here is my moses.ini
>
> #########################
> ### MOSES CONFIG FILE ###
> #########################
>
> # input factors
> [input-factors]
> 0
>
> # mapping steps
> [mapping]
> 0 T 0
>
> [distortion-limit]
> 6
>
> # feature functions
> [feature]
> UnknownWordPenalty
> WordPenalty
> PhrasePenalty
> PhraseDictionaryMemory name=TranslationModel0 table-limit=20
> num-features=4 path=/home/training/binarised-model/phrase-table
> input-factor=0 outp
> ut-factor=0
> LexicalReordering name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/home/training/
> binarised-model/reordering-table
> Distortion
> KENLM lazyken=0 name=LM0 factor=1
> path=/home/Massi/UN.en-fr.en.lm.blm.en order=5
>
> # dense weights for feature functions
> [weight]
> UnknownWordPenalty0= 1
> WordPenalty0= -1
> PhrasePenalty0= 0.2
> TranslationModel0= 0.2 0.2 0.2 0.2
> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
> Distortion0= 0.3
> LM0= 0.5
>
>
> Any suggestion ?
>
> Many thanks
>
> Massinissa
>
> --
>
> Description : Description : lingua_custodia_final full logo
>
> */The Translation Trustee/*
>
> *1, Place Charles de Gaulle*
>
> *78180 Montigny-le-Bretonneux*
>
> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>
> *Email :****massinissa.ahmim@linguacustodia.com
> <mailto:massinissa.ahmim@linguacustodia.com>***
>
> *Website :****www.linguacustodia.com <http://www.linguacustodia.com/>
> - www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*
>
> ?Pensez ? l?environnement, n?imprimez ce courriel que si n?cessaire.
>
> Please do not print this email unless it is absolutely necessary.
> Spread environmental awareness.
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support


--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



------------------------------

Message: 2
Date: Thu, 26 Jun 2014 22:30:46 -0700
From: charmaine ponay <csponay@gmail.com>
Subject: [Moses-support] escape-special-characters
To: moses-support@mit.edu
Message-ID:
<CAB0nykePVSkiZvw1zNjbcai3spv7cn+2AzznvFnZSdLUZkawcw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

hi please help...

I want to use the escape special characters function, is this the correct
way?

../mosesdecoder-master/scripts/tokenizer/escape-special-chars.perl -l en \
< `pwd`/Charmaine/training.en \
> `pwd`/Charmaine/preprocessed/training.tok.en
../mosesdecoder-master/scripts/tokenizer/escape-special-chars.perl -l fl \
< `pwd`/Charmaine/training.fl \
> `pwd`/Charmaine/preprocessed/training.tok.fl

then, do I have to make changes with the following commands?

../mosesdecoder-master/scripts/training/clean-corpus-n.perl \
`pwd`/Charmaine/training fl en \
`pwd`/Charmaine/preprocessed/training.clean 1 80

ngram-count -text Charmaine/preprocessed/training.clean.en -lm
Charmaine/preprocessed/training.clean.en.lm -order 3

../mosesdecoder-master/scripts/training/train-model.perl \
-external-bin-dir ../mosesdecoder-master/tools \
--root-dir `pwd`/Charmaine/training \
--corpus `pwd`/Charmaine/preprocessed/training.clean \
--f fl --e en \
--lm 0:3:`pwd`/Charmaine/

thank you!!!


Regards,

*Charmaine Salvador - Ponay*
Instructor
Information and Computer Studies Dept.
University of Santo Tomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140626/db0889bb/attachment-0001.htm

------------------------------

Message: 3
Date: Thu, 26 Jun 2014 22:34:09 -0700
From: charmaine ponay <csponay@gmail.com>
Subject: [Moses-support] phrase table first into compact or binary
format.
To: moses-support@mit.edu
Message-ID:
<CAB0nykevKU133zh5z_a2A=aDkHwKsrfyV4-TAh6mP_e_3VPFOA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

how do i convert table into compact or binary format?

Regards,

*Charmaine Salvador - Ponay*
Instructor
Information and Computer Studies Dept.
University of Santo Tomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140626/f75b9825/attachment-0001.htm

------------------------------

Message: 4
Date: Fri, 27 Jun 2014 10:54:22 +0200
From: Massinissa Ahmim <massinissa.ahmim@linguacustodia.com>
Subject: Re: [Moses-support] Loading binary Phrasetable
To: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CANN0mWZYVrYDZHB7NG5zmpYvWSvEp_aSarxzJ_W0RsUL5WEyGg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Barry,

Thanks a lot !

I switched both to 0 but I'm still getting this message :

FeatureFunction: TranslationModel0 start: 3 end: 3
Exception: moses/ScoreComponentCollection.cpp:242 in void
Moses::ScoreComponentCollection::Assign(const Moses::FeatureFunction*,
const std::vector<float>&) threw util::Exception'.
Feature function TranslationModel0 specified 1 dense scores or weights.
Actually has 4





2014-06-26 22:29 GMT+02:00 Barry Haddow <bhaddow@staffmail.ed.ac.uk>:

> Hi Massinissa
>
> Are you loading the right ini file? The one in the debug message is
> different from the one you show at the bottom of your mail. In particular,
> the debug message shows that you are mixing "TranslationModel0" and
> TranslationModel1" which is why Moses complains about missing weights,
>
> cheers - Barry
>
>
> On 26/06/14 20:48, Massinissa Ahmim wrote:
>
>> Dear all,
>>
>> I'm having some troubles loading the binary phrasetable using the latest
>> version of moses.
>>
>> When querying my moses.ini using :
>>
>> echo 'le monde' |/home/mosesdecoder/bin/moses -f moses.ini
>>
>>
>> I'm getting this error :
>>
>> Defined parameters (per moses.ini or switch):
>> config: moses.ini
>> distortion-limit: 6
>> feature: UnknownWordPenalty WordPenalty PhrasePenalty
>> PhraseDictionaryMemory name=TranslationModel0 table-limit=20
>> path=/home/training/binarised-model/phrase-table input-factor=0
>> output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6
>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>> path=/home/training/binarised-model/reordering-table Distortion KENLM
>> lazyken=0 name=LM0 factor=1 path=/home/Massi/UN.en-fr.en.lm.blm.en
>> order=5
>> input-factors: 0
>> mapping: 0 T 0
>> weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
>> TranslationModel1= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3
>> 0.3 Distortion0= 0.3 LM0= 0.5
>> line=UnknownWordPenalty
>> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
>> line=WordPenalty
>> FeatureFunction: WordPenalty0 start: 1 end: 1
>> line=PhrasePenalty
>> FeatureFunction: PhrasePenalty0 start: 2 end: 2
>> line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20
>> path=/home/training/binarised-model/phrase-table input-factor=0
>> output-factor=0
>> FeatureFunction: TranslationModel0 start: 3 end: 3
>> Exception: moses/ScoreComponentCollection.cpp:242 in void Moses::
>> ScoreComponentCollection::Assign(const Moses::FeatureFunction*, const
>> std::vector<float>&) threw util::Exception'.
>> Feature function TranslationModel0 specified 1 dense scores or weights.
>> Actually has 0
>>
>> Here is my moses.ini
>>
>> #########################
>> ### MOSES CONFIG FILE ###
>> #########################
>>
>> # input factors
>> [input-factors]
>> 0
>>
>> # mapping steps
>> [mapping]
>> 0 T 0
>>
>> [distortion-limit]
>> 6
>>
>> # feature functions
>> [feature]
>> UnknownWordPenalty
>> WordPenalty
>> PhrasePenalty
>> PhraseDictionaryMemory name=TranslationModel0 table-limit=20
>> num-features=4 path=/home/training/binarised-model/phrase-table
>> input-factor=0 outp
>> ut-factor=0
>> LexicalReordering name=LexicalReordering0 num-features=6
>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>> path=/home/training/
>> binarised-model/reordering-table
>> Distortion
>> KENLM lazyken=0 name=LM0 factor=1 path=/home/Massi/UN.en-fr.en.lm.blm.en
>> order=5
>>
>> # dense weights for feature functions
>> [weight]
>> UnknownWordPenalty0= 1
>> WordPenalty0= -1
>> PhrasePenalty0= 0.2
>> TranslationModel0= 0.2 0.2 0.2 0.2
>> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
>> Distortion0= 0.3
>> LM0= 0.5
>>
>>
>> Any suggestion ?
>>
>> Many thanks
>>
>> Massinissa
>>
>> --
>>
>> Description : Description : lingua_custodia_final full logo
>>
>> */The Translation Trustee/*
>>
>> *1, Place Charles de Gaulle*
>>
>> *78180 Montigny-le-Bretonneux*
>>
>> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>>
>> *Email :****massinissa.ahmim@linguacustodia.com <mailto:massinissa.ahmim@
>> linguacustodia.com>***
>>
>> *Website :****www.linguacustodia.com <http://www.linguacustodia.com/> -
>> www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*
>>
>> ?Pensez ? l?environnement, n?imprimez ce courriel que si n?cessaire.
>>
>>
>> Please do not print this email unless it is absolutely necessary. Spread
>> environmental awareness.
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>


--

[image: Description : Description : lingua_custodia_final full logo]

*The Translation Trustee*

*1, Place Charles de Gaulle*

*78180 Montigny-le-Bretonneux*

*Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*

*Email :* *massinissa.ahmim@linguacustodia.com
<massinissa.ahmim@linguacustodia.com>*

*Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> -
www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*

? Pensez ? l?environnement, n?imprimez ce courriel que si n?cessaire.

Please do not print this email unless it is absolutely necessary. Spread
environmental awareness.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140627/26fa9d10/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 4421 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140627/26fa9d10/attachment.jpg

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 92, Issue 48
*********************************************

0 Response to "Moses-support Digest, Vol 92, Issue 48"

Post a Comment

Subscribe to: Post Comments (Atom)