Moses-support Digest, Vol 111, Issue 35

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Tuning with no language model (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Wed, 13 Jan 2016 14:51:15 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Tuning with no language model
To: "Read, James C" <jcread@essex.ac.uk>
Cc: Moses Support <moses-support@mit.edu>
Message-ID:
<CAEKMkbg-J+BewdCiABcfNLaH2kvW62EeCY+KeNN223TAYsLZqw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

ok. The mert script create a temporary directory every time you run it. By
default it's named
mert-work
Since you ran mert with an incorrect moses.ini previously, it may have
polluted the temporary directory and cause problem now.

You should find this temporary directory and delete it before running mert
again.

ps ALL directories must be absolute, eg ../tuning_data/true.bg


Hieu Hoang
http://www.hoang.co.uk/hieu

On 13 January 2016 at 14:45, Read, James C <jcread@essex.ac.uk> wrote:

>
> /media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data$
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/
> mert-moses.pl -no-filter-phrase-table ../tuning_data/true.bg
> ../tuning_data/true.en
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/moses
> /media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data/binarised/moses-tm.ini
> --mertdir /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/
>
> with absolute path to moses-tm.ini does not resolve the problem. Still get
> the same error.
>
> ------------------------------
> *From:* Read, James C
> *Sent:* Wednesday, January 13, 2016 2:40 PM
> *To:* Hieu Hoang
>
> *Cc:* Moses Support
> *Subject:* Re: [Moses-support] Tuning with no language model
>
>
> This is what I get when I run the same command as you:
>
>
> LexicalReordering0= 0.300000 0.300000 0.300000 0.300000 0.300000 0.300000
> Distortion0= 0.300000
> UnknownWordPenalty0 UNTUNEABLE
> WordPenalty0= -1.000000
> PhrasePenalty0= 0.200000
> TranslationModel0= 0.200000 0.200000 0.200000 0.200000
>
> Looks just like your output. However, this is what I get when I run the
> mert script with the following command:
>
>
> /media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data$
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/
> mert-moses.pl -no-filter-phrase-table ../tuning_data/true.bg
> ../tuning_data/true.en
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/moses
> binarised/moses-tm.ini --mertdir
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/
>
>
> Loading UnknownWordPenalty0
> Loading WordPenalty0
> Loading PhrasePenalty0
> Loading LexicalReordering0
> Loading Distortion0
> Loading TranslationModel0
> LM0
> The following weights have no feature function. Maybe incorrectly spelt
> weights: LM0,Exit code: 1
> The decoder died. CONFIG WAS -weight-overwrite 'PhrasePenalty0= 0.043478
> WordPenalty0= -0.217391 TranslationModel0= 0.043478 0.043478 0.043478
> 0.043478 Distortion0= 0.065217 LM0= 0.108696 LexicalReordering0= 0.065217
> 0.065217 0.065217 0.065217 0.065217 0.065217'
>
>
>
>
>
> ------------------------------
> *From:* Hieu Hoang <hieuhoang@gmail.com>
> *Sent:* Wednesday, January 13, 2016 2:27 PM
> *To:* Read, James C
> *Cc:* Moses Support
> *Subject:* Re: [Moses-support] Tuning with no language model
>
> looks fine to me. This is what I get when I used it to run
> # $MOSES_DIR/bin/moses -f moses.ini -show-weights
> ....
> LexicalReordering0= 0.300000 0.300000 0.300000 0.300000 0.300000 0.300000
> Distortion0= 0.300000
> UnknownWordPenalty0 UNTUNEABLE
> WordPenalty0= -1.000000
> PhrasePenalty0= 0.200000
> TranslationModel0= 0.200000 0.200000 0.200000 0.200000
>
> Double check that the moses.ini file you sent me is truly what you're
> using.
>
> To be sure, you MUST use absolute paths for all file names, ie. not
> binarised/moses-tm.ini
> but
> /media/bigdata/whatever/binarised/moses-tm.ini
>
> On 13/01/16 14:16, Read, James C wrote:
>
> #########################
> ### MOSES CONFIG FILE ###
> #########################
>
> # input factors
> [input-factors]
> 0
>
> # mapping steps
> [mapping]
> 0 T 0
>
> [distortion-limit]
> 6
>
> # feature functions
> [feature]
> UnknownWordPenalty
> WordPenalty
> PhrasePenalty
> PhraseDictionaryCompact name=TranslationModel0 num-features=4
> path=/media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data/binarised/phrase-table
> input-factor=0 output-factor=0
> LexicalReordering name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/media/bigdata/jcread/llv/data/europarlv7/raw/aligned/bg-en/training_data/binarised/reordering-table
> Distortion
>
> # dense weights for feature functions
> [weight]
> # The default weights are NOT optimized for translation quality. You MUST
> tune the weights.
> # Documentation for tuning is here:
> http://www.statmt.org/moses/?n=FactoredTraining.Tuning
> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning>
> Moses - FactoredTraining/Tuning
> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning>
> www.statmt.org
> Tuning. Overview. During decoding, Moses scores translation hypotheses
> using a linear model. In the traditional approach, the features of the
> model are the ...
>
> UnknownWordPenalty0= 1
> WordPenalty0= -1
> PhrasePenalty0= 0.2
> TranslationModel0= 0.2 0.2 0.2 0.2
> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
> Distortion0= 0.3
>
>
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/
> mert-moses.pl -no-filter-phrase-table ../tuning_data/true.bg
> ../tuning_data/true.en
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/moses
> binarised/moses-tm.ini --mertdir
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/bin/
>
>
> ------------------------------
> *From:* Hieu Hoang <hieuhoang@gmail.com> <hieuhoang@gmail.com>
> *Sent:* Wednesday, January 13, 2016 2:05 PM
> *To:* Read, James C
> *Cc:* Moses Support
> *Subject:* Re: [Moses-support] Tuning with no language model
>
> please show me the EXACT moses.ini and command that you use.
>
>
>
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 13 January 2016 at 14:04, Read, James C <jcread@essex.ac.uk> wrote:
>
>> So now I get the following error:
>>
>>
>>
>> Loading UnknownWordPenalty0
>> Loading WordPenalty0
>> Loading PhrasePenalty0
>> Loading LexicalReordering0
>> Loading Distortion0
>> Loading TranslationModel0
>> LM0
>> The following weights have no feature function. Maybe incorrectly spelt
>> weights: LM0,Exit code: 1
>>
>> I didn't get this error without deleting LM0 line.
>>
>> ------------------------------
>> *From:* Hieu Hoang <hieuhoang@gmail.com>
>> *Sent:* Wednesday, January 13, 2016 1:58 PM
>>
>> *To:* Read, James C; Moses Support
>> *Subject:* Re: [Moses-support] Tuning with no language model
>>
>> there lies your problem. There's a weight for a non-existant feature
>> function
>>
>> On 13/01/16 13:57, Read, James C wrote:
>>
>> no
>>
>>
>> ------------------------------
>> *From:* Hieu Hoang <hieuhoang@gmail.com> <hieuhoang@gmail.com>
>> *Sent:* Wednesday, January 13, 2016 1:56 PM
>> *To:* Read, James C; Moses Support
>> *Subject:* Re: [Moses-support] Tuning with no language model
>>
>> did you delete the line
>> LM0= 0.5
>> too?
>>
>> On 13/01/16 13:53, Read, James C wrote:
>>
>> The following command works fine for when a language model is specified.
>> Deleting the line KENLM lazyken=0 name=LM0 factor=0 path=$PWD/blm.en order=3
>> causes the script to fail with error:
>>
>>
>> ERROR: Failed to run '/pathtomoses/mosesdecoder/bin/moses -config
>> /pathtodata/binarised/moses.ini -show-weights'. at
>> /pathtomoses/mosesdecoder/scripts/training/mert-moses.pl line 1744.
>>
>>
>> #########################
>> ### MOSES CONFIG FILE ###
>> #########################
>>
>> # input factors
>> [input-factors]
>> 0
>>
>> # mapping steps
>> [mapping]
>> 0 T 0
>>
>> [distortion-limit]
>> 6
>>
>> # feature functions
>> [feature]
>> UnknownWordPenalty
>> WordPenalty
>> PhrasePenalty
>> PhraseDictionaryCompact name=TranslationModel0 num-features=4
>> path=/pathtodata/binarised/phrase-table input-factor=0 output-factor=0
>> LexicalReordering name=LexicalReordering0 num-features=6
>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>> path=/pathtodata/binarised/reordering-table
>> Distortion
>> KENLM lazyken=0 name=LM0 factor=0 path=/pathtodata/blm.en order=3
>>
>> # dense weights for feature functions
>> [weight]
>> # The default weights are NOT optimized for translation quality. You MUST
>> tune the weights.
>> # Documentation for tuning is here:
>> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning>
>> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning>
>> http://www.statmt.org/moses/?n=FactoredTraining.Tuning
>> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning>
>> Moses - FactoredTraining/Tuning
>> <http://www.statmt.org/moses/?n=FactoredTraining.Tuning>
>> <http://www.statmt.org>www.statmt.org
>> Tuning. Overview. During decoding, Moses scores translation hypotheses
>> using a linear model. In the traditional approach, the features of the
>> model are the ...
>>
>> UnknownWordPenalty0= 1
>> WordPenalty0= -1
>> PhrasePenalty0= 0.2
>> TranslationModel0= 0.2 0.2 0.2 0.2
>> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
>> Distortion0= 0.3
>> LM0= 0.5
>>
>> /pathtomoses/mosesdecoder/scripts/training/mert-moses.pl
>> -no-filter-phrase-table true.fr true.en
>> /pathtomoses/mosesdecoder/bin/moses binarised/moses.ini --mertdir
>> /pathtomoses/mosesdecoder/bin/
>>
>>
>> --
>> Hieu Hoanghttp://www.hoang.co.uk/hieu
>>
>>
>> --
>> Hieu Hoanghttp://www.hoang.co.uk/hieu
>>
>>
>
> --
> Hieu Hoanghttp://www.hoang.co.uk/hieu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160113/5ae1f378/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 111, Issue 35
**********************************************

Tweet