Moses-support Digest, Vol 108, Issue 70

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Issue with tuning - decoder died (Hieu Hoang)
2. Re: Issue with tuning - decoder died (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Fri, 23 Oct 2015 12:32:15 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Issue with tuning - decoder died
To: Ya-chi Lin <yachilin@student.uef.fi>, "moses-support@mit.edu"
<moses-support@mit.edu>
Message-ID: <562A1ABF.3000209@gmail.com>
Content-Type: text/plain; charset="windows-1252"

you must escape the [ and ] characters. If you had used the moses
tokenizer, it will do this for you. If you use a different tokenizer,
then run your tokenized output through
scripts/tokenizer/escape-special-chars.perl

On 23/10/2015 01:22, Ya-chi Lin wrote:
>
> Hi ALL,
>
>
> I'm building an English to Traditional Chinese system but have
> encountered a problem while performing tuning. Below is the content of
> mert.out file. Can anyone please tell me what went wrong here? Any
> help will be greatly appreciated.
>
>
> Using SCRIPTS_ROOTDIR: /opt/mosesdecoder/scripts
> Assuming the tables are already filtered, reusing filtered/moses.ini
> Using cached features list: ./features.list
> MERT starting values and ranges for random generation:
> LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
> LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
> LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
> LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
> LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
> LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
> Distortion0 = 0.300 ( 0.00 .. 1.00)
> LM0 = 0.500 ( 0.00 .. 1.00)
> WordPenalty0 = -1.000 ( 0.00 .. 1.00)
> PhrasePenalty0 = 0.200 ( 0.00 .. 1.00)
> TranslationModel0 = 0.200 ( 0.00 .. 1.00)
> TranslationModel0 = 0.200 ( 0.00 .. 1.00)
> TranslationModel0 = 0.200 ( 0.00 .. 1.00)
> TranslationModel0 = 0.200 ( 0.00 .. 1.00)
> featlist: LexicalReordering0=0.300000
> featlist: LexicalReordering0=0.300000
> featlist: LexicalReordering0=0.300000
> featlist: LexicalReordering0=0.300000
> featlist: LexicalReordering0=0.300000
> featlist: LexicalReordering0=0.300000
> featlist: Distortion0=0.300000
> featlist: LM0=0.500000
> featlist: WordPenalty0=-1.000000
> featlist: PhrasePenalty0=0.200000
> featlist: TranslationModel0=0.200000
> featlist: TranslationModel0=0.200000
> featlist: TranslationModel0=0.200000
> featlist: TranslationModel0=0.200000
> run 1 start at Fri Oct 23 03:07:31 EEST 2015
> Parsing --decoder-flags: ||
> Saving new config to: ./run1.moses.ini
> Saved: ./run1.moses.ini
> Normalizing lambdas: 0.300000 0.300000 0.300000 0.300000 0.300000
> 0.300000 0.300000 0.500000 -1.000000 0.200000 0.200000 0.200000
> 0.200000 0.200000
> DECODER_CFG = -weight-overwrite 'LM0= 0.108696 LexicalReordering0=
> 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217
> TranslationModel0= 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0=
> 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391'
> Executing: /opt/mosesdecoder/bin/moses -config filtered/moses.ini
> -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217 0.065217
> 0.065217 0.065217 0.065217 0.065217 TranslationModel0= 0.043478
> 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 Distortion0=
> 0.065217 WordPenalty0= -0.217391' -n-best-list run1.best100.out 100
> distinct -input-file /home/yachilin/working/tune.true.en > run1.out
> Executing: /opt/mosesdecoder/bin/moses -config filtered/moses.ini
> -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217 0.065217
> 0.065217 0.065217 0.065217 0.065217 TranslationModel0= 0.043478
> 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478 Distortion0=
> 0.065217 WordPenalty0= -0.217391' -n-best-list run1.best100.out 100
> distinct -input-file /home/yachilin/working/tune.true.en > run1.out
> 1-10.20.2 0.2 0.2 0.20.3 0.3 0.3 0.3 0.3 0.30.30.5(1) run decoder to
> produce n-best lists
> params =
> decoder_config = -weight-overwrite 'LM0= 0.108696 LexicalReordering0=
> 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217
> TranslationModel0= 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0=
> 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391'
> Defined parameters (per moses.ini or switch):
> config: filtered/moses.ini
> distortion-limit: 6
> feature: UnknownWordPenalty WordPenalty PhrasePenalty
> PhraseDictionaryMemory name=TranslationModel0 num-features=4
> path=/home/yachilin/working/mert-work/filtered/phrase-table.0-0.1.1.gz
> input-factor=0 output-factor=0 LexicalReordering
> name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/home/yachilin/working/mert-work/filtered/reordering-table.wbe-msd-bidirectional-fe
> Distortion KENLM lazyken=0 name=LM0 factor=0
> path=/home/yachilin/lm/train.blm.tch order=3
> input-factors: 0
> input-file: /home/yachilin/working/tune.true.en
> mapping: 0 T 0
> n-best-list: run1.best100.out 100 distinct
> weight: UnknownWordPenalty0= 1 WordPenalty0= -1
> PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2
> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5
> weight-overwrite: LM0= 0.108696 LexicalReordering0= 0.065217
> 0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0=
> 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478
> Distortion0= 0.065217 WordPenalty0= -0.217391
> line=UnknownWordPenalty
> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
> line=WordPenalty
> FeatureFunction: WordPenalty0 start: 1 end: 1
> line=PhrasePenalty
> FeatureFunction: PhrasePenalty0 start: 2 end: 2
> line=PhraseDictionaryMemory name=TranslationModel0 num-features=4
> path=/home/yachilin/working/mert-work/filtered/phrase-table.0-0.1.1.gz
> input-factor=0 output-factor=0
> FeatureFunction: TranslationModel0 start: 3 end: 6
> line=LexicalReordering name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/home/yachilin/working/mert-work/filtered/reordering-table.wbe-msd-bidirectional-fe
> Initializing Lexical Reordering Feature..
> FeatureFunction: LexicalReordering0 start: 7 end: 12
> line=Distortion
> FeatureFunction: Distortion0 start: 13 end: 13
> line=KENLM lazyken=0 name=LM0 factor=0
> path=/home/yachilin/lm/train.blm.tch order=3
> FeatureFunction: LM0 start: 14 end: 14
> Loading UnknownWordPenalty0
> Loading WordPenalty0
> Loading PhrasePenalty0
> Loading LexicalReordering0
> Loading table into memory...done.
> Loading Distortion0
> Loading LM0
> Loading TranslationModel0
> Start loading text phrase table. Moses format : [0.719] seconds
> Reading /home/yachilin/working/mert-work/filtered/phrase-table.0-0.1.1.gz
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> ****************************************************************************************************
> Exception: moses/Phrase.cpp:217 in void
> Moses::Phrase::CreateFromString(Moses::FactorDirection, const
> std::vector<long unsigned int>&, const StringPiece&, Moses::Word**)
> threw util::Exception because `nextPos == string::npos'.
> Incorrect formatting of non-terminal. Should have 2 non-terms, eg.
> [X][X]. Current string: [HIV]
> Exit code: 1
> The decoder died. CONFIG WAS -weight-overwrite 'LM0= 0.108696
> LexicalReordering0= 0.065217 0.065217 0.065217 0.065217 0.065217
> 0.065217 TranslationModel0= 0.043478 0.043478 0.043478 0.043478
> PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391'
> [1]+ Exit 2 $MOSES/scripts/training/mert-moses.pl tune.true.en
> tune.true.tch $MOSES/bin/moses train/model/moses.ini --mertdir
> $MOSES/bin/ &> mert.out
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

--
Hieu Hoang
http://www.hoang.co.uk/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151023/cb6c86e0/attachment-0001.html

------------------------------

Message: 2
Date: Fri, 23 Oct 2015 13:21:22 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Issue with tuning - decoder died
To: Ya-chi Lin <yachilin@student.uef.fi>, moses-support
<moses-support@mit.edu>
Message-ID: <562A2642.7090807@gmail.com>
Content-Type: text/plain; charset="utf-8"

doubt it. Look at the script. The problem chars are
[]|<>&'"

On 23/10/2015 13:01, Ya-chi Lin wrote:
>
> Hi Hieu,
>
>
> Thank for pointing this out. Could you please let me know if Chinese
> square brackets ??are also a problem?
>
>
> Best,
>
> Ya-Chi
>
>
> ------------------------------------------------------------------------
> *???:* Hieu Hoang <hieuhoang@gmail.com>
> *????:* 2015?10?23? ?? 02:32
> *???:* Ya-chi Lin; moses-support@mit.edu
> *??:* Re: [Moses-support] Issue with tuning - decoder died
> you must escape the [ and ] characters. If you had used the moses
> tokenizer, it will do this for you. If you use a different tokenizer,
> then run your tokenized output through
> scripts/tokenizer/escape-special-chars.perl
>
> On 23/10/2015 01:22, Ya-chi Lin wrote:
>>
>> Hi ALL,
>>
>>
>> I'm building an English to Traditional Chinese system but have
>> encountered a problem while performing tuning. Below is the content
>> of mert.out file. Can anyone please tell me what went wrong here? Any
>> help will be greatly appreciated.
>>
>>
>> Using SCRIPTS_ROOTDIR: /opt/mosesdecoder/scripts
>> Assuming the tables are already filtered, reusing filtered/moses.ini
>> Using cached features list: ./features.list
>> MERT starting values and ranges for random generation:
>> LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
>> LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
>> LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
>> LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
>> LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
>> LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
>> Distortion0 = 0.300 ( 0.00 .. 1.00)
>> LM0 = 0.500 ( 0.00 .. 1.00)
>> WordPenalty0 = -1.000 ( 0.00 .. 1.00)
>> PhrasePenalty0 = 0.200 ( 0.00 .. 1.00)
>> TranslationModel0 = 0.200 ( 0.00 .. 1.00)
>> TranslationModel0 = 0.200 ( 0.00 .. 1.00)
>> TranslationModel0 = 0.200 ( 0.00 .. 1.00)
>> TranslationModel0 = 0.200 ( 0.00 .. 1.00)
>> featlist: LexicalReordering0=0.300000
>> featlist: LexicalReordering0=0.300000
>> featlist: LexicalReordering0=0.300000
>> featlist: LexicalReordering0=0.300000
>> featlist: LexicalReordering0=0.300000
>> featlist: LexicalReordering0=0.300000
>> featlist: Distortion0=0.300000
>> featlist: LM0=0.500000
>> featlist: WordPenalty0=-1.000000
>> featlist: PhrasePenalty0=0.200000
>> featlist: TranslationModel0=0.200000
>> featlist: TranslationModel0=0.200000
>> featlist: TranslationModel0=0.200000
>> featlist: TranslationModel0=0.200000
>> run 1 start at Fri Oct 23 03:07:31 EEST 2015
>> Parsing --decoder-flags: ||
>> Saving new config to: ./run1.moses.ini
>> Saved: ./run1.moses.ini
>> Normalizing lambdas: 0.300000 0.300000 0.300000 0.300000 0.300000
>> 0.300000 0.300000 0.500000 -1.000000 0.200000 0.200000 0.200000
>> 0.200000 0.200000
>> DECODER_CFG = -weight-overwrite 'LM0= 0.108696 LexicalReordering0=
>> 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217
>> TranslationModel0= 0.043478 0.043478 0.043478 0.043478
>> PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391'
>> Executing: /opt/mosesdecoder/bin/moses -config filtered/moses.ini
>> -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217
>> 0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0=
>> 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478
>> Distortion0= 0.065217 WordPenalty0= -0.217391' -n-best-list
>> run1.best100.out 100 distinct -input-file
>> /home/yachilin/working/tune.true.en > run1.out
>> Executing: /opt/mosesdecoder/bin/moses -config filtered/moses.ini
>> -weight-overwrite 'LM0= 0.108696 LexicalReordering0= 0.065217
>> 0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0=
>> 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478
>> Distortion0= 0.065217 WordPenalty0= -0.217391' -n-best-list
>> run1.best100.out 100 distinct -input-file
>> /home/yachilin/working/tune.true.en > run1.out
>> 1-10.20.2 0.2 0.2 0.20.3 0.3 0.3 0.3 0.3 0.30.30.5(1) run decoder to
>> produce n-best lists
>> params =
>> decoder_config = -weight-overwrite 'LM0= 0.108696 LexicalReordering0=
>> 0.065217 0.065217 0.065217 0.065217 0.065217 0.065217
>> TranslationModel0= 0.043478 0.043478 0.043478 0.043478
>> PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391'
>> Defined parameters (per moses.ini or switch):
>> config: filtered/moses.ini
>> distortion-limit: 6
>> feature: UnknownWordPenalty WordPenalty PhrasePenalty
>> PhraseDictionaryMemory name=TranslationModel0 num-features=4
>> path=/home/yachilin/working/mert-work/filtered/phrase-table.0-0.1.1.gz input-factor=0
>> output-factor=0 LexicalReordering name=LexicalReordering0
>> num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0
>> output-factor=0
>> path=/home/yachilin/working/mert-work/filtered/reordering-table.wbe-msd-bidirectional-fe
>> Distortion KENLM lazyken=0 name=LM0 factor=0
>> path=/home/yachilin/lm/train.blm.tch order=3
>> input-factors: 0
>> input-file: /home/yachilin/working/tune.true.en
>> mapping: 0 T 0
>> n-best-list: run1.best100.out 100 distinct
>> weight: UnknownWordPenalty0= 1 WordPenalty0= -1
>> PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2
>> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5
>> weight-overwrite: LM0= 0.108696 LexicalReordering0= 0.065217
>> 0.065217 0.065217 0.065217 0.065217 0.065217 TranslationModel0=
>> 0.043478 0.043478 0.043478 0.043478 PhrasePenalty0= 0.043478
>> Distortion0= 0.065217 WordPenalty0= -0.217391
>> line=UnknownWordPenalty
>> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
>> line=WordPenalty
>> FeatureFunction: WordPenalty0 start: 1 end: 1
>> line=PhrasePenalty
>> FeatureFunction: PhrasePenalty0 start: 2 end: 2
>> line=PhraseDictionaryMemory name=TranslationModel0 num-features=4
>> path=/home/yachilin/working/mert-work/filtered/phrase-table.0-0.1.1.gz input-factor=0
>> output-factor=0
>> FeatureFunction: TranslationModel0 start: 3 end: 6
>> line=LexicalReordering name=LexicalReordering0 num-features=6
>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>> path=/home/yachilin/working/mert-work/filtered/reordering-table.wbe-msd-bidirectional-fe
>> Initializing Lexical Reordering Feature..
>> FeatureFunction: LexicalReordering0 start: 7 end: 12
>> line=Distortion
>> FeatureFunction: Distortion0 start: 13 end: 13
>> line=KENLM lazyken=0 name=LM0 factor=0
>> path=/home/yachilin/lm/train.blm.tch order=3
>> FeatureFunction: LM0 start: 14 end: 14
>> Loading UnknownWordPenalty0
>> Loading WordPenalty0
>> Loading PhrasePenalty0
>> Loading LexicalReordering0
>> Loading table into memory...done.
>> Loading Distortion0
>> Loading LM0
>> Loading TranslationModel0
>> Start loading text phrase table. Moses format : [0.719] seconds
>> Reading /home/yachilin/working/mert-work/filtered/phrase-table.0-0.1.1.gz
>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>> ****************************************************************************************************
>> Exception: moses/Phrase.cpp:217 in void
>> Moses::Phrase::CreateFromString(Moses::FactorDirection, const
>> std::vector<long unsigned int>&, const StringPiece&, Moses::Word**)
>> threw util::Exception because `nextPos == string::npos'.
>> Incorrect formatting of non-terminal. Should have 2 non-terms, eg.
>> [X][X]. Current string: [HIV]
>> Exit code: 1
>> The decoder died. CONFIG WAS -weight-overwrite 'LM0= 0.108696
>> LexicalReordering0= 0.065217 0.065217 0.065217 0.065217 0.065217
>> 0.065217 TranslationModel0= 0.043478 0.043478 0.043478 0.043478
>> PhrasePenalty0= 0.043478 Distortion0= 0.065217 WordPenalty0= -0.217391'
>> [1]+ Exit 2 $MOSES/scripts/training/mert-moses.pl tune.true.en
>> tune.true.tch $MOSES/bin/moses train/model/moses.ini --mertdir
>> $MOSES/bin/ &> mert.out
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
> --
> Hieu Hoang
> http://www.hoang.co.uk/hieu

--
Hieu Hoang
http://www.hoang.co.uk/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151023/011a413e/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 108, Issue 70
**********************************************

Tweet