Moses-support Digest, Vol 89, Issue 37

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. train-model.perl creates moses.ini that moses can't handle
(Peter Kleiweg)
2. Re: train-model.perl creates moses.ini that moses can't
handle (Hieu Hoang)
3. Re: train-model.perl creates moses.ini that moses can't
handle (Peter Kleiweg)
4. Re: train-model.perl creates moses.ini that moses can't
handle (Philipp Koehn)


----------------------------------------------------------------------

Message: 1
Date: Mon, 17 Mar 2014 18:21:44 +0100 (CET)
From: Peter Kleiweg <p.c.j.kleiweg@rug.nl>
Subject: [Moses-support] train-model.perl creates moses.ini that moses
can't handle
To: moses-support@mit.edu
Message-ID: <alpine.DEB.2.00.1403171817560.3142@pebbe>
Content-Type: TEXT/PLAIN; charset=US-ASCII


I run train-model.perl, and it ends with "(9) create moses.ini"
without any error.

When I try to start moses with "moses -f moses.ini" it fails
with "Exception: bitset::set"

Is this a bug? What do I do?
What does it even mean, bitset::set?

--
Peter Kleiweg
http://pkleiweg.home.xs4all.nl/


------------------------------

Message: 2
Date: Mon, 17 Mar 2014 17:44:29 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] train-model.perl creates moses.ini that
moses can't handle
To: Peter Kleiweg <p.c.j.kleiweg@rug.nl>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbhtoDCgs_if0Y8Z_aRSCb+5aSxyT2_9LVY8dV8aVskQdA@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

can you send me me the exact command you executed, the moses.ini file used,
and a few lines of the input file


On 17 March 2014 17:21, Peter Kleiweg <p.c.j.kleiweg@rug.nl> wrote:

>
> I run train-model.perl, and it ends with "(9) create moses.ini"
> without any error.
>
> When I try to start moses with "moses -f moses.ini" it fails
> with "Exception: bitset::set"
>
> Is this a bug? What do I do?
> What does it even mean, bitset::set?
>
> --
> Peter Kleiweg
> http://pkleiweg.home.xs4all.nl/
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140317/a6c03ea3/attachment-0001.htm

------------------------------

Message: 3
Date: Mon, 17 Mar 2014 20:01:23 +0100 (CET)
From: Peter Kleiweg <p.c.j.kleiweg@rug.nl>
Subject: Re: [Moses-support] train-model.perl creates moses.ini that
moses can't handle
To: moses-support@mit.edu
Message-ID: <alpine.DEB.2.00.1403171951390.3975@pebbe>
Content-Type: TEXT/PLAIN; charset=US-ASCII

Hieu Hoang schreef op de 17e dag van de lentemaand van het jaar 2014:

> can you send me me the exact command you executed, the moses.ini file used,
> and a few lines of the input file

Here is the training command:

en_word=0
en_root=1
en_pos=2
en_rel=3
nl_word=0
nl_root=1
nl_pos=2
nl_postag=3
nl_rel=4
/net/aps/64/opt/moses/mosesdecoder/scripts/training/train-model.perl \
-cores 12 \
-root-dir train \
-corpus /net/aistaff/kleiweg/moses/europarl/factored/corpus/train-small.clean \
-f en -e nl \
-alignment grow-diag-final-and \
-reordering msd-bidirectional-fe \
--lm $nl_word:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-word:8 \
--lm $nl_root:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-root:8 \
--lm $nl_pos:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-pos:8 \
--lm $nl_postag:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-postag:8 \
--lm $nl_rel:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-rel:8 \
--alignment-factors $en_root-$nl_root \
--translation-factors $en_word-$nl_word+$en_root-$nl_root+$en_pos-$nl_pos+$en_pos-$nl_postag+$en_rel-$nl_rel \
--reordering-factors $en_word-$nl_word+$en_root-$nl_root+$en_pos-$nl_pos+$en_pos-$nl_postag+$en_rel-$nl_rel \
--generation-factors $nl_root,$nl_pos,$nl_postag,$nl_rel-$nl_word \
--decoding-steps t0:t1,t2,t3,t4,g0 \
-external-bin-dir /net/aps/64/opt/moses/mgizapp/bin \
-mgiza \
-mgiza-cpus=12 \
> training.out \
2> training.err

Here are three lines of training data, input:

I|I|prp|nsubj declare|declare|vbp|hd resumed|resume|vbd|hd the|the|dt|det session|session|nn|hd of|of|in|hd the|the|dt|det European|European|nnp|nn Parliament|Parliament|nnp|hd adjourned|adjourn|vbn|hd on|on|in|hd Friday|Friday|nnp|hd 17|17|cd|num December|December|nnp|hd 1999|1999|cd|num ,|,|,|punct and|and|cc|cc I|I|prp|nsubj would|would|md|aux like|like|vb|hd once|once|rb|advmod again|again|rb|hd to|to|to|aux wish|wish|vb|hd you|you|prp|nsubj a|a|dt|det happy|happy|jj|amod new|new|jj|amod year|year|nn|hd in|in|in|hd the|the|dt|det hope|hope|nn|hd that|that|in|mark you|you|prp|nsubj enjoyed|enjoy|vbd|hd a|a|dt|det pleasant|pleasant|jj|amod festive|festive|jj|amod period|period|nn|hd .|.|.|punct

although|although|in|mark ,|,|,|punct as|as|in|mark you|you|prp|nsubj will|will|md|aux have|have|vb|aux seen|see|vbn|hd ,|,|,|punct the|the|dt|det dreaded|dreaded|jj|amod &apos;|&apos;|&quot;|punct millennium|millennium|nn|nn bug|bug|nn|hd &apos;|&apos;|pos|punct failed|fail|vbn|hd to|to|to|hd materialise|materialise|nn|pobj ,|,|,|punct still|still|rb|advmod the|the|dt|det people|people|nns|hd in|in|in|hd a|a|dt|det number|number|nn|hd of|of|in|hd countries|country|nns|pobj suffered|suffer|vbd|hd a|a|dt|det series|series|nn|hd of|of|in|hd natural|natural|jj|amod disasters|disaster|nns|hd that|that|wdt|nsubj truly|truly|rb|advmod were|be|vbd|cop dreadful|dreadful|jj|hd .|.|.|punct

you|you|prp|nsubj have|have|vbp|aux requested|request|vbn|hd a|a|dt|det debate|debate|nn|hd on|on|in|hd this|this|dt|det subject|subject|nn|hd in|in|in|hd the|the|dt|det course|course|nn|hd of|of|in|hd the|the|dt|det next|next|jj|amod few|few|jj|amod days|day|nns|hd ,|,|,|punct during|during|in|hd this|this|dt|det part-session|part-session|nn|hd .|.|.|punct

And three lines of training data, target:

ik|ik|pron|VNW(pers,pron,nomin,vol,1,ev)|su verklaar|verklaar|verb|WW(pv,tgw,ev)|hd de|de|det|LID(bep,stan,rest)|det zitting|zitting|noun|N(soort,ev,basis,zijd,stan)|hd van|van|prep|VZ(init)|hd het|het|det|LID(bep,stan,evon)|det Europees|Europees|name|SPEC(deeleigen)|mwp Parlement|Parlement|name|SPEC(deeleigen)|mwp ,|,|punct|LET()|-- die|die|pron|VNW(betr,pron,stan,vol,persoon,getal)|rhd op|op|prep|VZ(init)|hd vrijdag|vrijdag|noun|N(eigen,ev,basis,zijd,stan)|hd 17|17|noun|TW(hoofd,vrij)|mwp december|december|noun|TW(hoofd,vrij)|mwp werd|word|verb|WW(pv,verl,ev)|hd onderbroken|onderbreek|verb|WW(vd,vrij,zonder)|hd ,|,|punct|LET()|-- te|te|comp|VZ(init)|cmp zijn|ben|verb|WW(inf,vrij,zonder)|hd hervat|hervat|verb|WW(vd,vrij,zonder)|vc .|.|punct|LET()|-- ik|ik|pron|VNW(pers,pron,nomin,vol,1,ev)|su wens|wens|verb|WW(pv,tgw,ev)|hd u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|hd allen|allen|adv|BW()|mod een|een|det|LID(onbep,stan,agr)|det gelukkig|gelukkig|adj|ADJ(prenom,basis,zonder!
)|mod nieuwjaar|nieuwjaar|noun|N(soort,ev,basis,onz,stan)|hd en|en|vg|VG(neven)|crd hoop|hoop|verb|WW(pv,tgw,ev)|hd dat|dat|comp|VG(onder)|cmp u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su een|een|det|LID(onbep,stan,agr)|det goede|goed|adj|ADJ(prenom,basis,met-e,stan)|mod vakantie|vakantie|noun|N(soort,ev,basis,zijd,stan)|hd heeft|heb|verb|WW(pv,tgw,met-t)|hd gehad|heb|verb|WW(vd,vrij,zonder)|hd .|.|punct|LET()|--

zoals|zoals|comp|VG(onder)|cmp u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su heeft|heb|verb|WW(pv,tgw,met-t)|hd kunnen|kan|verb|WW(inf,vrij,zonder)|hd constateren|constateer|verb|WW(inf,vrij,zonder)|hd ,|,|punct|LET()|-- is|ben|verb|WW(pv,tgw,ev)|hd de|de|det|LID(bep,stan,rest)|det grote|groot|adj|ADJ(prenom,basis,met-e,stan)|mod &quot;|&quot;|punct|LET()|-- millenniumbug|millenniumbug|name|N(eigen,ev,basis,zijd,stan)|hd &quot;|&quot;|punct|LET()|-- uitgebleven|blijf_uit|verb|WW(vd,vrij,zonder)|hd .|.|punct|LET()|-- de|de|det|LID(bep,stan,rest)|det burgers|burger|noun|N(soort,mv,basis)|hd van|van|prep|VZ(init)|hd een|een|det|LID(onbep,stan,agr)|det aantal|aantal|noun|N(soort,ev,basis,onz,stan)|hd van|van|prep|VZ(init)|hd onze|ons|det|VNW(bez,det,stan,vol,1,mv,prenom,met-e,rest)|det lidstaten|lidstaat|noun|N(soort,mv,basis)|hd zijn|ben|verb|WW(pv,tgw,mv)|hd daarentegen|daarentegen|adv|BW()|mod door|door|prep|VZ(init)|hd verschrikkelijke|verschrikkelijk|adj|ADJ(prenom,basis,me!
t-e,stan)|mod natuurrampen|natuur_ramp|noun|N(soort,mv,basis)|hd getroffen|tref|verb|WW(vd,vrij,zonder)|hd .|.|punct|LET()|--

u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su heeft|heb|verb|WW(pv,tgw,met-t)|hd aangegeven|geef_aan|verb|WW(vd,vrij,zonder)|hd dat|dat|comp|VG(onder)|cmp u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su deze|deze|det|VNW(aanw,det,stan,prenom,met-e,rest)|det vergaderperiode|vergader_periode|noun|N(soort,ev,basis,zijd,stan)|hd een|een|det|LID(onbep,stan,agr)|det debat|debat|noun|N(soort,ev,basis,onz,stan)|hd wilt|wil|verb|WW(pv,tgw,met-t)|hd over|over|prep|VZ(init)|hd deze|deze|det|VNW(aanw,det,stan,prenom,met-e,rest)|det rampen|ramp|noun|N(soort,mv,basis)|hd .|.|punct|LET()|--

Here is the moses.ini created by the training:

#########################
### MOSES CONFIG FILE ###
#########################

# input factors
[input-factors]
0
1
2
3

# mapping steps
[mapping]
0 T 0
1 T 1
1 T 2
1 T 3
1 T 4
1 G 0

[distortion-limit]
6

# feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.0-0.gz input-factor=0 output-factor=0
PhraseDictionaryMemory name=TranslationModel1 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.1-1.gz input-factor=1 output-factor=1
PhraseDictionaryMemory name=TranslationModel2 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-2.gz input-factor=2 output-factor=2
PhraseDictionaryMemory name=TranslationModel3 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-3.gz input-factor=2 output-factor=3
PhraseDictionaryMemory name=TranslationModel4 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.3-4.gz input-factor=3 output-factor=4
Generation name=GenerationModel0 num-features=2 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/generation.1,2,3,4-0.gz input-factor=1,2,3,4 output-factor=0
LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.0-0.wbe-msd-bidirectional-fe.gz
LexicalReordering name=LexicalReordering1 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=1 output-factor=1 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.1-1.wbe-msd-bidirectional-fe.gz
LexicalReordering name=LexicalReordering2 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=2 output-factor=2 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-2.wbe-msd-bidirectional-fe.gz
LexicalReordering name=LexicalReordering3 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=2 output-factor=3 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-3.wbe-msd-bidirectional-fe.gz
LexicalReordering name=LexicalReordering4 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=3 output-factor=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.3-4.wbe-msd-bidirectional-fe.gz
Distortion
KENLM lazyken=0 name=LM0 factor=0 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-word order=3
KENLM lazyken=0 name=LM1 factor=1 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-root order=3
KENLM lazyken=0 name=LM2 factor=2 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-pos order=3
KENLM lazyken=0 name=LM3 factor=3 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-postag order=3
KENLM lazyken=0 name=LM4 factor=4 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-rel order=3

# dense weights for feature functions
[weight]
UnknownWordPenalty0= 1
WordPenalty0= -1
PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2
TranslationModel1= 0.2 0.2 0.2 0.2
TranslationModel2= 0.2 0.2 0.2 0.2
TranslationModel3= 0.2 0.2 0.2 0.2
TranslationModel4= 0.2 0.2 0.2 0.2
GenerationModel0= 0.3 0
LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
LexicalReordering1= 0.3 0.3 0.3 0.3 0.3 0.3
LexicalReordering2= 0.3 0.3 0.3 0.3 0.3 0.3
LexicalReordering3= 0.3 0.3 0.3 0.3 0.3 0.3
LexicalReordering4= 0.3 0.3 0.3 0.3 0.3 0.3
Distortion0= 0.3
LM0= 0.5
LM1= 0.5
LM2= 0.5
LM3= 0.5
LM4= 0.5

This is the command I used:

moses -f train/model/moses.ini

And this is the output of that command, until it exits,
before any actual input:

Defined parameters (per moses.ini or switch):
config: train/model/moses.ini
distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.0-0.gz input-factor=0 output-factor=0 PhraseDictionaryMemory name=TranslationModel1 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.1-1.gz input-factor=1 output-factor=1 PhraseDictionaryMemory name=TranslationModel2 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-2.gz input-factor=2 output-factor=2 PhraseDictionaryMemory name=TranslationModel3 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-3.gz input-factor=2 output-factor=3 PhraseDictionaryMemory name=TranslationModel4 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/trai!
n/model/phrase-table.3-4.gz input-factor=3 output-factor=4 Generation name=GenerationModel0 num-features=2 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/generation.1,2,3,4-0.gz input-factor=1,2,3,4 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.0-0.wbe-msd-bidirectional-fe.gz LexicalReordering name=LexicalReordering1 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=1 output-factor=1 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.1-1.wbe-msd-bidirectional-fe.gz LexicalReordering name=LexicalReordering2 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=2 output-factor=2 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-2.wbe-msd-bidirectional-fe.gz LexicalReordering!
name=LexicalReordering3 num-features=6 type=wbe-msd-bidirectional-fe-
allff input-factor=2 output-factor=3 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-3.wbe-msd-bidirectional-fe.gz LexicalReordering name=LexicalReordering4 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=3 output-factor=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.3-4.wbe-msd-bidirectional-fe.gz Distortion KENLM lazyken=0 name=LM0 factor=0 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-word order=3 KENLM lazyken=0 name=LM1 factor=1 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-root order=3 KENLM lazyken=0 name=LM2 factor=2 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-pos order=3 KENLM lazyken=0 name=LM3 factor=3 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-postag order=3 KENLM lazyken=0 name=LM4 factor=4 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-rel order=3
input-factors: 0 1 2 3
mapping: 0 T 0 1 T 1 1 T 2 1 T 3 1 T 4 1 G 0
weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 TranslationModel1= 0.2 0.2 0.2 0.2 TranslationModel2= 0.2 0.2 0.2 0.2 TranslationModel3= 0.2 0.2 0.2 0.2 TranslationModel4= 0.2 0.2 0.2 0.2 GenerationModel0= 0.3 0 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering1= 0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering2= 0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering3= 0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering4= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 LM1= 0.5 LM2= 0.5 LM3= 0.5 LM4= 0.5
/net/aps/64/opt/moses/mosesdecoder/bin
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhrasePenalty
FeatureFunction: PhrasePenalty0 start: 2 end: 2
line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.0-0.gz input-factor=0 output-factor=0
FeatureFunction: TranslationModel0 start: 3 end: 6
line=PhraseDictionaryMemory name=TranslationModel1 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.1-1.gz input-factor=1 output-factor=1
FeatureFunction: TranslationModel1 start: 7 end: 10
line=PhraseDictionaryMemory name=TranslationModel2 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-2.gz input-factor=2 output-factor=2
FeatureFunction: TranslationModel2 start: 11 end: 14
line=PhraseDictionaryMemory name=TranslationModel3 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-3.gz input-factor=2 output-factor=3
FeatureFunction: TranslationModel3 start: 15 end: 18
line=PhraseDictionaryMemory name=TranslationModel4 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.3-4.gz input-factor=3 output-factor=4
FeatureFunction: TranslationModel4 start: 19 end: 22
Exception: bitset::set




>
> On 17 March 2014 17:21, Peter Kleiweg <p.c.j.kleiweg@rug.nl> wrote:
>
> >
> > I run train-model.perl, and it ends with "(9) create moses.ini"
> > without any error.
> >
> > When I try to start moses with "moses -f moses.ini" it fails
> > with "Exception: bitset::set"
> >
> > Is this a bug? What do I do?
> > What does it even mean, bitset::set?



--
Peter Kleiweg
http://pkleiweg.home.xs4all.nl/


------------------------------

Message: 4
Date: Mon, 17 Mar 2014 15:14:24 -0400
From: Philipp Koehn <pkoehn@inf.ed.ac.uk>
Subject: Re: [Moses-support] train-model.perl creates moses.ini that
moses can't handle
To: Peter Kleiweg <p.c.j.kleiweg@rug.nl>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDACw_oUgTzjwFr=zAS4woz6V6ikOQY5=K=jT9U6sfMLLA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

Hi,

by default, the maximum number of factors is 4, but you are using 5
output factors.

You can recompile Moses with an additional switch:
bjam --max-factors=5

I am a bit concerned about your model since it generates a large number
of output factors independently, which may lead to an explosion of possible
translation options where all good choices get pruned out before you get
to the generation step which will eliminate all the impossible combinations.

It would be probably a good idea to use this complex factored setup only
as a back-off for unknown words. For details, please look at the following
paper: http://amta2012.amtaweb.org/AMTA2012Files/papers/147.pdf

-phi

On Mon, Mar 17, 2014 at 3:01 PM, Peter Kleiweg <p.c.j.kleiweg@rug.nl> wrote:
> Hieu Hoang schreef op de 17e dag van de lentemaand van het jaar 2014:
>
>> can you send me me the exact command you executed, the moses.ini file used,
>> and a few lines of the input file
>
> Here is the training command:
>
> en_word=0
> en_root=1
> en_pos=2
> en_rel=3
> nl_word=0
> nl_root=1
> nl_pos=2
> nl_postag=3
> nl_rel=4
> /net/aps/64/opt/moses/mosesdecoder/scripts/training/train-model.perl \
> -cores 12 \
> -root-dir train \
> -corpus /net/aistaff/kleiweg/moses/europarl/factored/corpus/train-small.clean \
> -f en -e nl \
> -alignment grow-diag-final-and \
> -reordering msd-bidirectional-fe \
> --lm $nl_word:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-word:8 \
> --lm $nl_root:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-root:8 \
> --lm $nl_pos:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-pos:8 \
> --lm $nl_postag:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-postag:8 \
> --lm $nl_rel:3:/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-rel:8 \
> --alignment-factors $en_root-$nl_root \
> --translation-factors $en_word-$nl_word+$en_root-$nl_root+$en_pos-$nl_pos+$en_pos-$nl_postag+$en_rel-$nl_rel \
> --reordering-factors $en_word-$nl_word+$en_root-$nl_root+$en_pos-$nl_pos+$en_pos-$nl_postag+$en_rel-$nl_rel \
> --generation-factors $nl_root,$nl_pos,$nl_postag,$nl_rel-$nl_word \
> --decoding-steps t0:t1,t2,t3,t4,g0 \
> -external-bin-dir /net/aps/64/opt/moses/mgizapp/bin \
> -mgiza \
> -mgiza-cpus=12 \
> > training.out \
> 2> training.err
>
> Here are three lines of training data, input:
>
> I|I|prp|nsubj declare|declare|vbp|hd resumed|resume|vbd|hd the|the|dt|det session|session|nn|hd of|of|in|hd the|the|dt|det European|European|nnp|nn Parliament|Parliament|nnp|hd adjourned|adjourn|vbn|hd on|on|in|hd Friday|Friday|nnp|hd 17|17|cd|num December|December|nnp|hd 1999|1999|cd|num ,|,|,|punct and|and|cc|cc I|I|prp|nsubj would|would|md|aux like|like|vb|hd once|once|rb|advmod again|again|rb|hd to|to|to|aux wish|wish|vb|hd you|you|prp|nsubj a|a|dt|det happy|happy|jj|amod new|new|jj|amod year|year|nn|hd in|in|in|hd the|the|dt|det hope|hope|nn|hd that|that|in|mark you|you|prp|nsubj enjoyed|enjoy|vbd|hd a|a|dt|det pleasant|pleasant|jj|amod festive|festive|jj|amod period|period|nn|hd .|.|.|punct
>
> although|although|in|mark ,|,|,|punct as|as|in|mark you|you|prp|nsubj will|will|md|aux have|have|vb|aux seen|see|vbn|hd ,|,|,|punct the|the|dt|det dreaded|dreaded|jj|amod &apos;|&apos;|&quot;|punct millennium|millennium|nn|nn bug|bug|nn|hd &apos;|&apos;|pos|punct failed|fail|vbn|hd to|to|to|hd materialise|materialise|nn|pobj ,|,|,|punct still|still|rb|advmod the|the|dt|det people|people|nns|hd in|in|in|hd a|a|dt|det number|number|nn|hd of|of|in|hd countries|country|nns|pobj suffered|suffer|vbd|hd a|a|dt|det series|series|nn|hd of|of|in|hd natural|natural|jj|amod disasters|disaster|nns|hd that|that|wdt|nsubj truly|truly|rb|advmod were|be|vbd|cop dreadful|dreadful|jj|hd .|.|.|punct
>
> you|you|prp|nsubj have|have|vbp|aux requested|request|vbn|hd a|a|dt|det debate|debate|nn|hd on|on|in|hd this|this|dt|det subject|subject|nn|hd in|in|in|hd the|the|dt|det course|course|nn|hd of|of|in|hd the|the|dt|det next|next|jj|amod few|few|jj|amod days|day|nns|hd ,|,|,|punct during|during|in|hd this|this|dt|det part-session|part-session|nn|hd .|.|.|punct
>
> And three lines of training data, target:
>
> ik|ik|pron|VNW(pers,pron,nomin,vol,1,ev)|su verklaar|verklaar|verb|WW(pv,tgw,ev)|hd de|de|det|LID(bep,stan,rest)|det zitting|zitting|noun|N(soort,ev,basis,zijd,stan)|hd van|van|prep|VZ(init)|hd het|het|det|LID(bep,stan,evon)|det Europees|Europees|name|SPEC(deeleigen)|mwp Parlement|Parlement|name|SPEC(deeleigen)|mwp ,|,|punct|LET()|-- die|die|pron|VNW(betr,pron,stan,vol,persoon,getal)|rhd op|op|prep|VZ(init)|hd vrijdag|vrijdag|noun|N(eigen,ev,basis,zijd,stan)|hd 17|17|noun|TW(hoofd,vrij)|mwp december|december|noun|TW(hoofd,vrij)|mwp werd|word|verb|WW(pv,verl,ev)|hd onderbroken|onderbreek|verb|WW(vd,vrij,zonder)|hd ,|,|punct|LET()|-- te|te|comp|VZ(init)|cmp zijn|ben|verb|WW(inf,vrij,zonder)|hd hervat|hervat|verb|WW(vd,vrij,zonder)|vc .|.|punct|LET()|-- ik|ik|pron|VNW(pers,pron,nomin,vol,1,ev)|su wens|wens|verb|WW(pv,tgw,ev)|hd u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|hd allen|allen|adv|BW()|mod een|een|det|LID(onbep,stan,agr)|det gelukkig|gelukkig|adj|ADJ(prenom,basis,zond!
er!
> )|mod nieuwjaar|nieuwjaar|noun|N(soort,ev,basis,onz,stan)|hd en|en|vg|VG(neven)|crd hoop|hoop|verb|WW(pv,tgw,ev)|hd dat|dat|comp|VG(onder)|cmp u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su een|een|det|LID(onbep,stan,agr)|det goede|goed|adj|ADJ(prenom,basis,met-e,stan)|mod vakantie|vakantie|noun|N(soort,ev,basis,zijd,stan)|hd heeft|heb|verb|WW(pv,tgw,met-t)|hd gehad|heb|verb|WW(vd,vrij,zonder)|hd .|.|punct|LET()|--
>
> zoals|zoals|comp|VG(onder)|cmp u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su heeft|heb|verb|WW(pv,tgw,met-t)|hd kunnen|kan|verb|WW(inf,vrij,zonder)|hd constateren|constateer|verb|WW(inf,vrij,zonder)|hd ,|,|punct|LET()|-- is|ben|verb|WW(pv,tgw,ev)|hd de|de|det|LID(bep,stan,rest)|det grote|groot|adj|ADJ(prenom,basis,met-e,stan)|mod &quot;|&quot;|punct|LET()|-- millenniumbug|millenniumbug|name|N(eigen,ev,basis,zijd,stan)|hd &quot;|&quot;|punct|LET()|-- uitgebleven|blijf_uit|verb|WW(vd,vrij,zonder)|hd .|.|punct|LET()|-- de|de|det|LID(bep,stan,rest)|det burgers|burger|noun|N(soort,mv,basis)|hd van|van|prep|VZ(init)|hd een|een|det|LID(onbep,stan,agr)|det aantal|aantal|noun|N(soort,ev,basis,onz,stan)|hd van|van|prep|VZ(init)|hd onze|ons|det|VNW(bez,det,stan,vol,1,mv,prenom,met-e,rest)|det lidstaten|lidstaat|noun|N(soort,mv,basis)|hd zijn|ben|verb|WW(pv,tgw,mv)|hd daarentegen|daarentegen|adv|BW()|mod door|door|prep|VZ(init)|hd verschrikkelijke|verschrikkelijk|adj|ADJ(prenom,basis,!
me!
> t-e,stan)|mod natuurrampen|natuur_ramp|noun|N(soort,mv,basis)|hd getroffen|tref|verb|WW(vd,vrij,zonder)|hd .|.|punct|LET()|--
>
> u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su heeft|heb|verb|WW(pv,tgw,met-t)|hd aangegeven|geef_aan|verb|WW(vd,vrij,zonder)|hd dat|dat|comp|VG(onder)|cmp u|u|pron|VNW(pers,pron,nomin,vol,2b,getal)|su deze|deze|det|VNW(aanw,det,stan,prenom,met-e,rest)|det vergaderperiode|vergader_periode|noun|N(soort,ev,basis,zijd,stan)|hd een|een|det|LID(onbep,stan,agr)|det debat|debat|noun|N(soort,ev,basis,onz,stan)|hd wilt|wil|verb|WW(pv,tgw,met-t)|hd over|over|prep|VZ(init)|hd deze|deze|det|VNW(aanw,det,stan,prenom,met-e,rest)|det rampen|ramp|noun|N(soort,mv,basis)|hd .|.|punct|LET()|--
>
> Here is the moses.ini created by the training:
>
> #########################
> ### MOSES CONFIG FILE ###
> #########################
>
> # input factors
> [input-factors]
> 0
> 1
> 2
> 3
>
> # mapping steps
> [mapping]
> 0 T 0
> 1 T 1
> 1 T 2
> 1 T 3
> 1 T 4
> 1 G 0
>
> [distortion-limit]
> 6
>
> # feature functions
> [feature]
> UnknownWordPenalty
> WordPenalty
> PhrasePenalty
> PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.0-0.gz input-factor=0 output-factor=0
> PhraseDictionaryMemory name=TranslationModel1 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.1-1.gz input-factor=1 output-factor=1
> PhraseDictionaryMemory name=TranslationModel2 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-2.gz input-factor=2 output-factor=2
> PhraseDictionaryMemory name=TranslationModel3 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-3.gz input-factor=2 output-factor=3
> PhraseDictionaryMemory name=TranslationModel4 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.3-4.gz input-factor=3 output-factor=4
> Generation name=GenerationModel0 num-features=2 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/generation.1,2,3,4-0.gz input-factor=1,2,3,4 output-factor=0
> LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.0-0.wbe-msd-bidirectional-fe.gz
> LexicalReordering name=LexicalReordering1 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=1 output-factor=1 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.1-1.wbe-msd-bidirectional-fe.gz
> LexicalReordering name=LexicalReordering2 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=2 output-factor=2 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-2.wbe-msd-bidirectional-fe.gz
> LexicalReordering name=LexicalReordering3 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=2 output-factor=3 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-3.wbe-msd-bidirectional-fe.gz
> LexicalReordering name=LexicalReordering4 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=3 output-factor=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.3-4.wbe-msd-bidirectional-fe.gz
> Distortion
> KENLM lazyken=0 name=LM0 factor=0 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-word order=3
> KENLM lazyken=0 name=LM1 factor=1 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-root order=3
> KENLM lazyken=0 name=LM2 factor=2 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-pos order=3
> KENLM lazyken=0 name=LM3 factor=3 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-postag order=3
> KENLM lazyken=0 name=LM4 factor=4 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-rel order=3
>
> # dense weights for feature functions
> [weight]
> UnknownWordPenalty0= 1
> WordPenalty0= -1
> PhrasePenalty0= 0.2
> TranslationModel0= 0.2 0.2 0.2 0.2
> TranslationModel1= 0.2 0.2 0.2 0.2
> TranslationModel2= 0.2 0.2 0.2 0.2
> TranslationModel3= 0.2 0.2 0.2 0.2
> TranslationModel4= 0.2 0.2 0.2 0.2
> GenerationModel0= 0.3 0
> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
> LexicalReordering1= 0.3 0.3 0.3 0.3 0.3 0.3
> LexicalReordering2= 0.3 0.3 0.3 0.3 0.3 0.3
> LexicalReordering3= 0.3 0.3 0.3 0.3 0.3 0.3
> LexicalReordering4= 0.3 0.3 0.3 0.3 0.3 0.3
> Distortion0= 0.3
> LM0= 0.5
> LM1= 0.5
> LM2= 0.5
> LM3= 0.5
> LM4= 0.5
>
> This is the command I used:
>
> moses -f train/model/moses.ini
>
> And this is the output of that command, until it exits,
> before any actual input:
>
> Defined parameters (per moses.ini or switch):
> config: train/model/moses.ini
> distortion-limit: 6
> feature: UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.0-0.gz input-factor=0 output-factor=0 PhraseDictionaryMemory name=TranslationModel1 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.1-1.gz input-factor=1 output-factor=1 PhraseDictionaryMemory name=TranslationModel2 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-2.gz input-factor=2 output-factor=2 PhraseDictionaryMemory name=TranslationModel3 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-3.gz input-factor=2 output-factor=3 PhraseDictionaryMemory name=TranslationModel4 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/tr!
ai!
> n/model/phrase-table.3-4.gz input-factor=3 output-factor=4 Generation name=GenerationModel0 num-features=2 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/generation.1,2,3,4-0.gz input-factor=1,2,3,4 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.0-0.wbe-msd-bidirectional-fe.gz LexicalReordering name=LexicalReordering1 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=1 output-factor=1 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.1-1.wbe-msd-bidirectional-fe.gz LexicalReordering name=LexicalReordering2 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=2 output-factor=2 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-2.wbe-msd-bidirectional-fe.gz LexicalReorderi!
ng!
> name=LexicalReordering3 num-features=6 type=wbe-msd-bidirectional-fe-
> allff input-factor=2 output-factor=3 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.2-3.wbe-msd-bidirectional-fe.gz LexicalReordering name=LexicalReordering4 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=3 output-factor=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/reordering-table.3-4.wbe-msd-bidirectional-fe.gz Distortion KENLM lazyken=0 name=LM0 factor=0 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-word order=3 KENLM lazyken=0 name=LM1 factor=1 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-root order=3 KENLM lazyken=0 name=LM2 factor=2 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-pos order=3 KENLM lazyken=0 name=LM3 factor=3 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-postag order=3 KENLM lazyken=0 name=LM4 factor=4 path=/net/aistaff/kleiweg/moses/europarl/factored/lm/blm.nl-rel order=3
> input-factors: 0 1 2 3
> mapping: 0 T 0 1 T 1 1 T 2 1 T 3 1 T 4 1 G 0
> weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 TranslationModel1= 0.2 0.2 0.2 0.2 TranslationModel2= 0.2 0.2 0.2 0.2 TranslationModel3= 0.2 0.2 0.2 0.2 TranslationModel4= 0.2 0.2 0.2 0.2 GenerationModel0= 0.3 0 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering1= 0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering2= 0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering3= 0.3 0.3 0.3 0.3 0.3 0.3 LexicalReordering4= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 LM1= 0.5 LM2= 0.5 LM3= 0.5 LM4= 0.5
> /net/aps/64/opt/moses/mosesdecoder/bin
> line=UnknownWordPenalty
> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
> line=WordPenalty
> FeatureFunction: WordPenalty0 start: 1 end: 1
> line=PhrasePenalty
> FeatureFunction: PhrasePenalty0 start: 2 end: 2
> line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.0-0.gz input-factor=0 output-factor=0
> FeatureFunction: TranslationModel0 start: 3 end: 6
> line=PhraseDictionaryMemory name=TranslationModel1 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.1-1.gz input-factor=1 output-factor=1
> FeatureFunction: TranslationModel1 start: 7 end: 10
> line=PhraseDictionaryMemory name=TranslationModel2 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-2.gz input-factor=2 output-factor=2
> FeatureFunction: TranslationModel2 start: 11 end: 14
> line=PhraseDictionaryMemory name=TranslationModel3 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.2-3.gz input-factor=2 output-factor=3
> FeatureFunction: TranslationModel3 start: 15 end: 18
> line=PhraseDictionaryMemory name=TranslationModel4 table-limit=0 num-features=4 path=/net/aistaff/kleiweg/moses/europarl/factored/working2/train/model/phrase-table.3-4.gz input-factor=3 output-factor=4
> FeatureFunction: TranslationModel4 start: 19 end: 22
> Exception: bitset::set
>
>
>
>
>>
>> On 17 March 2014 17:21, Peter Kleiweg <p.c.j.kleiweg@rug.nl> wrote:
>>
>> >
>> > I run train-model.perl, and it ends with "(9) create moses.ini"
>> > without any error.
>> >
>> > When I try to start moses with "moses -f moses.ini" it fails
>> > with "Exception: bitset::set"
>> >
>> > Is this a bug? What do I do?
>> > What does it even mean, bitset::set?
>
>
>
> --
> Peter Kleiweg
> http://pkleiweg.home.xs4all.nl/
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support



------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 89, Issue 37
*********************************************

Tweet