Moses-support Digest, Vol 95, Issue 49

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Moses fails loading compact phrase table (Eleftherios Avramidis)
2. Re: Moses fails loading compact phrase table
(Marcin Junczys-Dowmunt)


----------------------------------------------------------------------

Message: 1
Date: Mon, 29 Sep 2014 21:35:20 +0200
From: Eleftherios Avramidis <eleftherios.avramidis@dfki.de>
Subject: [Moses-support] Moses fails loading compact phrase table
To: Moses-support <moses-support@mit.edu>
Cc: Lukas Poustka <lukas.poustka@dfki.de>
Message-ID: <5429B478.3070900@dfki.de>
Content-Type: text/plain; charset="utf-8"

Hi,

our Moses server has issues loading the compact phrase tables. We are
running Moses 2.1.1.
Attached you can find the log file from the binarization done by
processPhraseTableMin. We run two executions of the binarization (both
included in the log). I don't know if the output of the log file was
truncated, but the executions do not seem the same.
This is the error we get during loading of phrase-table.minphr by
mosesserver:

Loading table into memory...Defined parameters (per moses.ini or switch):
config:
/project/qtleap/software/mtmonkey/user/en-de/mt-2.1/models/deen/moses.ini.4.en-de

distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty
PhraseDictionaryCompact name=TranslationModel0 table-limit=20
num-features=4
path=/project/qtleap/pilot0/systems/en-de/model/phrase-table.2.minphr
input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0
num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0
output-factor=0
path=/project/qtleap/pilot0/systems/en-de/model/reordering-table.2.wbe-msd-bidirectional-fe.gz
Distortion KENLM lazyken=0 name=LM0 factor=0
path=/project/qtleap/pilot0/systems/en-de/lm/interpolated-binlm.2 order=5
input-factors: 0
mapping: 0 T 0
weight: LexicalReordering0= 0.106673 0.0371276 0.0124599
0.0642303 0.00514413 0.0893282 Distortion0= 0.0367167 LM0= 0.0755455
WordPenalty0= -0.25141 PhrasePenalty0= 0.080744 TranslationModel0=
-0.00207054 0.0663382 0.160812 0.0113997
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhrasePenalty
FeatureFunction: PhrasePenalty0 start: 2 end: 2
line=PhraseDictionaryCompact name=TranslationModel0 table-limit=20
num-features=4
path=/project/qtleap/pilot0/systems/en-de/model/phrase-table.2.minphr
input-factor=0 output-factor=0
FeatureFunction: TranslationModel0 start: 3 end: 6
line=LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=/project/qtleap/pilot0/systems/en-de/model/reordering-table.2.wbe-msd-bidirectional-fe.gz

FeatureFunction: LexicalReordering0 start: 7 end: 12
Initializing LexicalReordering..
line=Distortion
FeatureFunction: Distortion0 start: 13 end: 13
line=KENLM lazyken=0 name=LM0 factor=0
path=/project/qtleap/pilot0/systems/en-de/lm/interpolated-binlm.2 order=5
FeatureFunction: LM0 start: 14 end: 14
Loading UnknownWordPenalty0
Loading WordPenalty0
Loading PhrasePenalty0
Loading LexicalReordering0
Loading table into memory...done.
Loading Distortion0
Loading LM0
Loading TranslationModel0
terminate called after throwing an instance of 'util::Exception'
what():
moses/TranslationModel/CompactPT/PhraseDictionaryCompact.cpp:104 in
virtual void Moses::PhraseDictionaryCompact::Load() threw
util::Exception because `indexSize == 0 || coderSize == 0 || phraseSize
== 0'.
Not successfully loaded
best
Lefteris, Lukas


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140929/dc3236fd/attachment-0001.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nohup.out
Type: application/octet-stream
Size: 8265 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140929/dc3236fd/attachment-0001.obj

------------------------------

Message: 2
Date: Mon, 29 Sep 2014 21:42:40 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Moses fails loading compact phrase table
To: moses-support@mit.edu
Message-ID: <5429B630.90005@amu.edu.pl>
Content-Type: text/plain; charset="iso-8859-1"

Hi,
that's weird. The numbers in the 3rd pass should be the same as for the
other passes. It does indeed look truncated. Could it be that you are
running out of temporary space? (Although there should be some kind of
error in that case.)

You can set the temporary folder to with -T /path/to/temp to use a
custom folder other than /tmp which is used by default. Let's see if
that helps. Since the passes are uncompressed it needs more space than
the final compressed output. Give it at least as much space as the
uncompressed text phrase table. That should be more than enough.
Best,
Marcin

W dniu 29.09.2014 21:35, Eleftherios Avramidis pisze:
> Hi,
>
> our Moses server has issues loading the compact phrase tables. We are
> running Moses 2.1.1.
> Attached you can find the log file from the binarization done by
> processPhraseTableMin. We run two executions of the binarization (both
> included in the log). I don't know if the output of the log file was
> truncated, but the executions do not seem the same.
> This is the error we get during loading of phrase-table.minphr by
> mosesserver:
>
> Loading table into memory...Defined parameters (per moses.ini or switch):
> config:
> /project/qtleap/software/mtmonkey/user/en-de/mt-2.1/models/deen/moses.ini.4.en-de
>
> distortion-limit: 6
> feature: UnknownWordPenalty WordPenalty PhrasePenalty
> PhraseDictionaryCompact name=TranslationModel0 table-limit=20
> num-features=4
> path=/project/qtleap/pilot0/systems/en-de/model/phrase-table.2.minphr
> input-factor=0 output-factor=0 LexicalReordering
> name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/project/qtleap/pilot0/systems/en-de/model/reordering-table.2.wbe-msd-bidirectional-fe.gz
> Distortion KENLM lazyken=0 name=LM0 factor=0
> path=/project/qtleap/pilot0/systems/en-de/lm/interpolated-binlm.2 order=5
> input-factors: 0
> mapping: 0 T 0
> weight: LexicalReordering0= 0.106673 0.0371276 0.0124599
> 0.0642303 0.00514413 0.0893282 Distortion0= 0.0367167 LM0= 0.0755455
> WordPenalty0= -0.25141 PhrasePenalty0= 0.080744 TranslationModel0=
> -0.00207054 0.0663382 0.160812 0.0113997
> line=UnknownWordPenalty
> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
> line=WordPenalty
> FeatureFunction: WordPenalty0 start: 1 end: 1
> line=PhrasePenalty
> FeatureFunction: PhrasePenalty0 start: 2 end: 2
> line=PhraseDictionaryCompact name=TranslationModel0 table-limit=20
> num-features=4
> path=/project/qtleap/pilot0/systems/en-de/model/phrase-table.2.minphr
> input-factor=0 output-factor=0
> FeatureFunction: TranslationModel0 start: 3 end: 6
> line=LexicalReordering name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/project/qtleap/pilot0/systems/en-de/model/reordering-table.2.wbe-msd-bidirectional-fe.gz
>
> FeatureFunction: LexicalReordering0 start: 7 end: 12
> Initializing LexicalReordering..
> line=Distortion
> FeatureFunction: Distortion0 start: 13 end: 13
> line=KENLM lazyken=0 name=LM0 factor=0
> path=/project/qtleap/pilot0/systems/en-de/lm/interpolated-binlm.2 order=5
> FeatureFunction: LM0 start: 14 end: 14
> Loading UnknownWordPenalty0
> Loading WordPenalty0
> Loading PhrasePenalty0
> Loading LexicalReordering0
> Loading table into memory...done.
> Loading Distortion0
> Loading LM0
> Loading TranslationModel0
> terminate called after throwing an instance of 'util::Exception'
> what():
> moses/TranslationModel/CompactPT/PhraseDictionaryCompact.cpp:104 in
> virtual void Moses::PhraseDictionaryCompact::Load() threw
> util::Exception because `indexSize == 0 || coderSize == 0 ||
> phraseSize == 0'.
> Not successfully loaded
> best
> Lefteris, Lukas
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140929/d455c303/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 95, Issue 49
*********************************************

Tweet