Moses-support Digest, Vol 83, Issue 1

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Tuning and decoding of lattices in the new Moses. (Yulia Tsvetkov)
2. some problems with tuning step (=?gb18030?B?1ebLrs7ez+M=?=)
3. build a demonstration system (???)


----------------------------------------------------------------------

Message: 1
Date: Sun, 1 Sep 2013 16:46:34 -0400
From: Yulia Tsvetkov <yulia.tsvetkov@gmail.com>
Subject: [Moses-support] Tuning and decoding of lattices in the new
Moses.
To: moses-support@mit.edu
Message-ID:
<CA+Drf0sAcCexZ8nabgY86FXRrG2iGaK2b9YY7-yHuxKWhbpsTQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Dear Moses developers,

I am trying to use the a new version of Moses, seems like things have
changed quite a bit and I have hard time finding an up-to-date
documentation. For debugging I used very small train/tune/test corpora (10
lines each).

First thing is running the following command produces a phrase table with
only 4 features:
train-model.perl --root-dir $root_dir --corpus $root_dir/$corpus_name --f
$src_lng --e $trg_lng --alignment grow-diag-final --lm 0:3:$LM
-external-bin-dir $external_bin_dir`;

Here is a snippet from a produced moses.iniPhraseDictionaryMemory
name=TranslationModel0 table-limit=20 *num-features=4
*path=/usr1/projects/mt_proj/mt_eval/baselines/fr-base-1-lats/model/phrase-table.gz
input-factor=0 output-factor=0

Second, I am trying to run tuning and decoding of lattices in plf format.
Can you point me to example commands and moses.ini for running mert and
decoding lattices with the new Moses?
So far I tried training and tuning on text files and decoding on lattices
because I could not figure out the right settings for tuning.
According to some old documentation I am supposed to convert the phrase
table to a binary format. Is it still needed?

When I ran it with the following command:
moses *-inputtype 2 -weight-i 0.62 -weight-l 12.5* -f $tune_dir/moses.ini <
$eval_dir/69.plf > $eval_dir/69.plf.out
I got an error:
*Don't mix old and new ini file format*
What is the new equivalent of weight-i and weight-l?

Without those parameters I get a Segmentation Fault with both a .gz and a
binary phrase table.

Could you help me figuring out the right settings?

Thanks in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20130901/75db8839/attachment-0001.htm

------------------------------

Message: 2
Date: Mon, 2 Sep 2013 09:52:27 +0800
From: "=?gb18030?B?1ebLrs7ez+M=?=" <907739598@qq.com>
Subject: [Moses-support] some problems with tuning step
To: "=?gb18030?B?bW9zZXMtc3VwcG9ydA==?=" <moses-support@mit.edu>
Message-ID: <tencent_52CDF2742EDCE33F664C65CF@qq.com>
Content-Type: text/plain; charset="gb18030"

Hi:
I have a problem with tuning step.
My commend is : /home/tempadmin/mtdir/moses/scripts/training/mert-moses.pl /home/tempadmin/hp/tuning/CH400-fenci-quanbanjiao.txt /home/tempadmin/hp/tuning/LD-bf /home/tempadmin/mtdir/moses/moses-chart-cmd/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/moses_chart /home/tempadmin/hp/test/model/moses.ini --working-dir /home/tempadmin/hp/tuning/mert --filtercmd \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'/home/tempadmin/mtdir/moses/scripts/training/filter-model-given-input.pl -hierarchical\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\' -rootdir /home/tempadmin/mtdir/moses/scripts/ --mertdir /home/tempadmin/mtdir/moses/bin/ --decoder-flags \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"-v 0\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\" &>/home/tempadmin/hp/tuning/mert.out






mert.out document is as follow:
Using SCRIPTS_ROOTDIR: /home/tempadmin/mtdir/moses/scripts/
filtering the phrase tables... Mon Sep 2 08:57:49 CST 2013
exec: /home/tempadmin/mtdir/moses/scripts/training/filter-model-given-input.pl -hierarchical ./filtered /home/tempadmin/hp/test/model/moses.ini /home/tempadmin/hp/tuning/CH400-fenci-quanbanjiao.txt
Executing: /home/tempadmin/mtdir/moses/scripts/training/filter-model-given-input.pl -hierarchical ./filtered /home/tempadmin/hp/test/model/moses.ini /home/tempadmin/hp/tuning/CH400-fenci-quanbanjiao.txt > filterphrases.out 2> filterphrases.err
Asking moses for feature names and values from filtered/moses.ini
Executing: /home/tempadmin/mtdir/moses/moses-chart-cmd/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/moses_chart -v 0 -config filtered/moses.ini -inputtype 0 -show-weights > ./features.list
/home/tempadmin/mtdir/moses/moses-chart-cmd/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi
line=UnknownWordPenalty
WEIGHT UnknownWordPenalty0=1.000,
line=WordPenalty
WEIGHT WordPenalty0=-1.000,
line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=5 path=/home/tempadmin/hp/tuning/mert/filtered/phrase-table.0-0.1.1.gz input-factor=0 output-factor=0
WEIGHT TranslationModel0=0.200,0.200,0.200,0.200,0.200,
line=PhraseDictionaryMemory name=TranslationModel1 num-features=1 path=/home/tempadmin/hp/test/model/glue-grammar input-factor=0 output-factor=0
WEIGHT TranslationModel1=1.000,
line=SRILM name=LM0 factor=0 path=/home/tempadmin/hp/lm/mg-lm.txt order=3
WEIGHT LM0=0.500,
/home/tempadmin/hp/lm/mg-lm.txt: line 5614: warning: non-zero probability for <unk> in closed-vocabulary LM
Start loading text SCFG phrase table. Moses format : [2.000] seconds
Exception: moses/Word.cpp:108 in void Moses::Word::CreateFromString(Moses::FactorDirection, const std::vector<long unsigned int>&, const StringPiece&, bool) threw StrayFactorException because `fit\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\'.
You have configured 1 factors but the word || contains factor delimiter | too many times.
Exit code: 1
Failed to run moses with the config filtered/moses.ini at /home/tempadmin/mtdir/moses/scripts/training/mert-moses.pl line 1271.






Thank you for your help.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20130902/93c39734/attachment-0001.htm

------------------------------

Message: 3
Date: Mon, 02 Sep 2013 06:59:54 +0000 (GMT)
From: ??? <roarman.seo@samsung.com>
Subject: [Moses-support] build a demonstration system
To: ?? <moses-support@MIT.EDU>
Message-ID: <5E.F9.13008.A6734225@epcpsbgx1.samsung.com>
Content-Type: text/plain; charset="us-ascii"

An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20130902/df3d7870/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 14036 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20130902/df3d7870/attachment.gif

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 83, Issue 1
********************************************

Tweet