Moses-support Digest, Vol 83, Issue 3

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Tuning and decoding of lattices in the new Moses. (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Mon, 2 Sep 2013 17:03:17 +0200
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Tuning and decoding of lattices in the
new Moses.
To: Yulia Tsvetkov <yulia.tsvetkov@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbj+0QtJKF9cGJe=__SvF=1RRs3m=-mt3K0xWKEa1=pTQw@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi Yulia


On 1 September 2013 22:46, Yulia Tsvetkov <yulia.tsvetkov@gmail.com> wrote:

> Dear Moses developers,
>
> I am trying to use the a new version of Moses, seems like things have
> changed quite a bit and I have hard time finding an up-to-date
> documentation. For debugging I used very small train/tune/test corpora (10
> lines each).
>
> First thing is running the following command produces a phrase table with
> only 4 features:
> train-model.perl --root-dir $root_dir --corpus $root_dir/$corpus_name --f
> $src_lng --e $trg_lng --alignment grow-diag-final --lm 0:3:$LM
> -external-bin-dir $external_bin_dir`;
>
> Here is a snippet from a produced moses.iniPhraseDictionaryMemory
> name=TranslationModel0 table-limit=20 *num-features=4 *path=/usr1/projects/mt_proj/mt_eval/baselines/fr-base-1-lats/model/phrase-table.gz
> input-factor=0 output-factor=0
>

Yes, the phrase-table now has 4 scores, instead of 5. The 5th score was a
constant 2.718. This has now moved into it's own feature function,
PhrasePenalty.

it save 3% of disk space, and i think is better for research. eg. create
better, non-constant phrase penalty feature functions, if we have 2 phrase
tables do we need just 1 phrase penalty? etc.


> Second, I am trying to run tuning and decoding of lattices in plf format.
> Can you point me to example commands and moses.ini for running mert and
> decoding lattices with the new Moses?
>
an example ini file for lattices can be seen here

https://github.com/moses-smt/moses-regression-tests/blob/master/tests/phrase.lattice-surface/moses.ini

Mert should run like it has always did. However, if you upgrade the
decoder, you should use the upgraded mert script too.

Decoding with lattice is exactly the same as for a sentence, except 2 things
1. inputtype=2. This can be on the command line of in the ini file, eg.
./moses -inputtype 2

or
[inputtype]
2

2. You should use the InputFeature feature function. This is the score
of the path through the lattice. You can see the InputFeature in the ini
file:
[feature]
....
InputFeature num-features=1 num-input-features=1 real-word-count=0

[weight]
...
InputFeature0 = 1

Before the refactoring, this was hacked into as an extra feature in the
phrase-table

>
> So far I tried training and tuning on text files and decoding on lattices
> because I could not figure out the right settings for tuning.
> According to some old documentation I am supposed to convert the phrase
> table to a binary format. Is it still needed?
>
You no longer need to convert it to binary format. It's good to convert to
binary format to save memory, but it is not required. Lattice decoding
works with all phrase-table implmentations now

>
> When I ran it with the following command:
> moses *-inputtype 2 -weight-i 0.62 -weight-l 12.5* -f $tune_dir/moses.ini
> < $eval_dir/69.plf > $eval_dir/69.plf.out
> I got an error:
> *Don't mix old and new ini file format*
> What is the new equivalent of weight-i and weight-l?
>

-weight-i 0.62
now becomes
-weight-overwrite 'InputFeature0= 0.62'

-weight-l 12.5
now becomes
-weight-overwrite 'LM0= 12.5'

The updated mert script should be doing this anyway.

>
> Without those parameters I get a Segmentation Fault with both a .gz and a
> binary phrase table.
>

if you're still having problems, give me your ini file and exact command
you're executing and i'll try and debug it

>
> Could you help me figuring out the right settings?
>
> Thanks in advance.
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20130902/2ce1574e/attachment-0001.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 83, Issue 3
********************************************

Tweet