Moses-support Digest, Vol 91, Issue 26

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. filter and binarize source-syntax models (joerg)
2. Re: tuning tree-based models (Hieu Hoang)
3. Re: filter and binarize source-syntax models (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Wed, 21 May 2014 09:00:13 +0200
From: joerg <tiedeman@gmail.com>
Subject: [Moses-support] filter and binarize source-syntax models
To: moses-support <moses-support@mit.edu>
Message-ID: <4FD6A9F2-7194-4413-BAA6-17368F3D85A3@gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi again,

I have more questions about source-syntax models. I tried to filter my rule-table using
filter-model-given-input.pl -Hierarchical filtered moses.ini input.trees
but this reduces the parsed input file to plain text when calling filter-rule-table.py
Is that still correct?

Another question is about binarization: What would be the correct call to binarize a source-syntax hierarchical model?

And I still haven't figured out how to run tuning and how to use filtering in tuning with hierarchical models.
Any help would be appreciated.

Thanks a lot!
J?rg



**********************************************************************************
J?rg Tiedemann http://stp.lingfil.uu.se/~joerg/



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140521/f98d2db5/attachment-0001.htm

------------------------------

Message: 2
Date: Wed, 21 May 2014 11:12:17 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] tuning tree-based models
To: Jorg Tiedemann <tiedeman@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbhptEDNJJi4GgTMdr82pYS6Y4d7nt-D_uJ9ZKP5yrYoNA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

hi jorg

are you using relative paths? use absolute paths if you are.

this is my tree-to-string mert command i ran last week, if it helps. I use
the EMS to run it so i don't usually look at the details
/home/s0565741/workspace/github/mosesdecoder.hieu/scripts/training/
mert-moses.pl

/home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/input.lc.7

/home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/reference.lc.7

/home/s0565741/workspace/github/mosesdecoder.hieu/bin/moses_chart

/home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/moses.filtered.ini.7
--nbest 100
--working-dir
/home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/tmp.7
--decoder-flags "-threads 8 -v 0 "
--rootdir /home/s0565741/workspace/github/mosesdecoder.hieu/scripts
-mertdir /home/s0565741/workspace/github/mosesdecoder.hieu/bin
-threads 10
--no-filter-phrase-table
--inputtype 3




On 20 May 2014 15:40, Jorg Tiedemann <tiedeman@gmail.com> wrote:

>
> I'm struggling a bit with tuning my tree-based models. I try to run MERT
> on a syntax-to-string model but I have problems setting the flags for
> mert-moses.pl correctly. Now I tried something like
>
> mert-moses.pl newstest2011.trees.es newstest2011.tok.true.fr moses_chart
> es-fr/model/moses.ini \
> --mertdir /path/to/mosesdecoder/bin \
> --nbest 100 \
> --working-dir es-fr/tuning \
> --no-filter-phrase-table \
> --inputtype 3 \
> --rootdir /path/to/mosesdecoder/scripts
>
> but after translating the devset in the first iteration my process stops
> with
> Couldn't open es-fr/tuning/extractor.sh for writing: file or directory
> does not exist
>
> The directory exists and the n-best list looks fine. Is there yet another
> flag that I have to add to the mert-moses call?
>
> By the way, it took me some time to figure out that I have to set
> inputtype to 3 to make the system accept parsed input. Maybe someone could
> add this to the documentation (if it is correct like this anyway).
>
ok. done
http://www.statmt.org/moses/?n=FactoredTraining.Tuning

>
> Another question: What would be the correct call to make mert-moses to
> filter the model before running? It doesn't work to add
> '-Hierarchical' to the filtercmd as mert-moses then complains that the
> command is not found on the system (it doesn't remove flags before checking
> for existence). Or is it better to binarize the rule-tables anyway?
>
i use the EMS, which filters and binarize the phrase table before tuning

>
> Thanks for your help!
>
> J?rg
>
>
>
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140521/9da220d7/attachment-0001.htm

------------------------------

Message: 3
Date: Wed, 21 May 2014 12:13:39 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] filter and binarize source-syntax models
To: joerg <tiedeman@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbj-42S0DrgV=QJrZyQ_todLWGVw4B4cteY19BWtS1V5uw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

On 21 May 2014 08:00, joerg <tiedeman@gmail.com> wrote:

> Hi again,
>
> I have more questions about source-syntax models. I tried to filter my
> rule-table using
> filter-model-given-input.pl -Hierarchical filtered moses.ini input.trees
> but this reduces the parsed input file to plain text when calling
> filter-rule-table.py
> Is that still correct?
>
I think so. The filtering script doesn't look at the non-terms so it may
let through more rules than is absolutely required. However, at least it's
not filtering out rules that are required

>
> Another question is about binarization: What would be the correct call to
> binarize a source-syntax hierarchical model?
>
same as hiero model
.../CreateOnDiskPt 1 1 4 100 2 pt pt.bin


> And I still haven't figured out how to run tuning and how to use filtering
> in tuning with hierarchical models.
> Any help would be appreciated.
>
> Thanks a lot!
> J?rg
>
>
>
>
> **********************************************************************************
> J?rg Tiedemann
> http://stp.lingfil.uu.se/~joerg/
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140521/625cc941/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 91, Issue 26
*********************************************

Tweet