Moses-support Digest, Vol 91, Issue 27

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Hypothesis stages (Lakshya)
2. Re: tuning tree-based models (Jorg Tiedemann)
3. Re: tuning tree-based models (Kenneth Heafield)
4. Re: tuning tree-based models (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Wed, 21 May 2014 17:42:00 +0530
From: Lakshya <elekshman@gmail.com>
Subject: [Moses-support] Hypothesis stages
To: moses-support <moses-support@mit.edu>, Hieu Hoang
<hieuhoang@gmail.com>, Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Message-ID:
<CANiOhcPYVgftA1NQMGc-qGKM99u1+fAN6UbpALwt+tcqfkvgyw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

Could you please help me to get the steps by which the highest probability
translation is arrived at: all the intermediate 1-word, 2-words, 3-words
... hypotheses. Through v3 I am getting 1-word only.


Expecting your help.

Regards
lakshya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140521/9738c615/attachment-0001.htm

------------------------------

Message: 2
Date: Wed, 21 May 2014 14:47:54 +0200
From: Jorg Tiedemann <tiedeman@gmail.com>
Subject: Re: [Moses-support] tuning tree-based models
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <CCF429AB-AF33-4EA9-94C5-7AC054F879EF@gmail.com>
Content-Type: text/plain; charset="windows-1252"


oh man, now I feel stupid. of course, I used a relative path for the workdir and mert didn't like.
I thought I wouldn't make this mistake anymore ?
Thanks for helping!

J?rg





On May 21, 2014, at 12:12 PM, Hieu Hoang wrote:

> hi jorg
>
> are you using relative paths? use absolute paths if you are.
>
> this is my tree-to-string mert command i ran last week, if it helps. I use the EMS to run it so i don't usually look at the details
> /home/s0565741/workspace/github/mosesdecoder.hieu/scripts/training/mert-moses.pl
> /home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/input.lc.7
> /home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/reference.lc.7
> /home/s0565741/workspace/github/mosesdecoder.hieu/bin/moses_chart
> /home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/moses.filtered.ini.7
> --nbest 100
> --working-dir /home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/tmp.7
> --decoder-flags "-threads 8 -v 0 "
> --rootdir /home/s0565741/workspace/github/mosesdecoder.hieu/scripts
> -mertdir /home/s0565741/workspace/github/mosesdecoder.hieu/bin
> -threads 10
> --no-filter-phrase-table
> --inputtype 3
>
>
>
>
> On 20 May 2014 15:40, Jorg Tiedemann <tiedeman@gmail.com> wrote:
>
> I'm struggling a bit with tuning my tree-based models. I try to run MERT on a syntax-to-string model but I have problems setting the flags for mert-moses.pl correctly. Now I tried something like
>
> mert-moses.pl newstest2011.trees.es newstest2011.tok.true.fr moses_chart
> es-fr/model/moses.ini \
> --mertdir /path/to/mosesdecoder/bin \
> --nbest 100 \
> --working-dir es-fr/tuning \
> --no-filter-phrase-table \
> --inputtype 3 \
> --rootdir /path/to/mosesdecoder/scripts
>
> but after translating the devset in the first iteration my process stops with
> Couldn't open es-fr/tuning/extractor.sh for writing: file or directory does not exist
>
> The directory exists and the n-best list looks fine. Is there yet another flag that I have to add to the mert-moses call?
>
> By the way, it took me some time to figure out that I have to set inputtype to 3 to make the system accept parsed input. Maybe someone could add this to the documentation (if it is correct like this anyway).
> ok. done
> http://www.statmt.org/moses/?n=FactoredTraining.Tuning
>
> Another question: What would be the correct call to make mert-moses to filter the model before running? It doesn't work to add
> '-Hierarchical' to the filtercmd as mert-moses then complains that the command is not found on the system (it doesn't remove flags before checking for existence). Or is it better to binarize the rule-tables anyway?
> i use the EMS, which filters and binarize the phrase table before tuning
>
> Thanks for your help!
>
> J?rg
>
>
>
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140521/6ceaff9a/attachment-0001.htm

------------------------------

Message: 3
Date: Wed, 21 May 2014 08:55:21 -0700
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] tuning tree-based models
To: moses-support@mit.edu
Message-ID: <537CCC69.3090007@kheafield.com>
Content-Type: text/plain; charset=windows-1252

Hi,

Can we regard this as a bug in Moses? Any program should be able to
take relative paths. Preferably by never using cd or chdir. But
otherwise absolutizing as necessary.

Kenneth

On 05/21/14 05:47, Jorg Tiedemann wrote:
>
> oh man, now I feel stupid. of course, I used a relative path for the
> workdir and mert didn't like.
> I thought I wouldn't make this mistake anymore ?
> Thanks for helping!
>
> J?rg
>
>
>
>
>
> On May 21, 2014, at 12:12 PM, Hieu Hoang wrote:
>
>> hi jorg
>>
>> are you using relative paths? use absolute paths if you are.
>>
>> this is my tree-to-string mert command i ran last week, if it helps. I
>> use the EMS to run it so i don't usually look at the details
>>
>> /home/s0565741/workspace/github/mosesdecoder.hieu/scripts/training/mert-moses.pl
>> <http://mert-moses.pl/>
>>
>> /home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/input.lc.7
>>
>>
>> /home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/reference.lc.7
>>
>> /home/s0565741/workspace/github/mosesdecoder.hieu/bin/moses_chart
>>
>> /home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/moses.filtered.ini.7
>> --nbest 100
>> --working-dir
>> /home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/tmp.7
>> --decoder-flags "-threads 8 -v 0 "
>> --rootdir
>> /home/s0565741/workspace/github/mosesdecoder.hieu/scripts
>> -mertdir /home/s0565741/workspace/github/mosesdecoder.hieu/bin
>> -threads 10
>> --no-filter-phrase-table
>> --inputtype 3
>>
>>
>>
>>
>> On 20 May 2014 15:40, Jorg Tiedemann <tiedeman@gmail.com
>> <mailto:tiedeman@gmail.com>> wrote:
>>
>>
>> I'm struggling a bit with tuning my tree-based models. I try to
>> run MERT on a syntax-to-string model but I have problems setting
>> the flags for mert-moses.pl <http://mert-moses.pl/> correctly. Now
>> I tried something like
>>
>> mert-moses.pl <http://mert-moses.pl/> newstest2011.trees.es
>> <http://newstest2011.trees.es/> newstest2011.tok.true.fr
>> <http://newstest2011.tok.true.fr/> moses_chart
>> es-fr/model/moses.ini \
>> --mertdir /path/to/mosesdecoder/bin \
>> --nbest 100 \
>> --working-dir es-fr/tuning \
>> --no-filter-phrase-table \
>> --inputtype 3 \
>> --rootdir /path/to/mosesdecoder/scripts
>>
>> but after translating the devset in the first iteration my process
>> stops with
>> Couldn't open es-fr/tuning/extractor.sh for writing: file or
>> directory does not exist
>>
>> The directory exists and the n-best list looks fine. Is there yet
>> another flag that I have to add to the mert-moses call?
>>
>> By the way, it took me some time to figure out that I have to set
>> inputtype to 3 to make the system accept parsed input. Maybe
>> someone could add this to the documentation (if it is correct like
>> this anyway).
>>
>> ok. done
>> http://www.statmt.org/moses/?n=FactoredTraining.Tuning
>>
>>
>> Another question: What would be the correct call to make
>> mert-moses to filter the model before running? It doesn't work to add
>> '-Hierarchical' to the filtercmd as mert-moses then complains that
>> the command is not found on the system (it doesn't remove flags
>> before checking for existence). Or is it better to binarize the
>> rule-tables anyway?
>>
>> i use the EMS, which filters and binarize the phrase table before tuning
>>
>>
>> Thanks for your help!
>>
>> J?rg
>>
>>
>>
>>
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>>
>> --
>> Hieu Hoang
>> Research Associate
>> University of Edinburgh
>> http://www.hoang.co.uk/hieu
>>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

Message: 4
Date: Wed, 21 May 2014 17:04:49 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] tuning tree-based models
To: Kenneth Heafield <moses@kheafield.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbj9bPBk7LXaHHANVRdsn7jdB8CDn=e-gU-5LhEdU0ZCug@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

someone tried to fix a similar problem with spaces in path names.
Eventually, they had to give up and I had undo the changes

https://github.com/moses-smt/mosesdecoder/commit/b6094736455575b0d6f79bfaf08456fdfabe113d

try it if you like.I think it's a lot of hassle and we can't really police
it to prevent it happening again



On 21 May 2014 16:55, Kenneth Heafield <moses@kheafield.com> wrote:

> Hi,
>
> Can we regard this as a bug in Moses? Any program should be able
> to
> take relative paths. Preferably by never using cd or chdir. But
> otherwise absolutizing as necessary.
>
> Kenneth
>
> On 05/21/14 05:47, Jorg Tiedemann wrote:
> >
> > oh man, now I feel stupid. of course, I used a relative path for the
> > workdir and mert didn't like.
> > I thought I wouldn't make this mistake anymore ?
> > Thanks for helping!
> >
> > J?rg
> >
> >
> >
> >
> >
> > On May 21, 2014, at 12:12 PM, Hieu Hoang wrote:
> >
> >> hi jorg
> >>
> >> are you using relative paths? use absolute paths if you are.
> >>
> >> this is my tree-to-string mert command i ran last week, if it helps. I
> >> use the EMS to run it so i don't usually look at the details
> >>
> >> /home/s0565741/workspace/github/mosesdecoder.hieu/scripts/training/
> mert-moses.pl
> >> <http://mert-moses.pl/>
> >>
> >>
> /home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/input.lc.7
> >>
> >>
> >>
> /home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/reference.lc.7
> >>
> >>
> /home/s0565741/workspace/github/mosesdecoder.hieu/bin/moses_chart
> >>
> >>
> /home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/moses.filtered.ini.7
> >> --nbest 100
> >> --working-dir
> >> /home/s0565741/workspace/experiment/europarl/en-fr-mixed/tuning/tmp.7
> >> --decoder-flags "-threads 8 -v 0 "
> >> --rootdir
> >> /home/s0565741/workspace/github/mosesdecoder.hieu/scripts
> >> -mertdir /home/s0565741/workspace/github/mosesdecoder.hieu/bin
> >> -threads 10
> >> --no-filter-phrase-table
> >> --inputtype 3
> >>
> >>
> >>
> >>
> >> On 20 May 2014 15:40, Jorg Tiedemann <tiedeman@gmail.com
> >> <mailto:tiedeman@gmail.com>> wrote:
> >>
> >>
> >> I'm struggling a bit with tuning my tree-based models. I try to
> >> run MERT on a syntax-to-string model but I have problems setting
> >> the flags for mert-moses.pl <http://mert-moses.pl/> correctly. Now
> >> I tried something like
> >>
> >> mert-moses.pl <http://mert-moses.pl/> newstest2011.trees.es
> >> <http://newstest2011.trees.es/> newstest2011.tok.true.fr
> >> <http://newstest2011.tok.true.fr/> moses_chart
> >> es-fr/model/moses.ini \
> >> --mertdir /path/to/mosesdecoder/bin \
> >> --nbest 100 \
> >> --working-dir es-fr/tuning \
> >> --no-filter-phrase-table \
> >> --inputtype 3 \
> >> --rootdir /path/to/mosesdecoder/scripts
> >>
> >> but after translating the devset in the first iteration my process
> >> stops with
> >> Couldn't open es-fr/tuning/extractor.sh for writing: file or
> >> directory does not exist
> >>
> >> The directory exists and the n-best list looks fine. Is there yet
> >> another flag that I have to add to the mert-moses call?
> >>
> >> By the way, it took me some time to figure out that I have to set
> >> inputtype to 3 to make the system accept parsed input. Maybe
> >> someone could add this to the documentation (if it is correct like
> >> this anyway).
> >>
> >> ok. done
> >> http://www.statmt.org/moses/?n=FactoredTraining.Tuning
> >>
> >>
> >> Another question: What would be the correct call to make
> >> mert-moses to filter the model before running? It doesn't work to
> add
> >> '-Hierarchical' to the filtercmd as mert-moses then complains that
> >> the command is not found on the system (it doesn't remove flags
> >> before checking for existence). Or is it better to binarize the
> >> rule-tables anyway?
> >>
> >> i use the EMS, which filters and binarize the phrase table before tuning
> >>
> >>
> >> Thanks for your help!
> >>
> >> J?rg
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Moses-support mailing list
> >> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> >> http://mailman.mit.edu/mailman/listinfo/moses-support
> >>
> >>
> >>
> >>
> >> --
> >> Hieu Hoang
> >> Research Associate
> >> University of Edinburgh
> >> http://www.hoang.co.uk/hieu
> >>
> >
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140521/ca88fe3b/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 91, Issue 27
*********************************************

Tweet