Moses-support Digest, Vol 89, Issue 10

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Language Model Training failed (Barry Haddow)
2. Re: Language Model Training failed (Janez Kadivec)
3. Re: Incorrectly formated moses.ini with feature-overwrite?
(Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Wed, 05 Mar 2014 10:01:00 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Language Model Training failed
To: janez.kadivec@zop-cr.com
Cc: moses-support support <moses-support@mit.edu>
Message-ID: <5316F5DC.7040105@staffmail.ed.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Hi Janez

> In my opinion there are two things that need to be somehow described
> or corrected in the Moses baseline:
> 1. Notify the user about the location of the Giza++ utilities
> (mosesdecoder/tools or mosesdecoder/giza++) and need to rename the
> folders to the one used in command.

The instructions already ask you to copy the GIZA++ utilities to the
tools directory, you must have missed out that step.

> 2. Remove the last "&" char in the command, listed in the baseline.

The "&" is correct, it runs the command in the background, which is why
it returns instantly. This is quite a normal thing to do in a UNIX
environment.

cheers - Barry


On 05/03/14 09:43, Janez Kadivec wrote:
> Hi,
>
> thank you for your help. The added "yes" parameter resolved the
> situation. We are following the Moses baseline, published in the
> official Moses web site:
> http://www.statmt.org/moses/?n=moses.baseline
>
> Please correct the last command in the Language Model Traning section.
> The command is marked with red color.
> mkdir ~/lm
> cd ~/lm
> ~/irstlm/bin/add-start-end.sh \
> < ~/corpus/news-commentary-v8.fr-en.true.en \
> > news-commentary-v8.fr-en.sb.en
> export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh \
> -i news-commentary-v8.fr-en.sb.en \
> -t ./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en
> * ~/irstlm/bin/compile-lm --text news-commentary-v8.fr-en.lm.en.gz \
> news-commentary-v8.fr-en.arpa.en*
>
> ....................
> We followed the same baseline. In the Training the Translation System
> we found the next inconsistency:
> We installed the Moses and part of it was also Giza++ installed under
> the ...mosesdecoder\giza++1.0.7.
> We executed the following commands from the Training the Translation
> System section:
> mkdir ~/working
> cd ~/working
> nohup nice ~/mosesdecoder/scripts/training/train-model.perl -root-dir train \
> -corpus ~/corpus/news-commentary-v8.fr-en.clean \
> -f fr -e en -alignment grow-diag-final-and -reordering msd-bidirectional-fe \
> -lm 0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8 \
> -external-bin-dir ~/mosesdecoder/tools >& training.out &
> During the execution of the last command (marked with green color)
> here was an error saying that the mkcls utility is not found.
> It's not found because in the "initial" installation there is no tools
> subdirectory. We renamed the Giza++107 directory to "tools".
> The command was executed instantly with no results. So we removed the
> last character "&" from the command. It's working now for about half
> an hour. ;)
>
> In my opinion there are two things that need to be somehow described
> or corrected in the Moses baseline:
> 1. Notify the user about the location of the Giza++ utilities
> (mosesdecoder/tools or mosesdecoder/giza++) and need to rename the
> folders to the one used in command.
> 2. Remove the last "&" char in the command, listed in the baseline.
>
> Have a nice rest of the day.
> Janez
>
>
> Seth syggested you the right fix
>
> I just checked the IRSTLM documentation
> http://sourceforge.net/apps/mediawiki/irstlm/index.php?title=Estimating_gigantic_models
> and the correct notation is reported there.
>
> Could you please tell me from where do you get the "wrong" information
> So that I correct it.
>
>
> Nicola
> (on behalf of IRSTLM development team)
>
>
>
> On Mar 5, 2014, at 1:36 AM, Seth Jarrett wrote:
>
> First four commands were executed successfuly. The last one
> failed. Here
> is the result after entering the following command line:zzz <at>
> zzz-laptop:~/lm$ ~/moses/irstlm/bin/compile-lm --text
> news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
>
> inpfile: news-commentary-v8.fr-en.arpa.enloading up to the LM
> level 1000
> (if any)
> dub: 10000000Failed to open news-commentary-v8.fr-en.arpa.en!zzz <at>
> zzz-laptop:~/lm$ ----------------Where we made a mistake? I see the
> xxx.arpa.en listed as input file. Shouldn't be the xxx.arpa.en file an
> output file?Best regards!
>
>
> I was having the same problem when following the steps in the baseline
> instructions but I was able to get it to work by adding "yes"
> after --text.
>
> Try this:
>
> ~/moses/irstlm/bin/compile-lm --text yes
> news-commentary-v8.fr-en.lm.en.gz
> news-commentary-v8.fr-en.arpa.en
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> <mailto:Moses-support@mit.edu><mailto:Moses-support@mit.edu
> <mailto:Moses-support@mit.edu>>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> Call
> Send SMS
> Add to Skype
> You'll need Skype CreditFree via Skype
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support


--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



------------------------------

Message: 2
Date: Wed, 5 Mar 2014 11:33:32 +0100
From: Janez Kadivec <jankad@zop-cr.com>
Subject: Re: [Moses-support] Language Model Training failed
To: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Cc: moses-support support <moses-support@mit.edu>
Message-ID:
<CA+viJsfOr9vOZnPWTfZtSRnZiwa5Oix2WrvW+-s5SOGTC1uMSQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi,

I'm a beginnner in Linux... so I like to see things happening in the
foreground :)
If there is no need to change the command... don't do that... ;)

Obviously I overlooked and did not go through the section of copying the
giza++ utilities to the tools directory.

Thank you for your help.
Janez


On Wed, Mar 5, 2014 at 11:01 AM, Barry Haddow <bhaddow@staffmail.ed.ac.uk>wrote:

> Hi Janez
>
> In my opinion there are two things that need to be somehow described or
>> corrected in the Moses baseline:
>> 1. Notify the user about the location of the Giza++ utilities
>> (mosesdecoder/tools or mosesdecoder/giza++) and need to rename the folders
>> to the one used in command.
>>
>
> The instructions already ask you to copy the GIZA++ utilities to the tools
> directory, you must have missed out that step.
>
> 2. Remove the last "&" char in the command, listed in the baseline.
>>
>
> The "&" is correct, it runs the command in the background, which is why it
> returns instantly. This is quite a normal thing to do in a UNIX environment.
>
> cheers - Barry
>
>
> On 05/03/14 09:43, Janez Kadivec wrote:
>
>> Hi,
>>
>> thank you for your help. The added "yes" parameter resolved the
>> situation. We are following the Moses baseline, published in the official
>> Moses web site:
>> http://www.statmt.org/moses/?n=moses.baseline
>>
>> Please correct the last command in the Language Model Traning section.
>> The command is marked with red color.
>> mkdir ~/lm
>> cd ~/lm
>> ~/irstlm/bin/add-start-end.sh \
>> < ~/corpus/news-commentary-v8.fr-en.true.en \
>> > news-commentary-v8.fr-en.sb.en
>> export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh \
>> -i news-commentary-v8.fr-en.sb.en \
>> -t ./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en
>> * ~/irstlm/bin/compile-lm --text news-commentary-v8.fr-en.lm.en.gz \
>> news-commentary-v8.fr-en.arpa.en*
>>
>> ....................
>> We followed the same baseline. In the Training the Translation System we
>> found the next inconsistency:
>> We installed the Moses and part of it was also Giza++ installed under the
>> ...mosesdecoder\giza++1.0.7.
>> We executed the following commands from the Training the Translation
>> System section:
>> mkdir ~/working
>> cd ~/working
>> nohup nice ~/mosesdecoder/scripts/training/train-model.perl -root-dir
>> train \
>> -corpus ~/corpus/news-commentary-v8.fr-en.clean
>> \
>> -f fr -e en -alignment grow-diag-final-and -reordering
>> msd-bidirectional-fe \
>> -lm 0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8
>> \
>> -external-bin-dir ~/mosesdecoder/tools >& training.out &
>> During the execution of the last command (marked with green color) here
>> was an error saying that the mkcls utility is not found.
>> It's not found because in the "initial" installation there is no tools
>> subdirectory. We renamed the Giza++107 directory to "tools".
>> The command was executed instantly with no results. So we removed the
>> last character "&" from the command. It's working now for about half an
>> hour. ;)
>>
>> In my opinion there are two things that need to be somehow described or
>> corrected in the Moses baseline:
>> 1. Notify the user about the location of the Giza++ utilities
>> (mosesdecoder/tools or mosesdecoder/giza++) and need to rename the folders
>> to the one used in command.
>> 2. Remove the last "&" char in the command, listed in the baseline.
>>
>> Have a nice rest of the day.
>> Janez
>>
>>
>> Seth syggested you the right fix
>>
>> I just checked the IRSTLM documentation
>> http://sourceforge.net/apps/mediawiki/irstlm/index.php?
>> title=Estimating_gigantic_models
>> and the correct notation is reported there.
>>
>> Could you please tell me from where do you get the "wrong" information
>> So that I correct it.
>>
>>
>> Nicola
>> (on behalf of IRSTLM development team)
>>
>>
>>
>> On Mar 5, 2014, at 1:36 AM, Seth Jarrett wrote:
>>
>> First four commands were executed successfuly. The last one
>> failed. Here
>> is the result after entering the following command line:zzz <at>
>> zzz-laptop:~/lm$ ~/moses/irstlm/bin/compile-lm --text
>> news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
>>
>> inpfile: news-commentary-v8.fr-en.arpa.enloading up to the LM
>> level 1000
>> (if any)
>> dub: 10000000Failed to open news-commentary-v8.fr-en.arpa.en!zzz <at>
>> zzz-laptop:~/lm$ ----------------Where we made a mistake? I see the
>> xxx.arpa.en listed as input file. Shouldn't be the xxx.arpa.en file an
>> output file?Best regards!
>>
>>
>> I was having the same problem when following the steps in the baseline
>> instructions but I was able to get it to work by adding "yes"
>> after --text.
>>
>> Try this:
>>
>> ~/moses/irstlm/bin/compile-lm --text yes
>> news-commentary-v8.fr-en.lm.en.gz
>> news-commentary-v8.fr-en.arpa.en
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> <mailto:Moses-support@mit.edu><mailto:Moses-support@mit.edu
>> <mailto:Moses-support@mit.edu>>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>> Call
>> Send SMS
>> Add to Skype
>> You'll need Skype CreditFree via Skype
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
Call
Send SMS
Add to Skype
You'll need Skype CreditFree via Skype
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140305/1bc82807/attachment-0001.htm

------------------------------

Message: 3
Date: Wed, 5 Mar 2014 10:47:18 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Incorrectly formated moses.ini with
feature-overwrite?
To: Kenneth Heafield <moses@kheafield.com>
Cc: Moses <moses-support@mit.edu>
Message-ID:
<CAEKMkbh08PE1stE8uF3jd8dLCdUi3jYABqXXWipWgN44fDxu+Q@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

ah, i didn't know the mert-moses.pl script writes to the ini file with the
values from --decoder-flags.

I'm not gonna change mert-moses as it's likely break it.

I could change how the decoder -feature-overwrite works. At the moment,
it's
-feature-overwrite 'TranslationModel0 table-limit=100'
I can change it to
-feature-overwrite TranslationModel0 table-limit 100
i.e. don't have quotes or the equal sign.

This should work with mert-moses.pl. It will take some testing


On 4 March 2014 23:28, Kenneth Heafield <moses@kheafield.com> wrote:

> Dear Moses,
>
> As is the custom of Philipp's people, I ran
>
> $moses/scripts/training/mert-moses.pl $data/tuning/input.tc.*
> $data/tuning/reference.tc.* $moses/bin/moses $data/moses.ini --nbest 100
> --working-dir $mert --decoder-flags "-threads 32 -mp -search-algorithm
> 1 -cube-pruning-pop-limit 1000 -s 1000 -feature-overwrite
> 'TranslationModel0 table-limit=100' -max-trans-opt-per-coverage 100 -v 0
> " --rootdir $moses/scripts -mertdir $moses/bin --batch-mira
> --return-best-dev -maximum-iterations 25 --threads 32
> --no-filter-phrase-table --continue
>
> That failed part way through (it completed 12 iterations) and I got
> desperate enough to pick the best iteration by BLEU e.g.
> mert/run09.moses.ini . This file contains, in part, the following:
>
> [feature-overwrite]
> 'TranslationModel0
> table-limit=100'
>
> After editing the file to use test-set filtered evaluation set, I ran
> Moses and got this error:
>
> Exception: moses/StaticData.cpp:1153 in void
> Moses::StaticData::OverrideFeatures() threw util::Exception because
> `toks.size() <= 1'.
> Incorrect format for feature override: table-limit=100'
>
> It seems mert-moses.pl is generating moses.ini files that the decoder is
> unwilling to accept.
>
> Kenneth
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140305/1fb34471/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 89, Issue 10
*********************************************

Tweet