Moses-support Digest, Vol 105, Issue 61

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: EMS help (Vincent Nguyen)
2. Re: EMS help (Barry Haddow)


----------------------------------------------------------------------

Message: 1
Date: Tue, 28 Jul 2015 15:09:30 +0200
From: Vincent Nguyen <vnguyen@neuf.fr>
Subject: Re: [Moses-support] EMS help
To: Barry Haddow <bhaddow@staffmail.ed.ac.uk>, moses-support
<moses-support@mit.edu>
Message-ID: <55B77F0A.6070904@neuf.fr>
Content-Type: text/plain; charset="windows-1252"

same error:

#!/bin/bash
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
cd /home/moses/working
echo 'starting at '`date`' on '`hostname`
mkdir -p /home/moses/working/training
mkdir -p /home/moses/working/model
/home/moses/mosesdecoder/scripts/training/binarize-model.perl
/home/moses/working/model/moses.ini.5
/home/moses/working/model/moses.bin.ini.6 -Binarizer
"/home/moses/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2"

echo 'finished at '`date`
touch /home/moses/working/steps/6/TRAINING_binarize-config.6.DONE




Le 28/07/2015 14:47, Barry Haddow a ?crit :
> Hi Vincent
>
> It could be a bug. Could you edit
> mosesdecoder/scripts/ems/experiment.meta and change the line:
>
> template: $binarize-all IN OUT -Binarizer $ttable-binarizer
>
> to
>
> template: $binarize-all IN OUT -Binarizer "$ttable-binarizer"
>
> Note that I have added quotes. Then you'll have to delete the most
> recent run, and re-run experiment.perl. If it works, fine. If it
> doesn't, could you post the steps/6/TRAINING_binarize-config.6 script
> (hopefully I got the name right - you may need to change the number)
>
> cheers - Barry
>
>
> On 28/07/15 13:11, Vincent Nguyen wrote:
>> I know but this is what I have in my config.basic now:
>> # conversion of rule table into binary on-disk format
>> ttable-binarizer = "$moses-bin-dir/CreateOnDiskPt 1 1 4 100 2"
>> binarize-all = $moses-script-dir/training/binarize-model.perl
>>
>> I don't where else I can add the 5 arguments or if I need to
>> reference ttable-binarizer somewhere
>>
>>
>> Le 28/07/2015 13:49, Barry Haddow a ?crit :
>>> Hi Vincent
>>>
>>> If you look at the error log, you will see:
>>>
>>>> Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors
>>>> numTargetFactors numScores tableLimit sortScoreIndex inputPath
>>>> outputPath
>>> You are missing the first 5 arguments to CreateOnDiskPt, as given in
>>> config.basic.
>>>
>>> cheers - Barry
>>>
>>> On 28/07/15 12:37, Vincent Nguyen wrote:
>>>> I don't know why but the binarize crashes see below ....
>>>>
>>>>>
>>>>>> in my working directory I have 2 subdir,
>>>>>> "tuning" with inside moses.filtered.ini.5 moses.ini.5
>>>>>> moses.tuned.ini.5
>>>>>> and
>>>>>> "model" with inside moses.ini.5 (apparently this one does not
>>>>>> have the
>>>>>> tuned weights)
>>>>>>
>>>>>> those in the tuning subdir : the "tuned" one moses.tuned.ini.5
>>>>>> generated
>>>>>> after the moses.ini.5 seems to point on phrase-table.5.gz not
>>>>>> binarized
>>>>>> and the moses.5.ini seem to point on the binarized within
>>>>>> tuning/filtered.5/...
>>>>>> unclear to me on which one I should use.
>>>>> If you run EMS, there will be a filtered ini file inside the
>>>>> evaluation directory which can be used to translate the test set
>>>>> using the tuned weights. However this model is filtered for the
>>>>> test set, so you cannot use it on other sentences.
>>>>>
>>>>> If you want the full model binarised, then you should add:
>>>>>
>>>>> binarize-all = $moses-script-dir/training/binarize-model.perl
>>>>>
>>>>> to the [GENERAL] section of the EMS config and rerun EMS. In this
>>>>> case the moses.tuned.ini in tuning can be used to translate any
>>>>> sentences.
>>>>>
>>>>
>>>>
>>>> Executing:
>>>> /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>> /home/moses/working/model/moses.bin.ini.6.tables
>>>> /home/moses/working/model/moses.ini.5 /dev/null -nofilter
>>>> -Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt
>>>> Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables
>>>> Stripping XML...
>>>> Executing:
>>>> /home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl
>>>> < /dev/null >
>>>> /home/moses/working/model/moses.bin.ini.6.tables/input.34384
>>>> pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4
>>>> path=/home/moses/working/model/phrase-table.5 input-factor=0
>>>> output-factor=0
>>>> Considering factor 0
>>>> ro:LexicalReordering name=LexicalReordering0 num-features=6
>>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>>>> path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz
>>>>
>>>> Considering factor 0
>>>> Filtering files...
>>>> filtering /home/moses/working/model/phrase-table.5 ->
>>>> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1...
>>>>
>>>> Executing: ln -s /home/moses/working/model/phrase-table.5.gz
>>>> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz
>>>>
>>>> binarizing...
>>>> Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt
>>>> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz
>>>> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.bin
>>>>
>>>> Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt numSourceFactors
>>>> numTargetFactors numScores tableLimit sortScoreIndex inputPath
>>>> outputPath
>>>> Exit code: 1
>>>> Can't binarize at
>>>> /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>> line 417.
>>>> Exit code: 1
>>>> binarising failed at
>>>> /home/moses/mosesdecoder/scripts/training/binarize-model.perl line 43.
>>>>
>>>
>>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150728/bf951d83/attachment-0001.htm

------------------------------

Message: 2
Date: Tue, 28 Jul 2015 14:44:38 +0100
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] EMS help
To: Vincent Nguyen <vnguyen@neuf.fr>, moses-support
<moses-support@mit.edu>
Message-ID: <55B78746.7090003@staffmail.ed.ac.uk>
Content-Type: text/plain; charset="windows-1252"

Hi Vincent


I think the quotes are getting stripped off further down the pipeline.
You could work around by changing to the compact phrase table. Or try
editing binarize-model.perl to change

safesystem("$RealBin/filter-model-given-input.pl $targetdir
$input_config /dev/null $hierarchical -nofilter -Binarizer $binarizer")
|| die "binarising failed";

to

safesystem("$RealBin/filter-model-given-input.pl $targetdir
$input_config /dev/null $hierarchical -nofilter -Binarizer
\"$binarizer\"") || die "binarising failed";

Note the escaped quotes around the $binarizer.

cheers - Barry

On 28/07/15 14:09, Vincent Nguyen wrote:
> same error:
>
> #!/bin/bash
> PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
> cd /home/moses/working
> echo 'starting at '`date`' on '`hostname`
> mkdir -p /home/moses/working/training
> mkdir -p /home/moses/working/model
> /home/moses/mosesdecoder/scripts/training/binarize-model.perl
> /home/moses/working/model/moses.ini.5
> /home/moses/working/model/moses.bin.ini.6 -Binarizer
> "/home/moses/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2"
>
> echo 'finished at '`date`
> touch /home/moses/working/steps/6/TRAINING_binarize-config.6.DONE
>
>
>
>
> Le 28/07/2015 14:47, Barry Haddow a ?crit :
>> Hi Vincent
>>
>> It could be a bug. Could you edit
>> mosesdecoder/scripts/ems/experiment.meta and change the line:
>>
>> template: $binarize-all IN OUT -Binarizer $ttable-binarizer
>>
>> to
>>
>> template: $binarize-all IN OUT -Binarizer "$ttable-binarizer"
>>
>> Note that I have added quotes. Then you'll have to delete the most
>> recent run, and re-run experiment.perl. If it works, fine. If it
>> doesn't, could you post the steps/6/TRAINING_binarize-config.6 script
>> (hopefully I got the name right - you may need to change the number)
>>
>> cheers - Barry
>>
>>
>> On 28/07/15 13:11, Vincent Nguyen wrote:
>>> I know but this is what I have in my config.basic now:
>>> # conversion of rule table into binary on-disk format
>>> ttable-binarizer = "$moses-bin-dir/CreateOnDiskPt 1 1 4 100 2"
>>> binarize-all = $moses-script-dir/training/binarize-model.perl
>>>
>>> I don't where else I can add the 5 arguments or if I need to
>>> reference ttable-binarizer somewhere
>>>
>>>
>>> Le 28/07/2015 13:49, Barry Haddow a ?crit :
>>>> Hi Vincent
>>>>
>>>> If you look at the error log, you will see:
>>>>
>>>>> Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt
>>>>> numSourceFactors numTargetFactors numScores tableLimit
>>>>> sortScoreIndex inputPath outputPath
>>>> You are missing the first 5 arguments to CreateOnDiskPt, as given
>>>> in config.basic.
>>>>
>>>> cheers - Barry
>>>>
>>>> On 28/07/15 12:37, Vincent Nguyen wrote:
>>>>> I don't know why but the binarize crashes see below ....
>>>>>
>>>>>>
>>>>>>> in my working directory I have 2 subdir,
>>>>>>> "tuning" with inside moses.filtered.ini.5 moses.ini.5
>>>>>>> moses.tuned.ini.5
>>>>>>> and
>>>>>>> "model" with inside moses.ini.5 (apparently this one does not
>>>>>>> have the
>>>>>>> tuned weights)
>>>>>>>
>>>>>>> those in the tuning subdir : the "tuned" one moses.tuned.ini.5
>>>>>>> generated
>>>>>>> after the moses.ini.5 seems to point on phrase-table.5.gz not
>>>>>>> binarized
>>>>>>> and the moses.5.ini seem to point on the binarized within
>>>>>>> tuning/filtered.5/...
>>>>>>> unclear to me on which one I should use.
>>>>>> If you run EMS, there will be a filtered ini file inside the
>>>>>> evaluation directory which can be used to translate the test set
>>>>>> using the tuned weights. However this model is filtered for the
>>>>>> test set, so you cannot use it on other sentences.
>>>>>>
>>>>>> If you want the full model binarised, then you should add:
>>>>>>
>>>>>> binarize-all = $moses-script-dir/training/binarize-model.perl
>>>>>>
>>>>>> to the [GENERAL] section of the EMS config and rerun EMS. In this
>>>>>> case the moses.tuned.ini in tuning can be used to translate any
>>>>>> sentences.
>>>>>>
>>>>>
>>>>>
>>>>> Executing:
>>>>> /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>> /home/moses/working/model/moses.bin.ini.6.tables
>>>>> /home/moses/working/model/moses.ini.5 /dev/null -nofilter
>>>>> -Binarizer /home/moses/mosesdecoder/bin/CreateOnDiskPt
>>>>> Executing: mkdir -p /home/moses/working/model/moses.bin.ini.6.tables
>>>>> Stripping XML...
>>>>> Executing:
>>>>> /home/moses/mosesdecoder/scripts/training/../generic/strip-xml.perl <
>>>>> /dev/null >
>>>>> /home/moses/working/model/moses.bin.ini.6.tables/input.34384
>>>>> pt:PhraseDictionaryMemory name=TranslationModel0 num-features=4
>>>>> path=/home/moses/working/model/phrase-table.5 input-factor=0
>>>>> output-factor=0
>>>>> Considering factor 0
>>>>> ro:LexicalReordering name=LexicalReordering0 num-features=6
>>>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>>>>> path=/home/moses/working/model/reordering-table.5.wbe-msd-bidirectional-fe.gz
>>>>>
>>>>> Considering factor 0
>>>>> Filtering files...
>>>>> filtering /home/moses/working/model/phrase-table.5 ->
>>>>> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1...
>>>>>
>>>>> Executing: ln -s /home/moses/working/model/phrase-table.5.gz
>>>>> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz
>>>>>
>>>>> binarizing...
>>>>> Executing: /home/moses/mosesdecoder/bin/CreateOnDiskPt
>>>>> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.gz
>>>>> /home/moses/working/model/moses.bin.ini.6.tables/phrase-table.0-0.1.1.bin
>>>>>
>>>>> Usage: /home/moses/mosesdecoder/bin/CreateOnDiskPt
>>>>> numSourceFactors numTargetFactors numScores tableLimit
>>>>> sortScoreIndex inputPath outputPath
>>>>> Exit code: 1
>>>>> Can't binarize at
>>>>> /home/moses/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>> line 417.
>>>>> Exit code: 1
>>>>> binarising failed at
>>>>> /home/moses/mosesdecoder/scripts/training/binarize-model.perl line
>>>>> 43.
>>>>>
>>>>
>>>>
>>>
>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150728/608e9ae6/attachment.htm
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150728/608e9ae6/attachment.bat

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 105, Issue 61
**********************************************

Tweet