Moses-support Digest, Vol 121, Issue 13

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: error in mosesdecoder tuning (Barry Haddow)


----------------------------------------------------------------------

Message: 1
Date: Wed, 02 Nov 2016 12:28:59 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] error in mosesdecoder tuning
To: Hasan Sait ARSLAN <hasan.sait.arslan@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <5819DC0B.2020105@staffmail.ed.ac.uk>
Content-Type: text/plain; charset="utf-8"

Adding
-first-step 5 -last-step 5
will just run step 5 (phrase extraction)

On 02/11/16 12:01, Hasan Sait ARSLAN wrote:
> For instance, could you show me an example?
>
> Thanks,
>
> 2016-11-02 13:57 GMT+02:00 Barry Haddow <bhaddow@staffmail.ed.ac.uk
> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>
> Hi Hasan
>
> You can use the first-step and last-step arguments to run steps
> manually.
>
> cheers - Barry
>
>
> On 02/11/16 11:46, Hasan Sait ARSLAN wrote:
>> Sorry,
>>
>> as I see my model folder, I can say that Lexical translation part
>> also went well.
>>
>> so, the only steps I have to run manually are'
>>
>> * 5 Extract phrases
>> <http://www.statmt.org/moses/?n=FactoredTraining.ExtractPhrases>
>> * 6 Score phrases
>> <http://www.statmt.org/moses/?n=FactoredTraining.ScorePhrases>
>> * 7 Reordering model
>> <http://www.statmt.org/moses/?n=FactoredTraining.BuildReorderingModel>
>>
>> * 8 Generation model
>> <http://www.statmt.org/moses/?n=FactoredTraining.BuildGenerationModel>
>>
>> * 9 Configuration file
>> <http://www.statmt.org/moses/?n=FactoredTraining.CreateConfigurationFile>
>>
>>
>>
>>
>>
>>
>> 2016-11-02 13:43 GMT+02:00 Hasan Sait ARSLAN
>> <hasan.sait.arslan@gmail.com <mailto:hasan.sait.arslan@gmail.com>>:
>>
>> Hi Barry,
>>
>> Actually I know where I it stuck.
>>
>> As I check my train folder, I can see the all the files which
>> show that
>>
>> * 1 Prepare data
>> <http://www.statmt.org/moses/?n=FactoredTraining.PrepareData>
>>
>> * 2 Run GIZA
>> <http://www.statmt.org/moses/?n=FactoredTraining.RunGIZA>
>> * 3 Align words
>> <http://www.statmt.org/moses/?n=FactoredTraining.AlignWords>
>>
>> went well.
>>
>> So, is there any way to run the remaining
>>
>>
>> * 4 Lexical translation
>> <http://www.statmt.org/moses/?n=FactoredTraining.GetLexicalTranslationTable>
>>
>> * 5 Extract phrases
>> <http://www.statmt.org/moses/?n=FactoredTraining.ExtractPhrases>
>>
>> * 6 Score phrases
>> <http://www.statmt.org/moses/?n=FactoredTraining.ScorePhrases>
>>
>> * 7 Reordering model
>> <http://www.statmt.org/moses/?n=FactoredTraining.BuildReorderingModel>
>>
>> * 8 Generation model
>> <http://www.statmt.org/moses/?n=FactoredTraining.BuildGenerationModel>
>>
>> * 9 Configuration file
>> <http://www.statmt.org/moses/?n=FactoredTraining.CreateConfigurationFile>
>>
>>
>> steps manually?
>>
>>
>> 2016-11-02 13:27 GMT+02:00 Barry Haddow
>> <bhaddow@staffmail.ed.ac.uk <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>
>> Hi Hasan
>>
>> It's not really possible to debug without a log file, so
>> I think that unfortunately you should start again.
>>
>> I suggest you start with a much smaller corpus to try to
>> debug your setup. If you use 100,000 sentences it should
>> train within hours.
>>
>> cheers - Barry
>>
>>
>> On 02/11/16 11:13, Hasan Sait ARSLAN wrote:
>>> Hi Barry,
>>>
>>> Unfortunately I didn't keep the log file. Is it really a
>>> hopeless situation?
>>>
>>> 2016-11-02 13:10 GMT+02:00 Barry Haddow
>>> <bhaddow@staffmail.ed.ac.uk
>>> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>>
>>> Hi Hasan
>>>
>>> You should have run train_model.perl something like
>>> this:
>>>
>>> nohup nice /path/to/train_model.perl
>>> ARGS_FOR_TRAINING &> log &
>>>
>>> You just need the log file
>>>
>>> cheers - Barry
>>>
>>>
>>> On 02/11/16 11:00, Hasan Sait ARSLAN wrote:
>>>> Hi Barry,
>>>>
>>>> I don't have any idea about where it is saved.
>>>>
>>>> Cheers,
>>>>
>>>> 2016-11-02 12:55 GMT+02:00 Barry Haddow
>>>> <bhaddow@staffmail.ed.ac.uk
>>>> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>>>
>>>> Hi Hasan
>>>>
>>>> If your phrase table is empty, that would
>>>> explain why tuning didn't work. Something went
>>>> wrong earlier in the process. Could you post
>>>> your log file from train_model.perl,
>>>>
>>>> cheers - Barry
>>>>
>>>>
>>>> On 02/11/16 10:45, Hasan Sait ARSLAN wrote:
>>>>> Now currently I have a bigger problem. I have
>>>>> trained my data for 5 days, and the folder
>>>>> "train" is 39 G, but there is no any phrases
>>>>> saved on phrase table. It is annoying. What
>>>>> should I do now? I hope I won't need to rerun
>>>>> everything from the scratch
>>>>>
>>>>> 2016-11-02 12:26 GMT+02:00 Barry Haddow
>>>>> <bhaddow@staffmail.ed.ac.uk
>>>>> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>>>>
>>>>> Hi Hasan
>>>>>
>>>>> The error message should be written into
>>>>> filterphrases.err, inside your working
>>>>> directory,
>>>>>
>>>>> cheers - Barry
>>>>>
>>>>>
>>>>> On 02/11/16 10:02, Hasan Sait ARSLAN wrote:
>>>>>> Hi Hieu,
>>>>>>
>>>>>> I did in the way, you want. Plus, I am
>>>>>> sure the path and file names are
>>>>>> correctly spelt.
>>>>>> But still, I get the same error.
>>>>>>
>>>>>> 2016-11-01 21:41 GMT+02:00 Hasan Sait
>>>>>> ARSLAN <hasan.sait.arslan@gmail.com
>>>>>> <mailto:hasan.sait.arslan@gmail.com>>:
>>>>>>
>>>>>> Thank you Hieu, I will do it, and
>>>>>> give you feedback about it.
>>>>>>
>>>>>> Kind Regards,
>>>>>>
>>>>>> 2016-11-01 21:37 GMT+02:00 Hieu Hoang
>>>>>> <hieuhoang@gmail.com
>>>>>> <mailto:hieuhoang@gmail.com>>:
>>>>>>
>>>>>> the command you execute looks ok.
>>>>>> the problem is likely to be with
>>>>>> your using tuning data.
>>>>>>
>>>>>> You should look at your tuning
>>>>>> data, make sure you've spelt the
>>>>>> path and filename correctly, use
>>>>>> a very small subset (eg. start
>>>>>> with 1 parallel sentence) and
>>>>>> increase the number of sentences
>>>>>> until you find the problem
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 01/11/2016 12:30, Hasan Sait
>>>>>> ARSLAN wrote:
>>>>>>> Hello,
>>>>>>>
>>>>>>> I have trained dataset for 5
>>>>>>> days, and for 2-3 days I am
>>>>>>> dealing with the bug in tuning,
>>>>>>> but still couldn't solve the
>>>>>>> problem.
>>>>>>>
>>>>>>> I use the following command to run:
>>>>>>>
>>>>>>> */home/sait/mosesdecoder/scripts/training/mert-moses.pl
>>>>>>> <http://mert-moses.pl>
>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>>> <http://dev.clean.tr>
>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
>>>>>>> /home/sait/mosesdecoder/bin/moses /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>>> --working-dir
>>>>>>> /home/sait/Kairit/Task3/working_tr-en/mert
>>>>>>> --decoder-flags "--threads 32"
>>>>>>>
>>>>>>> *
>>>>>>> Then I get the following error:
>>>>>>>
>>>>>>> *Using SCRIPTS_ROOTDIR:
>>>>>>> /home/sait/mosesdecoder/scripts
>>>>>>> Assuming
>>>>>>> --mertdir=/home/sait/mosesdecoder/bin
>>>>>>> filtering the phrase tables... T
>>>>>>> nov 1 19:17:39 EET 2016
>>>>>>> exec:
>>>>>>> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>>> <http://filter-model-given-input.pl>
>>>>>>> ./filtered
>>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>>> <http://dev.clean.tr>
>>>>>>> Executing:
>>>>>>> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>>> <http://filter-model-given-input.pl>
>>>>>>> ./filtered
>>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>>> <http://dev.clean.tr> >
>>>>>>> filterphrases.out 2>
>>>>>>> filterphrases.err
>>>>>>> Exit code: 255
>>>>>>> ERROR: Failed to run
>>>>>>> '/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>>> <http://filter-model-given-input.pl>
>>>>>>> ./filtered
>>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>>> <http://dev.clean.tr>'. at
>>>>>>> /home/sait/mosesdecoder/scripts/training/mert-moses.pl
>>>>>>> <http://mert-moses.pl> line 1748.
>>>>>>>
>>>>>>> *
>>>>>>> I have searched over the
>>>>>>> internet about the error, but
>>>>>>> unfortunately I couldn't find
>>>>>>> any beneficial solution.
>>>>>>>
>>>>>>> It is so annoying, and eats my
>>>>>>> time, please help me to solve my
>>>>>>> problem.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Moses-support mailing list
>>>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Moses-support mailing list
>>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>
>>>>>
>>>>>
>>>>> The University of Edinburgh is a
>>>>> charitable body, registered in
>>>>> Scotland, with registration number SC005336.
>>>>>
>>>>>
>>>>
>>>>
>>>> The University of Edinburgh is a charitable
>>>> body, registered in
>>>> Scotland, with registration number SC005336.
>>>>
>>>>
>>>
>>>
>>> The University of Edinburgh is a charitable body,
>>> registered in
>>> Scotland, with registration number SC005336.
>>>
>>>
>>
>>
>> The University of Edinburgh is a charitable body,
>> registered in
>> Scotland, with registration number SC005336.
>>
>>
>>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161102/92c12137/attachment.html
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161102/92c12137/attachment.pl

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 121, Issue 13
**********************************************

Tweet