Moses-support Digest, Vol 121, Issue 16

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: error in mosesdecoder tuning (Hasan Sait ARSLAN)


----------------------------------------------------------------------

Message: 1
Date: Wed, 2 Nov 2016 17:32:48 +0200
From: Hasan Sait ARSLAN <hasan.sait.arslan@gmail.com>
Subject: Re: [Moses-support] error in mosesdecoder tuning
To: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAJj7HD3zd_r1ZjbYj0ikNNYauxe1KFA2MV4Ft1PfQ5vKtK=bSQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Barry,

I have rerun training starting from step 5, and I kept log file. I have
checked log file, and it doesn't show any bug about creating phrase-table,
but phrase-table is still not being created. I am attaching log file, could
you please check it, and show me where is the problem?

Thanks,?
log.gz
<https://drive.google.com/file/d/0BxvJK3H5ZKsnYzJiZmhjUWI0Qlk/view?usp=drive_web>
?

2016-11-02 14:28 GMT+02:00 Barry Haddow <bhaddow@staffmail.ed.ac.uk>:

> Adding
> -first-step 5 -last-step 5
> will just run step 5 (phrase extraction)
>
>
> On 02/11/16 12:01, Hasan Sait ARSLAN wrote:
>
> For instance, could you show me an example?
>
> Thanks,
>
> 2016-11-02 13:57 GMT+02:00 Barry Haddow <bhaddow@staffmail.ed.ac.uk>:
>
>> Hi Hasan
>>
>> You can use the first-step and last-step arguments to run steps manually.
>>
>> cheers - Barry
>>
>>
>> On 02/11/16 11:46, Hasan Sait ARSLAN wrote:
>>
>> Sorry,
>>
>> as I see my model folder, I can say that Lexical translation part also
>> went well.
>>
>> so, the only steps I have to run manually are'
>>
>>
>> - 5 Extract phrases
>> <http://www.statmt.org/moses/?n=FactoredTraining.ExtractPhrases>
>> - 6 Score phrases
>> <http://www.statmt.org/moses/?n=FactoredTraining.ScorePhrases>
>> - 7 Reordering model
>> <http://www.statmt.org/moses/?n=FactoredTraining.BuildReorderingModel>
>> - 8 Generation model
>> <http://www.statmt.org/moses/?n=FactoredTraining.BuildGenerationModel>
>> - 9 Configuration file
>> <http://www.statmt.org/moses/?n=FactoredTraining.CreateConfigurationFile>
>>
>>
>>
>>
>>
>> 2016-11-02 13:43 GMT+02:00 Hasan Sait ARSLAN <hasan.sait.arslan@gmail.com
>> >:
>>
>>> Hi Barry,
>>>
>>> Actually I know where I it stuck.
>>>
>>> As I check my train folder, I can see the all the files which show that
>>>
>>>
>>> - 1 Prepare data
>>> <http://www.statmt.org/moses/?n=FactoredTraining.PrepareData>
>>> - 2 Run GIZA
>>> <http://www.statmt.org/moses/?n=FactoredTraining.RunGIZA>
>>> - 3 Align words
>>> <http://www.statmt.org/moses/?n=FactoredTraining.AlignWords>
>>>
>>> went well.
>>>
>>> So, is there any way to run the remaining
>>>
>>>
>>>
>>> - 4 Lexical translation
>>> <http://www.statmt.org/moses/?n=FactoredTraining.GetLexicalTranslationTable>
>>> - 5 Extract phrases
>>> <http://www.statmt.org/moses/?n=FactoredTraining.ExtractPhrases>
>>> - 6 Score phrases
>>> <http://www.statmt.org/moses/?n=FactoredTraining.ScorePhrases>
>>> - 7 Reordering model
>>> <http://www.statmt.org/moses/?n=FactoredTraining.BuildReorderingModel>
>>> - 8 Generation model
>>> <http://www.statmt.org/moses/?n=FactoredTraining.BuildGenerationModel>
>>> - 9 Configuration file
>>> <http://www.statmt.org/moses/?n=FactoredTraining.CreateConfigurationFile>
>>>
>>> steps manually?
>>>
>>> 2016-11-02 13:27 GMT+02:00 Barry Haddow <bhaddow@staffmail.ed.ac.uk>:
>>>
>>>> Hi Hasan
>>>>
>>>> It's not really possible to debug without a log file, so I think that
>>>> unfortunately you should start again.
>>>>
>>>> I suggest you start with a much smaller corpus to try to debug your
>>>> setup. If you use 100,000 sentences it should train within hours.
>>>>
>>>> cheers - Barry
>>>>
>>>>
>>>> On 02/11/16 11:13, Hasan Sait ARSLAN wrote:
>>>>
>>>> Hi Barry,
>>>>
>>>> Unfortunately I didn't keep the log file. Is it really a hopeless
>>>> situation?
>>>>
>>>> 2016-11-02 13:10 GMT+02:00 Barry Haddow <bhaddow@staffmail.ed.ac.uk>:
>>>>
>>>>> Hi Hasan
>>>>>
>>>>> You should have run train_model.perl something like this:
>>>>>
>>>>> nohup nice /path/to/train_model.perl ARGS_FOR_TRAINING &> log &
>>>>>
>>>>> You just need the log file
>>>>>
>>>>> cheers - Barry
>>>>>
>>>>>
>>>>> On 02/11/16 11:00, Hasan Sait ARSLAN wrote:
>>>>>
>>>>> Hi Barry,
>>>>>
>>>>> I don't have any idea about where it is saved.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> 2016-11-02 12:55 GMT+02:00 Barry Haddow <bhaddow@staffmail.ed.ac.uk>:
>>>>>
>>>>>> Hi Hasan
>>>>>>
>>>>>> If your phrase table is empty, that would explain why tuning didn't
>>>>>> work. Something went wrong earlier in the process. Could you post your log
>>>>>> file from train_model.perl,
>>>>>>
>>>>>> cheers - Barry
>>>>>>
>>>>>>
>>>>>> On 02/11/16 10:45, Hasan Sait ARSLAN wrote:
>>>>>>
>>>>>> Now currently I have a bigger problem. I have trained my data for 5
>>>>>> days, and the folder "train" is 39 G, but there is no any phrases saved on
>>>>>> phrase table. It is annoying. What should I do now? I hope I won't need to
>>>>>> rerun everything from the scratch
>>>>>>
>>>>>> 2016-11-02 12:26 GMT+02:00 Barry Haddow <bhaddow@staffmail.ed.ac.uk>:
>>>>>>
>>>>>>> Hi Hasan
>>>>>>>
>>>>>>> The error message should be written into filterphrases.err, inside
>>>>>>> your working directory,
>>>>>>>
>>>>>>> cheers - Barry
>>>>>>>
>>>>>>>
>>>>>>> On 02/11/16 10:02, Hasan Sait ARSLAN wrote:
>>>>>>>
>>>>>>> Hi Hieu,
>>>>>>>
>>>>>>> I did in the way, you want. Plus, I am sure the path and file names
>>>>>>> are correctly spelt.
>>>>>>> But still, I get the same error.
>>>>>>>
>>>>>>> 2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN <
>>>>>>> hasan.sait.arslan@gmail.com>:
>>>>>>>
>>>>>>>> Thank you Hieu, I will do it, and give you feedback about it.
>>>>>>>>
>>>>>>>> Kind Regards,
>>>>>>>>
>>>>>>>> 2016-11-01 21:37 GMT+02:00 Hieu Hoang <hieuhoang@gmail.com>:
>>>>>>>>
>>>>>>>>> the command you execute looks ok. the problem is likely to be with
>>>>>>>>> your using tuning data.
>>>>>>>>>
>>>>>>>>> You should look at your tuning data, make sure you've spelt the
>>>>>>>>> path and filename correctly, use a very small subset (eg. start with 1
>>>>>>>>> parallel sentence) and increase the number of sentences until you find the
>>>>>>>>> problem
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 01/11/2016 12:30, Hasan Sait ARSLAN wrote:
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I have trained dataset for 5 days, and for 2-3 days I am dealing
>>>>>>>>> with the bug in tuning, but still couldn't solve the problem.
>>>>>>>>>
>>>>>>>>> I use the following command to run:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> */home/sait/mosesdecoder/scripts/training/mert-moses.pl
>>>>>>>>> <http://mert-moses.pl> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>>>>> <http://dev.clean.tr> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
>>>>>>>>> /home/sait/mosesdecoder/bin/moses
>>>>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini --working-dir
>>>>>>>>> /home/sait/Kairit/Task3/working_tr-en/mert --decoder-flags "--threads 32" *
>>>>>>>>> Then I get the following error:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Using SCRIPTS_ROOTDIR: /home/sait/mosesdecoder/scripts Assuming
>>>>>>>>> --mertdir=/home/sait/mosesdecoder/bin filtering the phrase tables... T
>>>>>>>>> nov 1 19:17:39 EET 2016 exec:
>>>>>>>>> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>>>>> <http://filter-model-given-input.pl> ./filtered
>>>>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr <http://dev.clean.tr>
>>>>>>>>> Executing:
>>>>>>>>> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>>>>> <http://filter-model-given-input.pl> ./filtered
>>>>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr <http://dev.clean.tr> >
>>>>>>>>> filterphrases.out 2> filterphrases.err Exit code: 255 ERROR: Failed to run
>>>>>>>>> '/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>>>>> <http://filter-model-given-input.pl> ./filtered
>>>>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr <http://dev.clean.tr>'.
>>>>>>>>> at /home/sait/mosesdecoder/scripts/training/mert-moses.pl
>>>>>>>>> <http://mert-moses.pl> line 1748. *
>>>>>>>>> I have searched over the internet about the error, but
>>>>>>>>> unfortunately I couldn't find any beneficial solution.
>>>>>>>>>
>>>>>>>>> It is so annoying, and eats my time, please help me to solve my
>>>>>>>>> problem.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The University of Edinburgh is a charitable body, registered in
>>>>>>> Scotland, with registration number SC005336.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> The University of Edinburgh is a charitable body, registered in
>>>>>> Scotland, with registration number SC005336.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> The University of Edinburgh is a charitable body, registered in
>>>>> Scotland, with registration number SC005336.
>>>>>
>>>>>
>>>>
>>>>
>>>> The University of Edinburgh is a charitable body, registered in
>>>> Scotland, with registration number SC005336.
>>>>
>>>>
>>>
>>
>>
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>>
>>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161102/d75056f7/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 121, Issue 16
**********************************************

Tweet