Moses-support Digest, Vol 121, Issue 11

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: error in mosesdecoder tuning (Barry Haddow)


----------------------------------------------------------------------

Message: 1
Date: Wed, 02 Nov 2016 11:57:58 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] error in mosesdecoder tuning
To: Hasan Sait ARSLAN <hasan.sait.arslan@gmail.com>, moses-support
<moses-support@mit.edu>
Message-ID: <5819D4C6.1020900@staffmail.ed.ac.uk>
Content-Type: text/plain; charset="utf-8"

Hi Hasan

You can use the first-step and last-step arguments to run steps manually.

cheers - Barry

On 02/11/16 11:46, Hasan Sait ARSLAN wrote:
> Sorry,
>
> as I see my model folder, I can say that Lexical translation part also
> went well.
>
> so, the only steps I have to run manually are'
>
> * 5 Extract phrases
> <http://www.statmt.org/moses/?n=FactoredTraining.ExtractPhrases>
> * 6 Score phrases
> <http://www.statmt.org/moses/?n=FactoredTraining.ScorePhrases>
> * 7 Reordering model
> <http://www.statmt.org/moses/?n=FactoredTraining.BuildReorderingModel>
>
> * 8 Generation model
> <http://www.statmt.org/moses/?n=FactoredTraining.BuildGenerationModel>
>
> * 9 Configuration file
> <http://www.statmt.org/moses/?n=FactoredTraining.CreateConfigurationFile>
>
>
>
>
>
>
> 2016-11-02 13:43 GMT+02:00 Hasan Sait ARSLAN
> <hasan.sait.arslan@gmail.com <mailto:hasan.sait.arslan@gmail.com>>:
>
> Hi Barry,
>
> Actually I know where I it stuck.
>
> As I check my train folder, I can see the all the files which show
> that
>
> * 1 Prepare data
> <http://www.statmt.org/moses/?n=FactoredTraining.PrepareData>
> * 2 Run GIZA
> <http://www.statmt.org/moses/?n=FactoredTraining.RunGIZA>
> * 3 Align words
> <http://www.statmt.org/moses/?n=FactoredTraining.AlignWords>
>
> went well.
>
> So, is there any way to run the remaining
>
>
> * 4 Lexical translation
> <http://www.statmt.org/moses/?n=FactoredTraining.GetLexicalTranslationTable>
>
> * 5 Extract phrases
> <http://www.statmt.org/moses/?n=FactoredTraining.ExtractPhrases>
> * 6 Score phrases
> <http://www.statmt.org/moses/?n=FactoredTraining.ScorePhrases>
> * 7 Reordering model
> <http://www.statmt.org/moses/?n=FactoredTraining.BuildReorderingModel>
>
> * 8 Generation model
> <http://www.statmt.org/moses/?n=FactoredTraining.BuildGenerationModel>
>
> * 9 Configuration file
> <http://www.statmt.org/moses/?n=FactoredTraining.CreateConfigurationFile>
>
>
> steps manually?
>
>
> 2016-11-02 13:27 GMT+02:00 Barry Haddow
> <bhaddow@staffmail.ed.ac.uk <mailto:bhaddow@staffmail.ed.ac.uk>>:
>
> Hi Hasan
>
> It's not really possible to debug without a log file, so I
> think that unfortunately you should start again.
>
> I suggest you start with a much smaller corpus to try to debug
> your setup. If you use 100,000 sentences it should train
> within hours.
>
> cheers - Barry
>
>
> On 02/11/16 11:13, Hasan Sait ARSLAN wrote:
>> Hi Barry,
>>
>> Unfortunately I didn't keep the log file. Is it really a
>> hopeless situation?
>>
>> 2016-11-02 13:10 GMT+02:00 Barry Haddow
>> <bhaddow@staffmail.ed.ac.uk <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>
>> Hi Hasan
>>
>> You should have run train_model.perl something like this:
>>
>> nohup nice /path/to/train_model.perl ARGS_FOR_TRAINING &>
>> log &
>>
>> You just need the log file
>>
>> cheers - Barry
>>
>>
>> On 02/11/16 11:00, Hasan Sait ARSLAN wrote:
>>> Hi Barry,
>>>
>>> I don't have any idea about where it is saved.
>>>
>>> Cheers,
>>>
>>> 2016-11-02 12:55 GMT+02:00 Barry Haddow
>>> <bhaddow@staffmail.ed.ac.uk
>>> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>>
>>> Hi Hasan
>>>
>>> If your phrase table is empty, that would explain
>>> why tuning didn't work. Something went wrong earlier
>>> in the process. Could you post your log file from
>>> train_model.perl,
>>>
>>> cheers - Barry
>>>
>>>
>>> On 02/11/16 10:45, Hasan Sait ARSLAN wrote:
>>>> Now currently I have a bigger problem. I have
>>>> trained my data for 5 days, and the folder "train"
>>>> is 39 G, but there is no any phrases saved on
>>>> phrase table. It is annoying. What should I do now?
>>>> I hope I won't need to rerun everything from the
>>>> scratch
>>>>
>>>> 2016-11-02 12:26 GMT+02:00 Barry Haddow
>>>> <bhaddow@staffmail.ed.ac.uk
>>>> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>>>
>>>> Hi Hasan
>>>>
>>>> The error message should be written into
>>>> filterphrases.err, inside your working directory,
>>>>
>>>> cheers - Barry
>>>>
>>>>
>>>> On 02/11/16 10:02, Hasan Sait ARSLAN wrote:
>>>>> Hi Hieu,
>>>>>
>>>>> I did in the way, you want. Plus, I am sure
>>>>> the path and file names are correctly spelt.
>>>>> But still, I get the same error.
>>>>>
>>>>> 2016-11-01 21:41 GMT+02:00 Hasan Sait ARSLAN
>>>>> <hasan.sait.arslan@gmail.com
>>>>> <mailto:hasan.sait.arslan@gmail.com>>:
>>>>>
>>>>> Thank you Hieu, I will do it, and give you
>>>>> feedback about it.
>>>>>
>>>>> Kind Regards,
>>>>>
>>>>> 2016-11-01 21:37 GMT+02:00 Hieu Hoang
>>>>> <hieuhoang@gmail.com
>>>>> <mailto:hieuhoang@gmail.com>>:
>>>>>
>>>>> the command you execute looks ok. the
>>>>> problem is likely to be with your
>>>>> using tuning data.
>>>>>
>>>>> You should look at your tuning data,
>>>>> make sure you've spelt the path and
>>>>> filename correctly, use a very small
>>>>> subset (eg. start with 1 parallel
>>>>> sentence) and increase the number of
>>>>> sentences until you find the problem
>>>>>
>>>>>
>>>>>
>>>>> On 01/11/2016 12:30, Hasan Sait ARSLAN
>>>>> wrote:
>>>>>> Hello,
>>>>>>
>>>>>> I have trained dataset for 5 days,
>>>>>> and for 2-3 days I am dealing with
>>>>>> the bug in tuning, but still couldn't
>>>>>> solve the problem.
>>>>>>
>>>>>> I use the following command to run:
>>>>>>
>>>>>> */home/sait/mosesdecoder/scripts/training/mert-moses.pl
>>>>>> <http://mert-moses.pl>
>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>> <http://dev.clean.tr>
>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
>>>>>> /home/sait/mosesdecoder/bin/moses
>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>> --working-dir
>>>>>> /home/sait/Kairit/Task3/working_tr-en/mert
>>>>>> --decoder-flags "--threads 32"
>>>>>>
>>>>>> *
>>>>>> Then I get the following error:
>>>>>>
>>>>>> *Using SCRIPTS_ROOTDIR:
>>>>>> /home/sait/mosesdecoder/scripts
>>>>>> Assuming
>>>>>> --mertdir=/home/sait/mosesdecoder/bin
>>>>>> filtering the phrase tables... T
>>>>>> nov 1 19:17:39 EET 2016
>>>>>> exec:
>>>>>> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>> <http://filter-model-given-input.pl>
>>>>>> ./filtered
>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>> <http://dev.clean.tr>
>>>>>> Executing:
>>>>>> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>> <http://filter-model-given-input.pl>
>>>>>> ./filtered
>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>> <http://dev.clean.tr> >
>>>>>> filterphrases.out 2> filterphrases.err
>>>>>> Exit code: 255
>>>>>> ERROR: Failed to run
>>>>>> '/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>> <http://filter-model-given-input.pl>
>>>>>> ./filtered
>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>> <http://dev.clean.tr>'. at
>>>>>> /home/sait/mosesdecoder/scripts/training/mert-moses.pl
>>>>>> <http://mert-moses.pl> line 1748.
>>>>>>
>>>>>> *
>>>>>> I have searched over the internet
>>>>>> about the error, but unfortunately I
>>>>>> couldn't find any beneficial solution.
>>>>>>
>>>>>> It is so annoying, and eats my time,
>>>>>> please help me to solve my problem.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Moses-support mailing list
>>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>
>>>>
>>>>
>>>> The University of Edinburgh is a charitable
>>>> body, registered in
>>>> Scotland, with registration number SC005336.
>>>>
>>>>
>>>
>>>
>>> The University of Edinburgh is a charitable body,
>>> registered in
>>> Scotland, with registration number SC005336.
>>>
>>>
>>
>>
>> The University of Edinburgh is a charitable body,
>> registered in
>> Scotland, with registration number SC005336.
>>
>>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161102/9cdff2a2/attachment.html
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161102/9cdff2a2/attachment.pl

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 121, Issue 11
**********************************************

Tweet