Moses-support Digest, Vol 121, Issue 17

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: error in mosesdecoder tuning (Barry Haddow)


----------------------------------------------------------------------

Message: 1
Date: Wed, 02 Nov 2016 15:39:24 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] error in mosesdecoder tuning
To: Hasan Sait ARSLAN <hasan.sait.arslan@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <581A08AC.6090601@staffmail.ed.ac.uk>
Content-Type: text/plain; charset="utf-8"

Hi Hasan

This is full of messages about incorrect alignments. It looks like
something went wrong at an earlier stage. I would start again from the
beginning with a small data set, in order to debug your setup,

cheers - Barry

On 02/11/16 15:32, Hasan Sait ARSLAN wrote:
> Hi Barry,
>
> I have rerun training starting from step 5, and I kept log file. I
> have checked log file, and it doesn't show any bug about creating
> phrase-table, but phrase-table is still not being created. I am
> attaching log file, could you please check it, and show me where is
> the problem?
>
> Thanks,?
> log.gz
> <https://drive.google.com/file/d/0BxvJK3H5ZKsnYzJiZmhjUWI0Qlk/view?usp=drive_web>
> ?
>
> 2016-11-02 14:28 GMT+02:00 Barry Haddow <bhaddow@staffmail.ed.ac.uk
> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>
> Adding
> -first-step 5 -last-step 5
> will just run step 5 (phrase extraction)
>
>
> On 02/11/16 12:01, Hasan Sait ARSLAN wrote:
>> For instance, could you show me an example?
>>
>> Thanks,
>>
>> 2016-11-02 13:57 GMT+02:00 Barry Haddow
>> <bhaddow@staffmail.ed.ac.uk <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>
>> Hi Hasan
>>
>> You can use the first-step and last-step arguments to run
>> steps manually.
>>
>> cheers - Barry
>>
>>
>> On 02/11/16 11:46, Hasan Sait ARSLAN wrote:
>>> Sorry,
>>>
>>> as I see my model folder, I can say that Lexical translation
>>> part also went well.
>>>
>>> so, the only steps I have to run manually are'
>>>
>>> * 5 Extract phrases
>>> <http://www.statmt.org/moses/?n=FactoredTraining.ExtractPhrases>
>>>
>>> * 6 Score phrases
>>> <http://www.statmt.org/moses/?n=FactoredTraining.ScorePhrases>
>>>
>>> * 7 Reordering model
>>> <http://www.statmt.org/moses/?n=FactoredTraining.BuildReorderingModel>
>>>
>>> * 8 Generation model
>>> <http://www.statmt.org/moses/?n=FactoredTraining.BuildGenerationModel>
>>>
>>> * 9 Configuration file
>>> <http://www.statmt.org/moses/?n=FactoredTraining.CreateConfigurationFile>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2016-11-02 13:43 GMT+02:00 Hasan Sait ARSLAN
>>> <hasan.sait.arslan@gmail.com
>>> <mailto:hasan.sait.arslan@gmail.com>>:
>>>
>>> Hi Barry,
>>>
>>> Actually I know where I it stuck.
>>>
>>> As I check my train folder, I can see the all the files
>>> which show that
>>>
>>> * 1 Prepare data
>>> <http://www.statmt.org/moses/?n=FactoredTraining.PrepareData>
>>>
>>> * 2 Run GIZA
>>> <http://www.statmt.org/moses/?n=FactoredTraining.RunGIZA>
>>>
>>> * 3 Align words
>>> <http://www.statmt.org/moses/?n=FactoredTraining.AlignWords>
>>>
>>>
>>> went well.
>>>
>>> So, is there any way to run the remaining
>>>
>>>
>>> * 4 Lexical translation
>>> <http://www.statmt.org/moses/?n=FactoredTraining.GetLexicalTranslationTable>
>>>
>>> * 5 Extract phrases
>>> <http://www.statmt.org/moses/?n=FactoredTraining.ExtractPhrases>
>>>
>>> * 6 Score phrases
>>> <http://www.statmt.org/moses/?n=FactoredTraining.ScorePhrases>
>>>
>>> * 7 Reordering model
>>> <http://www.statmt.org/moses/?n=FactoredTraining.BuildReorderingModel>
>>>
>>> * 8 Generation model
>>> <http://www.statmt.org/moses/?n=FactoredTraining.BuildGenerationModel>
>>>
>>> * 9 Configuration file
>>> <http://www.statmt.org/moses/?n=FactoredTraining.CreateConfigurationFile>
>>>
>>>
>>> steps manually?
>>>
>>>
>>> 2016-11-02 13:27 GMT+02:00 Barry Haddow
>>> <bhaddow@staffmail.ed.ac.uk
>>> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>>
>>> Hi Hasan
>>>
>>> It's not really possible to debug without a log
>>> file, so I think that unfortunately you should start
>>> again.
>>>
>>> I suggest you start with a much smaller corpus to
>>> try to debug your setup. If you use 100,000
>>> sentences it should train within hours.
>>>
>>> cheers - Barry
>>>
>>>
>>> On 02/11/16 11:13, Hasan Sait ARSLAN wrote:
>>>> Hi Barry,
>>>>
>>>> Unfortunately I didn't keep the log file. Is it
>>>> really a hopeless situation?
>>>>
>>>> 2016-11-02 13:10 GMT+02:00 Barry Haddow
>>>> <bhaddow@staffmail.ed.ac.uk
>>>> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>>>
>>>> Hi Hasan
>>>>
>>>> You should have run train_model.perl something
>>>> like this:
>>>>
>>>> nohup nice /path/to/train_model.perl
>>>> ARGS_FOR_TRAINING &> log &
>>>>
>>>> You just need the log file
>>>>
>>>> cheers - Barry
>>>>
>>>>
>>>> On 02/11/16 11:00, Hasan Sait ARSLAN wrote:
>>>>> Hi Barry,
>>>>>
>>>>> I don't have any idea about where it is saved.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> 2016-11-02 12:55 GMT+02:00 Barry Haddow
>>>>> <bhaddow@staffmail.ed.ac.uk
>>>>> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>>>>
>>>>> Hi Hasan
>>>>>
>>>>> If your phrase table is empty, that would
>>>>> explain why tuning didn't work. Something
>>>>> went wrong earlier in the process. Could
>>>>> you post your log file from train_model.perl,
>>>>>
>>>>> cheers - Barry
>>>>>
>>>>>
>>>>> On 02/11/16 10:45, Hasan Sait ARSLAN wrote:
>>>>>> Now currently I have a bigger problem. I
>>>>>> have trained my data for 5 days, and the
>>>>>> folder "train" is 39 G, but there is no
>>>>>> any phrases saved on phrase table. It is
>>>>>> annoying. What should I do now? I hope I
>>>>>> won't need to rerun everything from the
>>>>>> scratch
>>>>>>
>>>>>> 2016-11-02 12:26 GMT+02:00 Barry Haddow
>>>>>> <bhaddow@staffmail.ed.ac.uk
>>>>>> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>>>>>
>>>>>> Hi Hasan
>>>>>>
>>>>>> The error message should be written
>>>>>> into filterphrases.err, inside your
>>>>>> working directory,
>>>>>>
>>>>>> cheers - Barry
>>>>>>
>>>>>>
>>>>>> On 02/11/16 10:02, Hasan Sait ARSLAN
>>>>>> wrote:
>>>>>>> Hi Hieu,
>>>>>>>
>>>>>>> I did in the way, you want. Plus, I
>>>>>>> am sure the path and file names are
>>>>>>> correctly spelt.
>>>>>>> But still, I get the same error.
>>>>>>>
>>>>>>> 2016-11-01 21:41 GMT+02:00 Hasan
>>>>>>> Sait ARSLAN
>>>>>>> <hasan.sait.arslan@gmail.com
>>>>>>> <mailto:hasan.sait.arslan@gmail.com>>:
>>>>>>>
>>>>>>> Thank you Hieu, I will do it,
>>>>>>> and give you feedback about it.
>>>>>>>
>>>>>>> Kind Regards,
>>>>>>>
>>>>>>> 2016-11-01 21:37 GMT+02:00 Hieu
>>>>>>> Hoang <hieuhoang@gmail.com
>>>>>>> <mailto:hieuhoang@gmail.com>>:
>>>>>>>
>>>>>>> the command you execute
>>>>>>> looks ok. the problem is
>>>>>>> likely to be with your using
>>>>>>> tuning data.
>>>>>>>
>>>>>>> You should look at your
>>>>>>> tuning data, make sure
>>>>>>> you've spelt the path and
>>>>>>> filename correctly, use a
>>>>>>> very small subset (eg. start
>>>>>>> with 1 parallel sentence)
>>>>>>> and increase the number of
>>>>>>> sentences until you find the
>>>>>>> problem
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 01/11/2016 12:30, Hasan
>>>>>>> Sait ARSLAN wrote:
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I have trained dataset for
>>>>>>>> 5 days, and for 2-3 days I
>>>>>>>> am dealing with the bug in
>>>>>>>> tuning, but still couldn't
>>>>>>>> solve the problem.
>>>>>>>>
>>>>>>>> I use the following command
>>>>>>>> to run:
>>>>>>>>
>>>>>>>> */home/sait/mosesdecoder/scripts/training/mert-moses.pl
>>>>>>>> <http://mert-moses.pl>
>>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>>>> <http://dev.clean.tr>
>>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.en
>>>>>>>> /home/sait/mosesdecoder/bin/moses
>>>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>>>> --working-dir
>>>>>>>> /home/sait/Kairit/Task3/working_tr-en/mert
>>>>>>>> --decoder-flags "--threads 32"
>>>>>>>>
>>>>>>>> *
>>>>>>>> Then I get the following error:
>>>>>>>>
>>>>>>>> *Using SCRIPTS_ROOTDIR:
>>>>>>>> /home/sait/mosesdecoder/scripts
>>>>>>>> Assuming
>>>>>>>> --mertdir=/home/sait/mosesdecoder/bin
>>>>>>>> filtering the phrase
>>>>>>>> tables... T nov 1
>>>>>>>> 19:17:39 EET 2016
>>>>>>>> exec:
>>>>>>>> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>>>> <http://filter-model-given-input.pl>
>>>>>>>> ./filtered
>>>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>>>> <http://dev.clean.tr>
>>>>>>>> Executing:
>>>>>>>> /home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>>>> <http://filter-model-given-input.pl>
>>>>>>>> ./filtered
>>>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>>>> <http://dev.clean.tr> >
>>>>>>>> filterphrases.out 2>
>>>>>>>> filterphrases.err
>>>>>>>> Exit code: 255
>>>>>>>> ERROR: Failed to run
>>>>>>>> '/home/sait/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>>>>>> <http://filter-model-given-input.pl>
>>>>>>>> ./filtered
>>>>>>>> /home/sait/Kairit/Task3/working_tr-en/train/model/moses.ini
>>>>>>>> /home/sait/Kairit/Task3/dataset_tur_en/dev.clean.tr
>>>>>>>> <http://dev.clean.tr>'. at
>>>>>>>> /home/sait/mosesdecoder/scripts/training/mert-moses.pl
>>>>>>>> <http://mert-moses.pl> line
>>>>>>>> 1748.
>>>>>>>>
>>>>>>>> *
>>>>>>>> I have searched over the
>>>>>>>> internet about the error,
>>>>>>>> but unfortunately I
>>>>>>>> couldn't find any
>>>>>>>> beneficial solution.
>>>>>>>>
>>>>>>>> It is so annoying, and eats
>>>>>>>> my time, please help me to
>>>>>>>> solve my problem.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Moses-support mailing list
>>>>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Moses-support mailing list
>>>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>
>>>>>>
>>>>>>
>>>>>> The University of Edinburgh is a
>>>>>> charitable body, registered in
>>>>>> Scotland, with registration number
>>>>>> SC005336.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> The University of Edinburgh is a
>>>>> charitable body, registered in
>>>>> Scotland, with registration number SC005336.
>>>>>
>>>>>
>>>>
>>>>
>>>> The University of Edinburgh is a charitable
>>>> body, registered in
>>>> Scotland, with registration number SC005336.
>>>>
>>>>
>>>
>>>
>>> The University of Edinburgh is a charitable body,
>>> registered in
>>> Scotland, with registration number SC005336.
>>>
>>>
>>>
>>
>>
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>>
>>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161102/bd1f9417/attachment.html
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161102/bd1f9417/attachment.pl

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 121, Issue 17
**********************************************

Tweet