Home » Uncategories » Moses-support Digest, Vol 112, Issue 38

Moses-support Digest, Vol 112, Issue 38

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Segmentation Fault (Hieu Hoang)
2. Re: Segmentation Fault (Jasneet Sabharwal)


----------------------------------------------------------------------

Message: 1
Date: Sat, 20 Feb 2016 09:58:18 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Segmentation Fault
To: Jasneet Sabharwal <jasneet.sabharwal@sfu.ca>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbgEAK1Cp-uE4cioa1SPxWdrxbhpyt3hgwej4rW7b3z+xQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

it's great that you've written a new feature function but you will have to
debug it yourself. I suggest you put lots of debugging messages in your
code to find out where the problem is.

Moses has the Timer class in /moses/Timer.h which you can use to help your
debug your problem

Hieu Hoang
http://www.hoang.co.uk/hieu

On 20 February 2016 at 04:20, Jasneet Sabharwal <jasneet.sabharwal@sfu.ca>
wrote:

> Hi Hieu,
>
> Just to provide more info, I had compiled moses using the following
> command: "./bjam -j8 -q
> --with-cmph=/cs/natlang-user/jasneet/softwares/cmph-2.0/
> --with-boost=/cs/natlang-user/jasneet/softwares/boost/ --max-kenlm-order=8
> -a --with-mm --with-probing-pt?.
>
> Following are some more translation times from the logs using the command:
>
> $ grep ?Translation took? mert.log
>
> Line 53: Translation took 9504.886 seconds total
> Line 25: Translation took 16931.106 seconds total
> Line 20: Translation took 17477.958 seconds total
> Line 34: Translation took 18409.183 seconds total
> Line 36: Translation took 20495.204 seconds total
> Line 48: Translation took 16093.966 seconds total
> Line 68: Translation took 4773.139 seconds total
> Line 18: Translation took 22165.429 seconds total
> Line 10: Translation took 23794.930 seconds total
> Line 11: Translation took 26313.130 seconds total
> Line 74: Translation took 6238.326 seconds total
> Line 66: Translation took 14968.715 seconds total
> Line 3: Translation took 28973.902 seconds total
> Line 45: Translation took 27619.088 seconds total
> Line 81: Translation took 4666.394 seconds total
> Line 37: Translation took 36502.892 seconds total
> Line 83: Translation took 3143.882 seconds total
> Line 70: Translation took 20143.743 seconds total
> Line 1: Translation took 38498.391 seconds total
> Line 19: Translation took 39683.472 seconds total
> Line 15: Translation took 39903.566 seconds total
> Line 33: Translation took 40047.447 seconds total
>
> The times are extremely high and I?m not really sure why it is taking so
> much time.
>
> Regards,
> Jasneet
>
> On Feb 18, 2016, at 11:04 AM, Jasneet Sabharwal <jasneet.sabharwal@sfu.ca>
> wrote:
>
> Hi,
>
> I was able to solve the segmentation fault issue. It was because of OOVs.
> I?m currently trying to tune the parameters using mert, but it is running
> extremely slow. For example, from the logs:
>
> Translating: ?? ? ? ?? ? ? ? ? ? ??????? ? ? ? ? ? ? ?? ? , ? ? ?? ?? ??
> ??? ? ? ? ?? ? ? ? ??????? ? ?? ?? ?
> Line 43: Initialize search took 0.007 seconds total
> Line 43: Collecting options took 0.191 seconds at moses/Manager.cpp:117
> Line 38: Search took 1092.075 seconds
> Line 38: Decision rule took 0.000 seconds total
> Line 38: Additional reporting took 0.041 seconds total
> Line 38: Translation took 1092.132 seconds total
>
> I tried to time the functions in my feature function
> <https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp> using
> clock_t but all of them show up as 0.000. I?m not sure why tuning is taking
> too much time. My moses.ini is attached in this email.
>
> Any suggestions would be helpful.
>
> Regards,
> Jasneet
>
> <moses.ini>
>
> On Feb 12, 2016, at 3:58 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
> I think it's
> FeatureFunction::GetScoreProducerDescription()
>
> On 12/02/16 23:56, Jasneet Sabharwal wrote:
>
> Thanks, will give that a try.
>
> Also, is it possible to get the value of feature name inside the feature
> function. I?m specifically talking about ?name? parameter in moses.ini. I?m
> running multiple copies of my feature function with different parameter as
> follows:
>
> CoarseBiLM name=CoarseBiLM tgtWordId...
> CoarseBiLM name=CoarseLM100 tgtWordId?
> CoarseBiLM name=CoarseLM1600 tgtWordId...
> CoarseBiLM name=CoarseBiLMWithoutClustering tgtWordId?
>
> Thanks,
> Jasneet
>
> On Feb 12, 2016, at 3:39 PM, Hieu Hoang < <hieuhoang@gmail.com>
> hieuhoang@gmail.com> wrote:
>
> you can run the decoder
> ./moses -v 3
> however, you should put debugging messages in your feature functions to
> find out where the problem is. It looks like its in the Load() method so
> add lots of debugging message in there and all functions it calls
>
> On 12/02/16 23:34, Jasneet Sabharwal wrote:
>
> Thanks Hieu for your reply.
>
> Is it possible to do a verbose output of what?s happening, so that I can
> identify when it?s going out of memory? I?m only running it for 1928
> sentences. I have almost 170gb of free memory and additional 400gb memory
> in buffer.
>
> Thanks,
> Jasneet
>
> On Feb 12, 2016, at 2:36 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
> looks like it's run out of memory.
>
> On 11/02/16 23:23, Jasneet Sabharwal wrote:
>
> Hi,
>
> I was adding a new feature function in Moses (
> <https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp>
> https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp).
> It works fine when I test it for 1-2 sentences, but when I?m trying to tune
> my parameters, I?m getting segmentation faults or sometimes it is
> bad_alloc. Following was one of the commands that was executed during the
> tuning process which caused the Segmentation Fault or bad_alloc:
>
> moses -threads 40 -v 0 -config filtered/moses.ini -weight-overwrite
> 'CoarseLM100= 0.075758 LM0= 0.075758 CoarseBiLMNotClustered= 0.075758
> WordPenalty0= -0.151515 PhrasePenalty0= 0.030303 CoarseBiLMClustered=
> 0.075758 TranslationModel0= 0.030303 0.030303 0.030303 0.030303
> Distortion0= 0.045455 CoarseLM1600= 0.075758 LexicalReordering0= 0.045455
> 0.045455 0.045455 0.045455 0.045455 0.045455' -n-best-list run1.best100.out
> 100 distinct -input-file tune.word.lc.cn
>
> The log is enclosed in this email.
>
> Any pointers would be very useful.
>
> Thanks,
> Jasneet
>
>
>
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> --
> Hieu Hoanghttp://www.hoang.co.uk/hieu
>
>
>
> --
> Hieu Hoanghttp://www.hoang.co.uk/hieu
>
>
>
> --
> Hieu Hoanghttp://www.hoang.co.uk/hieu
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160220/d167dfa4/attachment-0001.html

------------------------------

Message: 2
Date: Sat, 20 Feb 2016 03:05:58 -0800
From: Jasneet Sabharwal <jasneet.sabharwal@sfu.ca>
Subject: Re: [Moses-support] Segmentation Fault
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <26054A6F-7B12-43C1-A52D-1ECC0B712F34@sfu.ca>
Content-Type: text/plain; charset="utf-8"

Thanks for the reply Hieu. The Timer returns time in seconds or milliseconds?
> On Feb 20, 2016, at 1:58 AM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
> it's great that you've written a new feature function but you will have to debug it yourself. I suggest you put lots of debugging messages in your code to find out where the problem is.
>
> Moses has the Timer class in /moses/Timer.h which you can use to help your debug your problem
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu <http://www.hoang.co.uk/hieu>
>
> On 20 February 2016 at 04:20, Jasneet Sabharwal <jasneet.sabharwal@sfu.ca <mailto:jasneet.sabharwal@sfu.ca>> wrote:
> Hi Hieu,
>
> Just to provide more info, I had compiled moses using the following command: "./bjam -j8 -q --with-cmph=/cs/natlang-user/jasneet/softwares/cmph-2.0/ --with-boost=/cs/natlang-user/jasneet/softwares/boost/ --max-kenlm-order=8 -a --with-mm --with-probing-pt?.
>
> Following are some more translation times from the logs using the command:
>
> $ grep ?Translation took? mert.log
>
> Line 53: Translation took 9504.886 seconds total
> Line 25: Translation took 16931.106 seconds total
> Line 20: Translation took 17477.958 seconds total
> Line 34: Translation took 18409.183 seconds total
> Line 36: Translation took 20495.204 seconds total
> Line 48: Translation took 16093.966 seconds total
> Line 68: Translation took 4773.139 seconds total
> Line 18: Translation took 22165.429 seconds total
> Line 10: Translation took 23794.930 seconds total
> Line 11: Translation took 26313.130 seconds total
> Line 74: Translation took 6238.326 seconds total
> Line 66: Translation took 14968.715 seconds total
> Line 3: Translation took 28973.902 seconds total
> Line 45: Translation took 27619.088 seconds total
> Line 81: Translation took 4666.394 seconds total
> Line 37: Translation took 36502.892 seconds total
> Line 83: Translation took 3143.882 seconds total
> Line 70: Translation took 20143.743 seconds total
> Line 1: Translation took 38498.391 seconds total
> Line 19: Translation took 39683.472 seconds total
> Line 15: Translation took 39903.566 seconds total
> Line 33: Translation took 40047.447 seconds total
>
> The times are extremely high and I?m not really sure why it is taking so much time.
>
> Regards,
> Jasneet
>> On Feb 18, 2016, at 11:04 AM, Jasneet Sabharwal <jasneet.sabharwal@sfu.ca <mailto:jasneet.sabharwal@sfu.ca>> wrote:
>>
>> Hi,
>>
>> I was able to solve the segmentation fault issue. It was because of OOVs. I?m currently trying to tune the parameters using mert, but it is running extremely slow. For example, from the logs:
>>
>> Translating: ?? ? ? ?? ? ? ? ? ? ??????? ? ? ? ? ? ? ?? ? , ? ? ?? ?? ?? ??? ? ? ? ?? ? ? ? ??????? ? ?? ?? ?
>> Line 43: Initialize search took 0.007 seconds total
>> Line 43: Collecting options took 0.191 seconds at moses/Manager.cpp:117
>> Line 38: Search took 1092.075 seconds
>> Line 38: Decision rule took 0.000 seconds total
>> Line 38: Additional reporting took 0.041 seconds total
>> Line 38: Translation took 1092.132 seconds total
>>
>> I tried to time the functions in my feature function <https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp> using clock_t but all of them show up as 0.000. I?m not sure why tuning is taking too much time. My moses.ini is attached in this email.
>>
>> Any suggestions would be helpful.
>>
>> Regards,
>> Jasneet
>>
>> <moses.ini>
>>> On Feb 12, 2016, at 3:58 PM, Hieu Hoang <hieuhoang@gmail.com <mailto:hieuhoang@gmail.com>> wrote:
>>>
>>> I think it's
>>> FeatureFunction::GetScoreProducerDescription()
>>>
>>> On 12/02/16 23:56, Jasneet Sabharwal wrote:
>>>> Thanks, will give that a try.
>>>>
>>>> Also, is it possible to get the value of feature name inside the feature function. I?m specifically talking about ?name? parameter in moses.ini. I?m running multiple copies of my feature function with different parameter as follows:
>>>> CoarseBiLM name=CoarseBiLM tgtWordId...
>>>> CoarseBiLM name=CoarseLM100 tgtWordId?
>>>> CoarseBiLM name=CoarseLM1600 tgtWordId...
>>>> CoarseBiLM name=CoarseBiLMWithoutClustering tgtWordId?
>>>>
>>>> Thanks,
>>>> Jasneet
>>>>> On Feb 12, 2016, at 3:39 PM, Hieu Hoang < <mailto:hieuhoang@gmail.com>hieuhoang@gmail.com <mailto:hieuhoang@gmail.com>> wrote:
>>>>>
>>>>> you can run the decoder
>>>>> ./moses -v 3
>>>>> however, you should put debugging messages in your feature functions to find out where the problem is. It looks like its in the Load() method so add lots of debugging message in there and all functions it calls
>>>>>
>>>>> On 12/02/16 23:34, Jasneet Sabharwal wrote:
>>>>>> Thanks Hieu for your reply.
>>>>>>
>>>>>> Is it possible to do a verbose output of what?s happening, so that I can identify when it?s going out of memory? I?m only running it for 1928 sentences. I have almost 170gb of free memory and additional 400gb memory in buffer.
>>>>>>
>>>>>> Thanks,
>>>>>> Jasneet
>>>>>>
>>>>>>> On Feb 12, 2016, at 2:36 PM, Hieu Hoang <hieuhoang@gmail.com <mailto:hieuhoang@gmail.com>> wrote:
>>>>>>>
>>>>>>> looks like it's run out of memory.
>>>>>>>
>>>>>>> On 11/02/16 23:23, Jasneet Sabharwal wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I was adding a new feature function in Moses ( <https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp>https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp <https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp>). It works fine when I test it for 1-2 sentences, but when I?m trying to tune my parameters, I?m getting segmentation faults or sometimes it is bad_alloc. Following was one of the commands that was executed during the tuning process which caused the Segmentation Fault or bad_alloc:
>>>>>>>>
>>>>>>>> moses -threads 40 -v 0 -config filtered/moses.ini -weight-overwrite 'CoarseLM100= 0.075758 LM0= 0.075758 CoarseBiLMNotClustered= 0.075758 WordPenalty0= -0.151515 PhrasePenalty0= 0.030303 CoarseBiLMClustered= 0.075758 TranslationModel0= 0.030303 0.030303 0.030303 0.030303 Distortion0= 0.045455 CoarseLM1600= 0.075758 LexicalReordering0= 0.045455 0.045455 0.045455 0.045455 0.045455 0.045455' -n-best-list run1.best100.out 100 distinct -input-file tune.word.lc.cn <http://tune.word.lc.cn/>
>>>>>>>>
>>>>>>>> The log is enclosed in this email.
>>>>>>>>
>>>>>>>> Any pointers would be very useful.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Jasneet
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Moses-support mailing list
>>>>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>
>>>>>>>
>>>>>>> --
>>>>>>> Hieu Hoang
>>>>>>> http://www.hoang.co.uk/hieu <http://www.hoang.co.uk/hieu>
>>>>>
>>>>> --
>>>>> Hieu Hoang
>>>>> http://www.hoang.co.uk/hieu <http://www.hoang.co.uk/hieu>
>>>
>>> --
>>> Hieu Hoang
>>> http://www.hoang.co.uk/hieu <http://www.hoang.co.uk/hieu>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160220/e2373790/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 112, Issue 38
**********************************************

0 Response to "Moses-support Digest, Vol 112, Issue 38"

Post a Comment

Subscribe to: Post Comments (Atom)