Moses-support Digest, Vol 97, Issue 14

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Can't find Bleu Score (Apoorvi Singh)
2. The decoder died with 'Segmentation fault' (Huei-Chi Lin)
3. Re: Can't find Bleu Score (Prashant Mathur)


----------------------------------------------------------------------

Message: 1
Date: Wed, 12 Nov 2014 14:58:25 +0100
From: Apoorvi Singh <apoorvisingh@gmail.com>
Subject: Re: [Moses-support] Can't find Bleu Score
To: Prashant Mathur <prashant@fbk.eu>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAFB5vfRhRF28hpMw4aWDrHrzXVw6Kydft-OCC=umoi9jWBmRng@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Bleu.out file is attached.

However, I could manage to obtain the bleu score when I skipped the above
script to filter the phrases in phrase and reordering table.
I directly ran the bleu script by using the moses.ini obtained after
tuning. I am not sure to what extent the bypassing has affected the score.

On Wed, Nov 12, 2014 at 2:38 PM, Prashant Mathur <prashant@fbk.eu> wrote:

> Your config file looks ok as long as you have 1 feature in your phrase
> table.
>
> This script is used to filter the phrases in your phrase table and
> reordering table based on "bleu.true.sorc" file.
> *~/btpp/mosesdecoder/scripts/training/filter-model-given-input.pl
> <http://filter-model-given-input.pl/> filtered-newstest2011
> mert-work/moses.ini ~/btpp/corpus/testingBleu/bleu.true.sorc -Binarizer
> ~/btpp/mosesdecoder/bin/processPhraseTable*
>
> Your problem is related to decoding. Can you send this log file :
> *~/btpp/working/bleu.out *
> It will better explain where the problem occurs.
>
>
> On Wed, Nov 12, 2014 at 4:54 AM, Apoorvi Singh <apoorvisingh@gmail.com>
> wrote:
>
>> Hi Prashant,
>>
>> -- The modified configuration file 'moses.ini' is attached.
>>
>> Changes done: num-features chnaged from 4 to 1 and TranslationModel0= 0.2.
>>
>> -- Exact Bleu script used to produce the output:
>>
>> > ~/btpp/mosesdecoder/scripts/training/filter-model-given-input.pl
>> filtered-newstest2011 mert-work/moses.ini
>> ~/btpp/corpus/testingBleu/bleu.true.sorc -Binarizer
>> ~/btpp/mosesdecoder/bin/processPhraseTable
>> [This script runs well]
>>
>> > nohup nice ~/btpp/mosesdecoder/bin/moses -f
>> ~/btpp/working/filtered-newstest2011/moses.ini <
>> ~/btpp/corpus/testingBleu/bleu.true.sorc >
>> ~/btpp/working/bleu.translated.dest 2> ~/btpp/working/bleu.out
>> [This command returns Aborted core dumped]
>>
>>
>> -Apoorvi
>>
>>
>> On Wed, Nov 12, 2014 at 12:17 AM, Prashant Mathur <prashant@fbk.eu>
>> wrote:
>>
>>> Hi Apoorvi,
>>>
>>> Can you send us your configuration file and also the exact command you
>>> ran to produce the output?
>>>
>>> Which script did you use to calculate BLEU score? How did you use it?
>>>
>>> --Prashant
>>>
>>> On Tue, Nov 11, 2014 at 8:19 PM, Apoorvi Singh <apoorvisingh@gmail.com>
>>> wrote:
>>>
>>>> Dear,
>>>>
>>>> I am using the moses.ini configuration file with num-features=1.
>>>>
>>>> While running the bleu script of the baseline system to find the bleu
>>>> score with this new configuration, I encounter Aborted-core dumped error.
>>>>
>>>> I am unable to calculate bleu score. Do I need to make other changes in
>>>> configuration. Kindly help me with this.
>>>>
>>>> Thanks and Regards,
>>>> Apoorvi
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>>
>>>
>>
>>
>> --
>> *Apoorvi Singh*
>> *Undergraduate Student,*
>> *Department of Electrical Engineering,*
>> *IIT Delhi.*
>>
>
>


--
*Apoorvi Singh*
*Undergraduate Student,*
*Department of Electrical Engineering,*
*IIT Delhi.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141112/025d1150/attachment-0001.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bleu.out
Type: application/octet-stream
Size: 2640 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20141112/025d1150/attachment-0001.obj

------------------------------

Message: 2
Date: Wed, 12 Nov 2014 15:07:44 +0100
From: Huei-Chi Lin <lin@lium.univ-lemans.fr>
Subject: [Moses-support] The decoder died with 'Segmentation fault'
To: Moses-support@mit.edu
Message-ID: <546369B0.80707@lium.univ-lemans.fr>
Content-Type: text/plain; charset=utf-8; format=flowed

Hello,

When I ran mert-moses.pl of 2014-05-29 version, the messages
'Segmentation fault' and 'The decoder died.' were written into the log
file. This problem occurs randomly, if the mert are run several times.
Running mert within a backtracking debugger, the messages are as follows:

(gdb) exec-file /path/to/the/directory/bin/moses
(gdb) r "mert options"

Translating line 2340 in thread id 140734307309312

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff44e56700 (LWP 22396)]
0x00000000005e6625 in ?? ()

Could someone help me to resolve this problem, thanks?

best regards,
Huei-Chi Lin


------------------------------

Message: 3
Date: Wed, 12 Nov 2014 15:35:23 +0100
From: Prashant Mathur <prashant@fbk.eu>
Subject: Re: [Moses-support] Can't find Bleu Score
To: Apoorvi Singh <apoorvisingh@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAK3pNh+GTEXaDTuC6NqzpkA6xpWW4wuNxvLct9FfRGRw9x5gJQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

the decoder cannot locate these files
/home/apoorvi/btpp/working/filtered-newstest2011/phrase-table.0-0.1.1.binphr.*

I think your filter script didn't work as it was supposed to. I don't know
what happened there.
If your phrase-table is huge filtering the phrase table makes sense,
otherwise you can directly binarize the phrase-table like this

cd /home/apoorvi/btpp/working
gunzip -c /home/apoorvi/btpp/working/train/model/phrase-table.gz | LC_ALL=C
sort | ~/btpp/mosesdecoder/bin/processPhraseTable -ttable 0 0 - -nscores 1
-out phrase-table.gz
and then just replace PhraseDictionaryMemory with PhraseDictionaryBinary

even better use Compact Phrase Table structure, this also uses less memory
while decoding.
read about it here :
http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc12

--Prashant




On Wed, Nov 12, 2014 at 2:58 PM, Apoorvi Singh <apoorvisingh@gmail.com>
wrote:

> Bleu.out file is attached.
>
> However, I could manage to obtain the bleu score when I skipped the above
> script to filter the phrases in phrase and reordering table.
> I directly ran the bleu script by using the moses.ini obtained after
> tuning. I am not sure to what extent the bypassing has affected the score.
>
> On Wed, Nov 12, 2014 at 2:38 PM, Prashant Mathur <prashant@fbk.eu> wrote:
>
>> Your config file looks ok as long as you have 1 feature in your phrase
>> table.
>>
>> This script is used to filter the phrases in your phrase table and
>> reordering table based on "bleu.true.sorc" file.
>> *~/btpp/mosesdecoder/scripts/training/filter-model-given-input.pl
>> <http://filter-model-given-input.pl/> filtered-newstest2011
>> mert-work/moses.ini ~/btpp/corpus/testingBleu/bleu.true.sorc -Binarizer
>> ~/btpp/mosesdecoder/bin/processPhraseTable*
>>
>> Your problem is related to decoding. Can you send this log file :
>> *~/btpp/working/bleu.out *
>> It will better explain where the problem occurs.
>>
>>
>> On Wed, Nov 12, 2014 at 4:54 AM, Apoorvi Singh <apoorvisingh@gmail.com>
>> wrote:
>>
>>> Hi Prashant,
>>>
>>> -- The modified configuration file 'moses.ini' is attached.
>>>
>>> Changes done: num-features chnaged from 4 to 1 and TranslationModel0=
>>> 0.2.
>>>
>>> -- Exact Bleu script used to produce the output:
>>>
>>> > ~/btpp/mosesdecoder/scripts/training/filter-model-given-input.pl
>>> filtered-newstest2011 mert-work/moses.ini
>>> ~/btpp/corpus/testingBleu/bleu.true.sorc -Binarizer
>>> ~/btpp/mosesdecoder/bin/processPhraseTable
>>> [This script runs well]
>>>
>>> > nohup nice ~/btpp/mosesdecoder/bin/moses -f
>>> ~/btpp/working/filtered-newstest2011/moses.ini <
>>> ~/btpp/corpus/testingBleu/bleu.true.sorc >
>>> ~/btpp/working/bleu.translated.dest 2> ~/btpp/working/bleu.out
>>> [This command returns Aborted core dumped]
>>>
>>>
>>> -Apoorvi
>>>
>>>
>>> On Wed, Nov 12, 2014 at 12:17 AM, Prashant Mathur <prashant@fbk.eu>
>>> wrote:
>>>
>>>> Hi Apoorvi,
>>>>
>>>> Can you send us your configuration file and also the exact command you
>>>> ran to produce the output?
>>>>
>>>> Which script did you use to calculate BLEU score? How did you use it?
>>>>
>>>> --Prashant
>>>>
>>>> On Tue, Nov 11, 2014 at 8:19 PM, Apoorvi Singh <apoorvisingh@gmail.com>
>>>> wrote:
>>>>
>>>>> Dear,
>>>>>
>>>>> I am using the moses.ini configuration file with num-features=1.
>>>>>
>>>>> While running the bleu script of the baseline system to find the bleu
>>>>> score with this new configuration, I encounter Aborted-core dumped error.
>>>>>
>>>>> I am unable to calculate bleu score. Do I need to make other changes
>>>>> in configuration. Kindly help me with this.
>>>>>
>>>>> Thanks and Regards,
>>>>> Apoorvi
>>>>>
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Apoorvi Singh*
>>> *Undergraduate Student,*
>>> *Department of Electrical Engineering,*
>>> *IIT Delhi.*
>>>
>>
>>
>
>
> --
> *Apoorvi Singh*
> *Undergraduate Student,*
> *Department of Electrical Engineering,*
> *IIT Delhi.*
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141112/5e7d6b6e/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 97, Issue 14
*********************************************

Tweet