Moses-support Digest, Vol 97, Issue 15

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Can't find Bleu Score (Apoorvi Singh)


----------------------------------------------------------------------

Message: 1
Date: Wed, 12 Nov 2014 16:21:25 +0100
From: Apoorvi Singh <apoorvisingh@gmail.com>
Subject: Re: [Moses-support] Can't find Bleu Score
To: Prashant Mathur <prashant@fbk.eu>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAFB5vfQMGg_YpEowab30JbzBdFOUCNeUW1=g=FnB+6eJf1kfsQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Dear Prashant,
Your assistance helped! Thanks a lot for your help with the issue.

Regards,
Apoorvi

On Wed, Nov 12, 2014 at 3:35 PM, Prashant Mathur <prashant@fbk.eu> wrote:

> the decoder cannot locate these files
>
> /home/apoorvi/btpp/working/filtered-newstest2011/phrase-table.0-0.1.1.binphr.*
>
> I think your filter script didn't work as it was supposed to. I don't know
> what happened there.
> If your phrase-table is huge filtering the phrase table makes sense,
> otherwise you can directly binarize the phrase-table like this
>
> cd /home/apoorvi/btpp/working
> gunzip -c /home/apoorvi/btpp/working/train/model/phrase-table.gz |
> LC_ALL=C sort | ~/btpp/mosesdecoder/bin/processPhraseTable -ttable 0 0 -
> -nscores 1 -out phrase-table.gz
> and then just replace PhraseDictionaryMemory with PhraseDictionaryBinary
>
> even better use Compact Phrase Table structure, this also uses less memory
> while decoding.
> read about it here :
> http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc12
>
> --Prashant
>
>
>
>
> On Wed, Nov 12, 2014 at 2:58 PM, Apoorvi Singh <apoorvisingh@gmail.com>
> wrote:
>
>> Bleu.out file is attached.
>>
>> However, I could manage to obtain the bleu score when I skipped the above
>> script to filter the phrases in phrase and reordering table.
>> I directly ran the bleu script by using the moses.ini obtained after
>> tuning. I am not sure to what extent the bypassing has affected the score.
>>
>> On Wed, Nov 12, 2014 at 2:38 PM, Prashant Mathur <prashant@fbk.eu> wrote:
>>
>>> Your config file looks ok as long as you have 1 feature in your phrase
>>> table.
>>>
>>> This script is used to filter the phrases in your phrase table and
>>> reordering table based on "bleu.true.sorc" file.
>>> *~/btpp/mosesdecoder/scripts/training/filter-model-given-input.pl
>>> <http://filter-model-given-input.pl/> filtered-newstest2011
>>> mert-work/moses.ini ~/btpp/corpus/testingBleu/bleu.true.sorc -Binarizer
>>> ~/btpp/mosesdecoder/bin/processPhraseTable*
>>>
>>> Your problem is related to decoding. Can you send this log file :
>>> *~/btpp/working/bleu.out *
>>> It will better explain where the problem occurs.
>>>
>>>
>>> On Wed, Nov 12, 2014 at 4:54 AM, Apoorvi Singh <apoorvisingh@gmail.com>
>>> wrote:
>>>
>>>> Hi Prashant,
>>>>
>>>> -- The modified configuration file 'moses.ini' is attached.
>>>>
>>>> Changes done: num-features chnaged from 4 to 1 and TranslationModel0=
>>>> 0.2.
>>>>
>>>> -- Exact Bleu script used to produce the output:
>>>>
>>>> > ~/btpp/mosesdecoder/scripts/training/filter-model-given-input.pl
>>>> filtered-newstest2011 mert-work/moses.ini
>>>> ~/btpp/corpus/testingBleu/bleu.true.sorc -Binarizer
>>>> ~/btpp/mosesdecoder/bin/processPhraseTable
>>>> [This script runs well]
>>>>
>>>> > nohup nice ~/btpp/mosesdecoder/bin/moses -f
>>>> ~/btpp/working/filtered-newstest2011/moses.ini <
>>>> ~/btpp/corpus/testingBleu/bleu.true.sorc >
>>>> ~/btpp/working/bleu.translated.dest 2> ~/btpp/working/bleu.out
>>>> [This command returns Aborted core dumped]
>>>>
>>>>
>>>> -Apoorvi
>>>>
>>>>
>>>> On Wed, Nov 12, 2014 at 12:17 AM, Prashant Mathur <prashant@fbk.eu>
>>>> wrote:
>>>>
>>>>> Hi Apoorvi,
>>>>>
>>>>> Can you send us your configuration file and also the exact command you
>>>>> ran to produce the output?
>>>>>
>>>>> Which script did you use to calculate BLEU score? How did you use it?
>>>>>
>>>>> --Prashant
>>>>>
>>>>> On Tue, Nov 11, 2014 at 8:19 PM, Apoorvi Singh <apoorvisingh@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Dear,
>>>>>>
>>>>>> I am using the moses.ini configuration file with num-features=1.
>>>>>>
>>>>>> While running the bleu script of the baseline system to find the bleu
>>>>>> score with this new configuration, I encounter Aborted-core dumped error.
>>>>>>
>>>>>> I am unable to calculate bleu score. Do I need to make other changes
>>>>>> in configuration. Kindly help me with this.
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Apoorvi
>>>>>>
>>>>>> _______________________________________________
>>>>>> Moses-support mailing list
>>>>>> Moses-support@mit.edu
>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Apoorvi Singh*
>>>> *Undergraduate Student,*
>>>> *Department of Electrical Engineering,*
>>>> *IIT Delhi.*
>>>>
>>>
>>>
>>
>>
>> --
>> *Apoorvi Singh*
>> *Undergraduate Student,*
>> *Department of Electrical Engineering,*
>> *IIT Delhi.*
>>
>
>


--
*Apoorvi Singh*
*Undergraduate Student,*
*Department of Electrical Engineering,*
*IIT Delhi.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141112/b5401aa5/attachment-0001.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 97, Issue 15
*********************************************

Tweet