Moses-support Digest, Vol 94, Issue 22

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Adding score to phrase table (Hubert Soyer)
2. Re: Adding score to phrase table (Hieu Hoang)
3. Re: Adding score to phrase table (Hubert Soyer)


----------------------------------------------------------------------

Message: 1
Date: Mon, 18 Aug 2014 13:34:26 +0900
From: Hubert Soyer <hubert.soyer@gmail.com>
Subject: [Moses-support] Adding score to phrase table
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAM7TO-g2nf5n97fbAgp8WPydzkK0Eqt0JbTUBi_wfPS=6HZEDg@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hello everybody,

I am currently trying to add an externally computed score to the phrase table.
The format looks like this:

in europa ||| in europe ||| 0.5 0.03125 1 0.340909 ||| 0-0 1-1 ||| 1 1 1 ||| |||

As far as I understand the first and the second column contain the
phrase in both languages and the third column contains a list of
scores, separated by white spaces.

I would now use an external tool to add another score to the third column.
My first attempt was to simply compute the score and add it -
separated by another white space - to the third column.

This is what it could look like:
in europa ||| in europe ||| 0.5 0.03125 1 0.340909 0.97 ||| 0-0 1-1
||| 1 1 1 ||| |||

Since now I have 5 instead of 4 features I changed the num-features
option in my moses.ini:

PhraseDictionaryMemory name=TranslationModel0 table-limit=20
num-features=5 path=... input-factor=0 output-factor=0

Also, I added one more inital weight to the TranslationModel0 line:

[weight]
TranslationModel0= 0.2 0.2 0.2 0.2 0.2

The moses.ini that I changed is located in my working directory under
"train/model/".
The phrase table is saved under "/train/model/phrase-table.gz".

When I run the tuning as
$mosesbase/mosesdecoder/scripts/training/mert-moses.pl
$mosesbase/corpus/train.100.ja $mosesbase/corpus/train.100.en
$mosesbase/mosesdecoder/bin/moses train/model/moses.ini --mertdir
$mosesbase/mosesdecoder/bin/ --rootdir
$mosesbase/mosesdecoder/scripts/ &> translation.log

the decoder exits with
Size of scoreVector != number (4!=5) of score components on line 0

After looking around I found out that the phrase table filtering
somehow deletes my newly added score.
In my initial phrase table located in the "train/model" directory my
score is present but in the phrase table in the mert-work directory
"mert-work/filtered/phrase-table.0-0.1.1.gz" for some reason the score
is gone.

Is there anything I need to specify to keep my score?

Thank you in advance!

Best,

Hubert


------------------------------

Message: 2
Date: Mon, 18 Aug 2014 08:32:43 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Adding score to phrase table
To: Hubert Soyer <hubert.soyer@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAEKMkbhLk+qn-7AJijazE0+-vKCFCcLY-ZSoEjN56TcHq-3CCQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

everything seems correct.

Try testing that the moses.ini file is correct by just running the decoder:
$mosesbase/mosesdecoder/bin/moses -f train/model/moses.ini

if there is a tmp directory from a previous mert tuning step, delete it
before rerunning mert



On 18 August 2014 05:34, Hubert Soyer <hubert.soyer@gmail.com> wrote:

> Hello everybody,
>
> I am currently trying to add an externally computed score to the phrase
> table.
> The format looks like this:
>
> in europa ||| in europe ||| 0.5 0.03125 1 0.340909 ||| 0-0 1-1 ||| 1 1 1
> ||| |||
>
> As far as I understand the first and the second column contain the
> phrase in both languages and the third column contains a list of
> scores, separated by white spaces.
>
> I would now use an external tool to add another score to the third column.
> My first attempt was to simply compute the score and add it -
> separated by another white space - to the third column.
>
> This is what it could look like:
> in europa ||| in europe ||| 0.5 0.03125 1 0.340909 0.97 ||| 0-0 1-1
> ||| 1 1 1 ||| |||
>
> Since now I have 5 instead of 4 features I changed the num-features
> option in my moses.ini:
>
> PhraseDictionaryMemory name=TranslationModel0 table-limit=20
> num-features=5 path=... input-factor=0 output-factor=0
>
> Also, I added one more inital weight to the TranslationModel0 line:
>
> [weight]
> TranslationModel0= 0.2 0.2 0.2 0.2 0.2
>
> The moses.ini that I changed is located in my working directory under
> "train/model/".
> The phrase table is saved under "/train/model/phrase-table.gz".
>
> When I run the tuning as
> $mosesbase/mosesdecoder/scripts/training/mert-moses.pl
> $mosesbase/corpus/train.100.ja $mosesbase/corpus/train.100.en
> $mosesbase/mosesdecoder/bin/moses train/model/moses.ini --mertdir
> $mosesbase/mosesdecoder/bin/ --rootdir
> $mosesbase/mosesdecoder/scripts/ &> translation.log
>
> the decoder exits with
> Size of scoreVector != number (4!=5) of score components on line 0
>
> After looking around I found out that the phrase table filtering
> somehow deletes my newly added score.
> In my initial phrase table located in the "train/model" directory my
> score is present but in the phrase table in the mert-work directory
> "mert-work/filtered/phrase-table.0-0.1.1.gz" for some reason the score
> is gone.
>
> Is there anything I need to specify to keep my score?
>
> Thank you in advance!
>
> Best,
>
> Hubert
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140818/5bf666fa/attachment-0001.htm

------------------------------

Message: 3
Date: Mon, 18 Aug 2014 16:50:25 +0900
From: Hubert Soyer <hubert.soyer@gmail.com>
Subject: Re: [Moses-support] Adding score to phrase table
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAM7TO-jswDYxYhTnV9y3m-PPu23n4zDoBSP1=OsZNHkacq5c4w@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Thank you very much. I found the error and it was my own fault.
I had an old path in the moses.ini and since it is almost the same
path as the new one I didn't notice the whole time.
Running "$mosesbase/mosesdecoder/bin/moses -f train/model/moses.ini"
gave me an error message that revealed my mistake.

Thank you again!

Best,

Hubert

On Mon, Aug 18, 2014 at 4:32 PM, Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:
> everything seems correct.
>
> Try testing that the moses.ini file is correct by just running the decoder:
> $mosesbase/mosesdecoder/bin/moses -f train/model/moses.ini
>
> if there is a tmp directory from a previous mert tuning step, delete it
> before rerunning mert
>
>
>
> On 18 August 2014 05:34, Hubert Soyer <hubert.soyer@gmail.com> wrote:
>>
>> Hello everybody,
>>
>> I am currently trying to add an externally computed score to the phrase
>> table.
>> The format looks like this:
>>
>> in europa ||| in europe ||| 0.5 0.03125 1 0.340909 ||| 0-0 1-1 ||| 1 1 1
>> ||| |||
>>
>> As far as I understand the first and the second column contain the
>> phrase in both languages and the third column contains a list of
>> scores, separated by white spaces.
>>
>> I would now use an external tool to add another score to the third column.
>> My first attempt was to simply compute the score and add it -
>> separated by another white space - to the third column.
>>
>> This is what it could look like:
>> in europa ||| in europe ||| 0.5 0.03125 1 0.340909 0.97 ||| 0-0 1-1
>> ||| 1 1 1 ||| |||
>>
>> Since now I have 5 instead of 4 features I changed the num-features
>> option in my moses.ini:
>>
>> PhraseDictionaryMemory name=TranslationModel0 table-limit=20
>> num-features=5 path=... input-factor=0 output-factor=0
>>
>> Also, I added one more inital weight to the TranslationModel0 line:
>>
>> [weight]
>> TranslationModel0= 0.2 0.2 0.2 0.2 0.2
>>
>> The moses.ini that I changed is located in my working directory under
>> "train/model/".
>> The phrase table is saved under "/train/model/phrase-table.gz".
>>
>> When I run the tuning as
>> $mosesbase/mosesdecoder/scripts/training/mert-moses.pl
>> $mosesbase/corpus/train.100.ja $mosesbase/corpus/train.100.en
>> $mosesbase/mosesdecoder/bin/moses train/model/moses.ini --mertdir
>> $mosesbase/mosesdecoder/bin/ --rootdir
>> $mosesbase/mosesdecoder/scripts/ &> translation.log
>>
>> the decoder exits with
>> Size of scoreVector != number (4!=5) of score components on line 0
>>
>> After looking around I found out that the phrase table filtering
>> somehow deletes my newly added score.
>> In my initial phrase table located in the "train/model" directory my
>> score is present but in the phrase table in the mert-work directory
>> "mert-work/filtered/phrase-table.0-0.1.1.gz" for some reason the score
>> is gone.
>>
>> Is there anything I need to specify to keep my score?
>>
>> Thank you in advance!
>>
>> Best,
>>
>> Hubert
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>


------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 94, Issue 22
*********************************************

Tweet