Moses-support Digest, Vol 88, Issue 68

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Binarising the phrase table (Per Tunedal)
2. Re: Binarising the phrase table (Massinissa Ahmim)


----------------------------------------------------------------------

Message: 1
Date: Fri, 28 Feb 2014 10:42:08 +0100
From: Per Tunedal <per.tunedal@operamail.com>
Subject: Re: [Moses-support] Binarising the phrase table
To: Massinissa Ahmim <massinissa.ahmim@linguacustodia.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<1393580528.9983.88871281.29C0C0B6@webmail.messagingengine.com>
Content-Type: text/plain; charset="UTF-8"

Hi Massinissa,
my commands:
~/mosesdecoder/bin/processPhraseTable \
-ttable 0 0 train/model/Kryptering1.sv-fr.phrase-table.pruned \
-nscores 5 -out train/binarised-model/Kryptering1.sv-fr.phrase-table

~/mosesdecoder/bin/processLexicalTable \
-in train/model/reordering-table.wbe-msd-bidirectional-fe.gz \
-out train/binarised-model/Kryptering1.sv-fr.reordering-table

I tried the following changes in moses.ini:

# OLD FORMAT a binary table type (1) is assumed
[ttable-file]
#0 0 0 5 /home/per/working/train/model/phrase-table.gz
0 0 0 5
/home/per/working/train/binarised-model/Kryptering1.sv-fr.phrase-table

# distortion (reordering) files
[distortion-file]
#0-0 wbe-msd-bidirectional-fe-allff 6
/home/per/working/train/model/reordering-table.wbe-msd-bidirectional-fe.gz

0-0 wbe-msd-bidirectional-fe-allff 6
/home/per/working/train/binarised-model/Kryptering1.sv-fr.reordering-table

But it doesn't work.
Maybe changing the zeros to ones would do the trick?
1 0 0 5
/home/per/working/train/binarised-model/Kryptering1.sv-fr.phrase-table
1-0 wbe-msd-bidirectional-fe-allff 6
/home/per/working/train/binarised-model/Kryptering1.sv-fr.reordering-table

Yours,
Per Tunedal

On Fri, Feb 28, 2014, at 9:55, Massinissa Ahmim wrote:
> Hi Per,
>
> Could you please past here the command your ran to binarise the
> Phrase-table?
>
> The current paths in your moses.ini should be to the out-put of
> processPhraseTable,
> for instance, if the output was
> /home/per/working/train/model/whatever/phrase-table/
> , then you have to change your current path by this one making sure to
> use
> the stemming (taking the .gz off as I did above) and switching the first
> moses parameter (from 0 to 1) as follows :
>
>
> 1 0 0 5 /home/per/working/train/model/whatever/phrase-table/
>
>
> Same thing with the reordering table
>
> Regards
>
> Massinissa
>
>
> 2014-02-28 7:47 GMT+01:00 Per Tunedal <per.tunedal@operamail.com>:
>
> >
> > Hi,
> > tried to binarise the phrase table and got in to trouble.
> >
> > 1. Error messages, as below. What's that?
> > distinct source phrases: 439511 distinct first words of source phrases:
> > 67600 number of phrase pairs (line count): 940438
> > Count of lines with missing alignments: 0/940438
> > WARNING: there are src voc entries with no phrase translation: count
> > 2168
> > There exists phrase translations for 65432 entries
> >
> > 2. Modify the moses.ini file. I've found this on the Moses/Baseline
> > page:
> > 1. Change PhraseDictionaryMemory to PhraseDictionaryBinary
> > 2. Set the path of the PhraseDictionary feature to point to
> > $HOME/working/train/binarised-model/Kryptering1.sv-fr.phrase-table
> > 3. Set the path of the LexicalReordering feature to point to
> > $HOME/working/train/binarised-model/Kryptering1.sv-fr.reordering-table
> >
> > But I cannot find any such entries in my moses.ini - maybe because I'm
> > running a somewhat older version of Moses. I've found e.g. the following
> > lines in my ini-file:
> >
> > [ttable-file]
> > 0 0 0 5 /home/per/working/train/model/phrase-table.gz
> >
> > # distortion (reordering) files
> > [distortion-file]
> > 0-0 wbe-msd-bidirectional-fe-allff 6
> > /home/per/working/train/model/reordering-table.wbe-msd-bidirectional-fe.gz
> >
> > How should I change the entries to use my binarised model?
> >
> > Yours,
> > Per Tunedal
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
>
>
>
> --
>
> [image: Description : Description : lingua_custodia_final full logo]
>
> *The Translation Trustee*
>
> *1, Place Charles de Gaulle*
>
> *78180 Montigny-le-Bretonneux*
>
> *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
>
> *Email :* *massinissa.ahmim@linguacustodia.com
> <massinissa.ahmim@linguacustodia.com>*
>
> *Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> -
> www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*
>
> ? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.
>
> Please do not print this email unless it is absolutely necessary. Spread
> environmental awareness.
> Email had 1 attachment:
> + image001.jpg
> 6k (image/jpeg)



------------------------------

Message: 2
Date: Fri, 28 Feb 2014 10:54:43 +0100
From: Massinissa Ahmim <massinissa.ahmim@linguacustodia.com>
Subject: Re: [Moses-support] Binarising the phrase table
To: Per Tunedal <per.tunedal@operamail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CANN0mWbeLyoiHxq1FhxkM=maQ-z4HdYweosPdZ6j6d1wnPMztw@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi Per,

Yes, you have change the 0 into 1 to use binary files. Actually you should
do that only for the phrase-table, meaning :

1 0 0 5 /home/per/working/train/binarised-model/Kryptering1.
sv-fr.phrase-table
0-0 wbe-msd-bidirectional-fe-allff 6 /home/per/working/train/
binarised-model/Kryptering1.sv-fr.reordering-table

My last comment could be kinda misleading but by the 'same thing' I meant
changing the paths.

It should work now assuming the paths are accurate

Regards

Massinissa


2014-02-28 10:42 GMT+01:00 Per Tunedal <per.tunedal@operamail.com>:

> Hi Massinissa,
> my commands:
> ~/mosesdecoder/bin/processPhraseTable \
> -ttable 0 0 train/model/Kryptering1.sv-fr.phrase-table.pruned \
> -nscores 5 -out train/binarised-model/Kryptering1.sv-fr.phrase-table
>
> ~/mosesdecoder/bin/processLexicalTable \
> -in train/model/reordering-table.wbe-msd-bidirectional-fe.gz \
> -out train/binarised-model/Kryptering1.sv-fr.reordering-table
>
> I tried the following changes in moses.ini:
>
> # OLD FORMAT a binary table type (1) is assumed
> [ttable-file]
> #0 0 0 5 /home/per/working/train/model/phrase-table.gz
> 0 0 0 5
> /home/per/working/train/binarised-model/Kryptering1.sv-fr.phrase-table
>
> # distortion (reordering) files
> [distortion-file]
> #0-0 wbe-msd-bidirectional-fe-allff 6
> /home/per/working/train/model/reordering-table.wbe-msd-bidirectional-fe.gz
>
> 0-0 wbe-msd-bidirectional-fe-allff 6
> /home/per/working/train/binarised-model/Kryptering1.sv-fr.reordering-table
>
> But it doesn't work.
> Maybe changing the zeros to ones would do the trick?
> 1 0 0 5
> /home/per/working/train/binarised-model/Kryptering1.sv-fr.phrase-table
> 1-0 wbe-msd-bidirectional-fe-allff 6
> /home/per/working/train/binarised-model/Kryptering1.sv-fr.reordering-table
>
> Yours,
> Per Tunedal
>
> On Fri, Feb 28, 2014, at 9:55, Massinissa Ahmim wrote:
> > Hi Per,
> >
> > Could you please past here the command your ran to binarise the
> > Phrase-table?
> >
> > The current paths in your moses.ini should be to the out-put of
> > processPhraseTable,
> > for instance, if the output was
> > /home/per/working/train/model/whatever/phrase-table/
> > , then you have to change your current path by this one making sure to
> > use
> > the stemming (taking the .gz off as I did above) and switching the first
> > moses parameter (from 0 to 1) as follows :
> >
> >
> > 1 0 0 5 /home/per/working/train/model/whatever/phrase-table/
> >
> >
> > Same thing with the reordering table
> >
> > Regards
> >
> > Massinissa
> >
> >
> > 2014-02-28 7:47 GMT+01:00 Per Tunedal <per.tunedal@operamail.com>:
> >
> > >
> > > Hi,
> > > tried to binarise the phrase table and got in to trouble.
> > >
> > > 1. Error messages, as below. What's that?
> > > distinct source phrases: 439511 distinct first words of source phrases:
> > > 67600 number of phrase pairs (line count): 940438
> > > Count of lines with missing alignments: 0/940438
> > > WARNING: there are src voc entries with no phrase translation: count
> > > 2168
> > > There exists phrase translations for 65432 entries
> > >
> > > 2. Modify the moses.ini file. I've found this on the Moses/Baseline
> > > page:
> > > 1. Change PhraseDictionaryMemory to PhraseDictionaryBinary
> > > 2. Set the path of the PhraseDictionary feature to point to
> > > $HOME/working/train/binarised-model/Kryptering1.sv-fr.phrase-table
> > > 3. Set the path of the LexicalReordering feature to point to
> > >
> $HOME/working/train/binarised-model/Kryptering1.sv-fr.reordering-table
> > >
> > > But I cannot find any such entries in my moses.ini - maybe because I'm
> > > running a somewhat older version of Moses. I've found e.g. the
> following
> > > lines in my ini-file:
> > >
> > > [ttable-file]
> > > 0 0 0 5 /home/per/working/train/model/phrase-table.gz
> > >
> > > # distortion (reordering) files
> > > [distortion-file]
> > > 0-0 wbe-msd-bidirectional-fe-allff 6
> > >
> /home/per/working/train/model/reordering-table.wbe-msd-bidirectional-fe.gz
> > >
> > > How should I change the entries to use my binarised model?
> > >
> > > Yours,
> > > Per Tunedal
> > >
> > > _______________________________________________
> > > Moses-support mailing list
> > > Moses-support@mit.edu
> > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > >
> >
> >
> >
> > --
> >
> > [image: Description : Description : lingua_custodia_final full logo]
> >
> > *The Translation Trustee*
> >
> > *1, Place Charles de Gaulle*
> >
> > *78180 Montigny-le-Bretonneux*
> >
> > *Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*
> >
> > *Email :* *massinissa.ahmim@linguacustodia.com
> > <massinissa.ahmim@linguacustodia.com>*
> >
> > *Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> -
> > www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*
> >
> > ? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.
> >
> > Please do not print this email unless it is absolutely necessary. Spread
> > environmental awareness.
> > Email had 1 attachment:
> > + image001.jpg
> > 6k (image/jpeg)
>



--

[image: Description : Description : lingua_custodia_final full logo]

*The Translation Trustee*

*1, Place Charles de Gaulle*

*78180 Montigny-le-Bretonneux*

*Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*

*Email :* *massinissa.ahmim@linguacustodia.com
<massinissa.ahmim@linguacustodia.com>*

*Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> -
www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*

? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.

Please do not print this email unless it is absolutely necessary. Spread
environmental awareness.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140228/22933773/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 4421 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140228/22933773/attachment.jpg

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 88, Issue 68
*********************************************

Tweet