Moses-support Digest, Vol 103, Issue 27

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Adding a language model built on Google Web (Hieu Hoang)
2. Re: Multiple Translation Tables and Back-off Models in Moses
(Philipp Koehn)


----------------------------------------------------------------------

Message: 1
Date: Thu, 07 May 2015 21:46:02 +0400
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Adding a language model built on Google
Web
To: Alla Rozovskaya <sigaliyah@gmail.com>, moses-support
<moses-support@mit.edu>
Message-ID: <554BA4DA.9070909@gmail.com>
Content-Type: text/plain; charset="utf-8"



On 07/05/2015 20:43, Alla Rozovskaya wrote:
> Hi Hieu,
>
> Please see below.
>
> (1) Thank you for your help. I am using a standard LM training for
> now, without the N-gram corpus. Could you please let me know how I can
> train a translation model without any reordering model (I did not find
> anything in the manual about it).
> what's the exact command you're using to create the translation model
> and reordering model?
>
> $mosesDir/scripts/training/train-model.perl -root-dir
> $outputDir/mosesFiles/ -corpus $corpusDir/$trainFile.clean -f
> $source -e $target -alignment grow-diag-final-and -reordering
> msd-bidirectional-fe -lm 0:$order:$lm1 -external-bin-dir
> /scratch/rozovska/mosesdecoder/tools/ -cores 2 --max-phrase-length 7
>
I don't know how to prevent the creation of the re-ordering model.
However, if you don't want to use it in decoding, then just delete the
LexicalReordering and Distortion model entries in the moses.ini file
BEFORE you tune.

> I tried removing Distortion and LexicalReordering from the moses.ini
> file but I still found these cases in the translated file.
>
> (2) Also, can I use multiple language models when training a
> translation model, or is it only possible to specify multiple LMs in
> the moses.ini file?
> the LM isn't used when creating the translation model. They're only
> used in decoding
>
> What is the -lm parameter in train-model.perl used for?
the -lm parameter is just used to add the LM model entry to the
moses.ini file.

In fact, if you don't like the moses.ini that is created automatically
for you, create your own that has just the models you need and use that
for tuning and decoding
>
> Thanks a lot for your help,
>
> Alla
>
> On Wed, May 6, 2015 at 12:12 AM, Hieu Hoang <hieuhoang@gmail.com
> <mailto:hieuhoang@gmail.com>> wrote:
>
>
>
> Hieu Hoang
> Researcher
> New York University, Abu Dhabi
> http://www.hoang.co.uk/hieu
>
> On 6 May 2015 at 01:36, Alla Rozovskaya <sigaliyah@gmail.com
> <mailto:sigaliyah@gmail.com>> wrote:
>
> Hi Hieu,
>
> Thank you for your help. I am using a standard LM training for
> now, without the N-gram corpus. Could you please let me know
> how I can train a translation model without any reordering
> model (I did not find anything in the manual about it).
>
> what's the exact command you're using to create the translation
> model and reordering model?
>
> Also, can I use multiple language models when training a
> translation model, or is it only possible to specify multiple
> LMs in the moses.ini file?
>
> the LM isn't used when creating the translation model. They're
> only used in decoding
>
>
> Thanks again,
>
> Alla
>
> On Tue, Apr 28, 2015 at 10:06 AM, Hieu Hoang
> <hieuhoang@gmail.com <mailto:hieuhoang@gmail.com>> wrote:
>
> I spoke to Ken about using KenLM to train a standard
> backoff LM with the n-gram corpus. It's not supported yet
> or recommended.
>
> I'm not sure whether the moses' SRILM wrapper will support
> the count-based LM. And how much memory it will consume.
> Try it and please let us know.
>
> People have also been using the Common Crawl corpus to
> build huge backoff LM. They're very difficult to use as it
> consumes a lot of memory
>
>
> On 25/04/2015 20:24, Alla Rozovskaya wrote:
>> Hello,
>>
>> I have built an interpolated count-based LM on the Google
>> Web N-gram corpus using SRILM toolkit, as specified here:
>> http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html
>>
>> Is it possible to use it in moses? In particular, since
>> this model uses count files and a file specifying
>> weights, what is the right way to specify the path in
>> moses.ini?
>>
>> Thank you,
>>
>> Alla
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
> --
> Hieu Hoang
> Researcher
> New York University, Abu Dhabi
> http://www.hoang.co.uk/hieu
>
>
>
>

--
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150507/4c30191b/attachment-0001.htm

------------------------------

Message: 2
Date: Thu, 7 May 2015 13:52:08 -0400
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] Multiple Translation Tables and Back-off
Models in Moses
To: Tomas Fulajtar <TomasFu@moravia.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDBpaC2wc6gPBMyTN6uiyFFszqsYAvSsVvDSVzuG_mAdOw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hi,

these default scores were added fairly recently, so they do not work
for version 2.0.

-phi

On Thu, May 7, 2015 at 7:53 AM, Tomas Fulajtar <TomasFu@moravia.com> wrote:
> Hi,
>
>
>
> Thanks for an excellent explanation. Does this ?default-scores? parameter
> work in 2.x release as well?
>
>
>
> Tomas
>
>
>
>
>
> From: Philipp Koehn [mailto:phi@jhu.edu]
> Sent: Thursday, May 7, 2015 3:22 AM
> To: xinzyang@mail.ustc.edu.cn
> Cc: moses-support@mit.edu
> Subject: Re: [Moses-support] Multiple Translation Tables and Back-off Models
> in Moses
>
>
>
> Hi,
>
>
>
> I added a section to the Moses documentation on how to deal with this
> situation:
>
>
>
> http://www.statmt.org/moses/?n=Advanced.Models#ntoc5
>
>
>
> -phi
>
> Caveat: Multiple Translation Tables and Lexicalized Reordering You may
> specify any number of lexicalized reordering models. Each of them will score
> any translation option, no matter where it comes from. If a lexicalized
> reordering table does not have an entry for a translation options, it will
> not assign any score to it. In other words, such a translation option is
> given the probability 1 no matter how it is reordered. This may not be the
> way you want to handle it.
>
> For instance, if you have an "in-domain" translation table and an
> out-of-domain translation table, you can also provide an in-domain
> reordering table and anout-of-domain reordering table. If a phrase pair
> occurs in both translation tables, it will be scored by both reordering
> tables. However, if a phrase pairs occurs only in one of the phrase tables
> (and hence reordering tables), it will be only score by one of them and get
> a free ride with the other. this will have the undesirable effect of
> discouraging phrase pairs that occur in both tables.
>
> To avoid this, you can add default scores to the reordering table:
>
> LexicalReordering name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff [...]
> default-scores=0.5,0.3,0.2,0.5,0.3,0.2
>
> LexicalReordering name=LexicalReordering1 num-features=6
> type=wbe-msd-bidirectional-fe-allff [...]
> default-scores=0.5,0.3,0.2,0.5,0.3,0.2
>
>
>
>
>
> On Wed, May 6, 2015 at 3:00 AM, <xinzyang@mail.ustc.edu.cn> wrote:
>
> Hi,
>
>
>
> I am studying moses to do some research abou SMT. I have read advanced
> models in moses at the website
> http://www.statmt.org/moses/?n=Advanced.Models#ntoc5 . I have some questions
> about the introduction of "Multiple Translation Tables and Back-off Models".
>
>
>
> From the website, I know that moses allows the use of multiple translation
> tables. But the lexicalized reordering model is not mentioned in the
> introduction of "Multiple Translation Tables and Back-off Models". So, could
> I use two translation tables, which I denote translation table1 and
> translation table2 respectively, and one lexicalized reordering table
> according to the introduction of "Multiple Translation Tables and Back-off
> Models"? The phrase pairs in reordering table is the same with translation
> table1, but not the same with translation table2.
>
>
>
> When I use multiple translation tables and only one reordering table, a
> phrase pair may exist in only one translation table and don't exist in the
> reordering model, how to calculate the probability of lexicalized reordering
> model?
>
>
>
> I'm sorry for bothering you. Thanks and Best Regards,
>
>
>
> Zhenxin
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 103, Issue 27
**********************************************

Tweet