Moses-support Digest, Vol 90, Issue 18

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: how to integrate a long-span LM into moses (Hieu Hoang)
2. Moses on Windows translation issue. (Steven)
3. Re: Moses on Windows translation issue. (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Mon, 7 Apr 2014 12:19:16 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] how to integrate a long-span LM into
moses
To: David Mrva <davidm@cantabresearch.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbi7nrA91Ne526Y364CtCkCoEjCiU48XNrYVixRWys+GvQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

It uses the state to recombine hypotheses, discarding the worse scoring
hypo where 2 have the same states.
http://www.statmt.org/moses/?n=Moses.Background
It doesn't have anything to do with search-algorithm


On 4 April 2014 16:14, David Mrva <davidm@cantabresearch.com> wrote:

>
> >> end of the hypothesis.
> > Your LM state is dependent on the entire target phrase? ie. these
> > target phrases have difference states:
> > a b c d e f g h i j
> > z b c d e f g h i j
> > This would probably negatively impact search as the stacks will have
> > to be pruned more often, leading to search errors.
> >
> > I think this is also the experience of people trying to add syntactic
> > LM to SMT decoders
>
> Hi Hieu,
>
> How does moses use the LM state? If I used the same state for both
> phrases in your example and different LM scores, would moses keep both
> hypotheses in its search space immediately after appending your two
> phrases or would it discard one of them? Is this behaviour dependant on
> the choice of the search-algorithm?
>
> David
>
> >> And when a hypothesis is being extended, its LM
> >> state is extended by one target word at a time in a loop over the new
> >> phrase from start to finish. Ngram LM implementation does not work in
> >> this way and it seems to harm ngram performance. Can anyone shed some
> >> light on the motivation behind the behaviour described above in
> >> points 1-3?
> >>
> >> I used moses with its default, a.k.a. "normal", search algorithm (no
> >> [search-algorithm] variable specified in my config). For completeness,
> >> my config when using moses with its Kenlm class is pasted below.
> >>
> >> Best regards,
> >> David
> >>
> >>
> >> # input factors
> >> [input-factors]
> >> 0
> >>
> >> # mapping steps
> >> [mapping]
> >> 0 T 0
> >>
> >> [distortion-limit]
> >> 6
> >>
> >> # feature functions
> >> [feature]
> >> UnknownWordPenalty
> >> WordPenalty
> >> PhrasePenalty
> >> PhraseDictionaryMemory name=TranslationModel0 table-limit=20
> >> num-features=4 path=model/phrase-table.1.gz input-factor=0
> >> output-factor=0
> >> LexicalReordering name=LexicalReordering0 num-features=6
> >> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> >> path=model/reordering-table.1.wbe-msd-bidirectional-fe.gz
> >> Distortion
> >> KENLM lazyken=1 name=LM0 factor=0 path=lm/europarl.binlm.1 order=5
> >>
> >> # dense weights for feature functions
> >> [weight]
> >> UnknownWordPenalty0= 1
> >> WordPenalty0= -1
> >> PhrasePenalty0= 0.2
> >> TranslationModel0= 0.2 0.2 0.2 0.2
> >> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
> >> Distortion0= 0.3
> >> LM0= 0.5
> >>
> >>
> >>
> >> _______________________________________________
> >> Moses-support mailing list
> >> Moses-support@mit.edu
> >> http://mailman.mit.edu/mailman/listinfo/moses-support
> >>
> >
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140407/796afcbe/attachment-0001.htm

------------------------------

Message: 2
Date: Mon, 7 Apr 2014 10:49:30 -0400
From: "Steven" <steven.xu@lba.ca>
Subject: [Moses-support] Moses on Windows translation issue.
To: <moses-support@mit.edu>
Message-ID: <034101cf5270$94e7aea0$beb70be0$@xu@lba.ca>
Content-Type: text/plain; charset="us-ascii"

Hi, All,

I am running Moses on Windows 7. I installed the Release 2.1 build by
Capita TI: Release 2.1 MSI Installer with GUI for Windows 32bit (Maximum
6-gram LM).



I translated same English Text into French through GUI and command line.

I don't understand where I get different translations since everything is
same for Moses.

The translation done by GUI is better than by command line. In the command
line, there are two words: CONTENT, CORRECTIONS not translated at all.



Those translations are consistent among multiple runs by GUI or command
line.



1) Original English Text:

Everyone can and should contribute content, corrections or comments.

2) Translated through GUI:

Tout le monde peut et doit contribuer contenu, de corrections - ou
d'observations

3) Translated though command line ( Command:
C:\Users\steven.xu\mosesgui\Model_2014-03-21_153233\model>moses-cmd -f
moses.ini )

Toute personne peut et doit contribuer content, corrections ou comments.



Thanks.



Steven.





-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140407/6436c823/attachment-0001.htm

------------------------------

Message: 3
Date: Mon, 7 Apr 2014 16:53:31 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Moses on Windows translation issue.
To: Steven <steven.xu@lba.ca>, moses-support <moses-support@mit.edu>
Cc: Jie Jiang <jie.jiang@capita-ti.com>
Message-ID:
<CAEKMkbhXBRrEWpU9tnFv+o29YMHJXfQ8XmV86ukNLqzjN8DOiw@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

It looks like you haven't tokenized your command line input.You also have
to lowercase or truecase your input, and detokenize and recase your output.

The GUI does all or most of this for you, but not the command line Moses.




On 7 April 2014 16:46, Steven <steven.xu@lba.ca> wrote:

> Hi, Hieu, Jie Jiang
>
>
>
> Thank you all for the reply.
>
>
>
> The language model is the "A small English to French sample model built
> from Europarl: http://statmt.org/~jie/models/EnFr4MosesGUI.zip".
>
>
>
> All moses tools are installed by the Capita TI: Release 2.1 MSI Installer.
>
>
>
> I added "C:\Program Files (x86)\Moses\MosesDecoder" to my environment path.
>
>
>
> Here is the copy of how I do on command line:
>
>
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
> Microsoft Windows [Version 6.1.7601]
>
> Copyright (c) 2009 Microsoft Corporation. All rights reserved.
>
>
>
> C:\Users\steven.xu>cd
> C:\Users\steven.xu\mosesgui\Model_2014-03-21_153233\model
>
>
>
> C:\Users\steven.xu\mosesgui\Model_2014-03-21_153233\model>moses-cmd -f
> moses.ini
>
>
>
>
>
> line=KENLM factor=0 order=5 num-features=1 lazyken=1 path=enfr200k.fr.lm
>
> Loading the LM will be faster if you build a binary file.
>
> Reading enfr200k.fr.lm
>
>
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80
>
> ---85---90---95--100
>
>
> ********************************************************************************
>
> ********************
>
> line=Distortion
>
> line=LexicalReordering type=wbe-msd-bidirectional-fe-allff input-factor=0
> output
>
> -factor=0 num-features=6 path=reordering-table.wbe-msd-bidirectional-fe
>
> Initializing LexicalReordering..
>
> line=WordPenalty
>
> line=UnknownWordPenalty
>
> line=PhraseDictionaryBinary input-factor=0 output-factor=0
> path=phrase-table num
>
> -features=5 table-limit=50
>
> Everyone can and should contribute content, corrections or comments.
>
> Toute personne peut et doit contribuer content, corrections ou comments.
>
> BEST TRANSLATION: Toute personne peut et doit contribuer
> content,|UNK|UNK|UNK co
>
> rrections ou comments.|UNK|UNK|UNK [111111111] [total=-206.692]
> core=(-73.396,0
>
>
> .000,-2.184,0.000,0.000,-2.039,0.000,0.000,-10.000,-200.000,-4.238,-6.468,-3.253
>
> ,-9.343,4.999)
>
>
>
> Thanks,
>
>
>
> Steven Xu
>
>
>
>
>
>
>
> *From:* hieuhoang@gmail.com [mailto:hieuhoang@gmail.com] *On Behalf Of *Hieu
> Hoang
> *Sent:* Monday, April 07, 2014 11:23 AM
> *To:* Jie Jiang
> *Cc:* Steven
> *Subject:* ***PROBABLE PHISHING*** Re: [Moses-support] Moses on Windows
> translation issue.
>
>
>
> hey steve
>
> what's the exact command line you're running? What do you get from the
> command line & gui moses?
>
> are your models using recasing or truecasing, and did you apply those as
> well?
>
>
>
> On 7 April 2014 15:56, Jie Jiang <jie.jiang@capita-ti.com> wrote:
>
> Hi Steven,
> Sorry for the problem, we will look at it asap and keep you posted.
>
>
>
> Best regards,
>
> *Jie Jiang*
> Senior Language Technology Specialist
>
>
> Capita Translation and Interpreting
>
> Riverside Court, Huddersfield Road, Delph, Oldham, OL3 5FZ | Tel (UK): +44
> 845 367 7000 | Tel (US): +1 (800) 579-5010
> Tel Direct: +44 (0)844 854 8984 | mail.jie.jiang@gmail.com | Skype ID:
> jie.jiang-capita-ti
> www.capitatranslationinterpreting.com
>
>
>
> 2014-04-07 15:49 GMT+01:00 Steven <steven.xu@lba.ca>:
>
> Hi, All,
>
> I am running Moses on Windows 7. I installed the Release 2.1 build by
> Capita TI: Release 2.1 MSI Installer with GUI for Windows 32bit (Maximum
> 6-gram LM).
>
>
>
> I translated same English Text into French through GUI and command line.
>
> I don't understand where I get different translations since everything
> is same for Moses.
>
> The translation done by GUI is better than by command line. In the
> command line, there are two words: CONTENT, CORRECTIONS not translated at
> all.
>
>
>
> Those translations are consistent among multiple runs by GUI or command
> line.
>
>
>
> 1) Original English Text:
>
> Everyone can and should contribute content, corrections or comments.
>
> 2) Translated through GUI:
>
> Tout le monde peut et doit contribuer contenu, de corrections - ou
> d'observations
>
> 3) Translated though command line ( Command:
> C:\Users\steven.xu\mosesgui\Model_2014-03-21_153233\model>moses-cmd -f
> moses.ini )
>
> Toute personne peut et doit contribuer content, corrections ou comments.
>
>
>
> Thanks.
>
>
>
> Steven.
>
>
>
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140407/e2260c1b/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 90, Issue 18
*********************************************

Tweet