Moses-support Digest, Vol 104, Issue 81

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Major bug found in Moses (Read, James C)
2. Re: Major bug found in Moses (Read, James C)


----------------------------------------------------------------------

Message: 1
Date: Wed, 24 Jun 2015 13:00:20 +0000
From: "Read, James C" <jcread@essex.ac.uk>
Subject: Re: [Moses-support] Major bug found in Moses
To: Raphael Payen <raphael.payen@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<DB3PR06MB0713B97BBF711CEA5816D99F85AF0@DB3PR06MB0713.eurprd06.prod.outlook.com>

Content-Type: text/plain; charset="iso-8859-1"

Improvements in 37 BLEU points over the default behaviour was not enough to show that there are problems with the default?


James


________________________________
From: Raphael Payen <raphael.payen@gmail.com>
Sent: Sunday, June 21, 2015 5:29 PM
To: Read, James C
Cc: moses-support@mit.edu
Subject: Re: [Moses-support] Major bug found in Moses

James, did you try the modifications Philip suggested (removing the word penalty and lowering p(f|e)?
(I doubt it will be enough to get a best paper award, but it would probably improve your bleu, that's always a good start :) )



On Friday, June 19, 2015, Read, James C <jcread@essex.ac.uk<mailto:jcread@essex.ac.uk>> wrote:

So, all I did was filter out the less likely phrase pairs and the BLEU score shot up. Was that such a stroke of genius? Was that not blindingly obvious?


Your telling me that redesigning the search algorithm to prefer higher scoring phrase pairs is all we need to do to get a best paper at ACL?


James


________________________________
From: Lane Schwartz <dowobeha@gmail.com>
Sent: Friday, June 19, 2015 7:40 PM
To: Read, James C
Cc: Philipp Koehn; Burger, John D.; moses-support@mit.edu
Subject: Re: [Moses-support] Major bug found in Moses

On Fri, Jun 19, 2015 at 11:28 AM, Read, James C <jcread@essex.ac.uk> wrote:

What I take issue with is the en-masse denial that there is a problem with the system if it behaves in such a way with no LM + no pruning and/or tuning.

There is no mass denial taking place.

Regardless of whether or not you tune, the decoder will do its best to find translations with the highest model score. That is the expected behavior.

What I have tried to tell you, and what other people have tried to tell you, is that translations with high model scores are not necessarily good translations.

We all want our models to be such that high model scores correspond to good translations, and that low model scores correspond with bad translations. But unfortunately, our models do not innately have this characteristic. We all know this. We also know a good way to deal with this shortcoming, namely tuning. Tuning is the process by which we attempt to ensure that high model scores correspond to high quality translations, and that low model scores correspond to low quality translations.

If you can design models that naturally correspond with translation quality without tuning, that's great. If you can do that, you've got a great shot at winning a Best Paper award at ACL.

In the meantime, you may want to consider an apology for your rude behavior and unprofessional attitude.

Goodbye.
Lane

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150624/8eab072b/attachment-0001.htm

------------------------------

Message: 2
Date: Wed, 24 Jun 2015 13:04:43 +0000
From: "Read, James C" <jcread@essex.ac.uk>
Subject: Re: [Moses-support] Major bug found in Moses
To: Matthias Huck <mhuck@inf.ed.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>, "Arnold, Doug"
<doug@essex.ac.uk>
Message-ID:
<DB3PR06MB07138F4C7FF199EC1B64BBF485AF0@DB3PR06MB0713.eurprd06.prod.outlook.com>

Content-Type: text/plain; charset="iso-8859-1"

I think it is you who seems to have missed the point.

If the default behaviour is giving BLEU scores considerably lower than the BLEU score obtained from merely selecting the most likely translation of each phrase then there is evidently something very wrong with the default behaviour.

If you can't see something as blindingly simple as that then at this point I'm thinking this really isn't a field I want anything to do wiht.

James

________________________________________
From: Matthias Huck <mhuck@inf.ed.ac.uk>
Sent: Friday, June 19, 2015 10:45 PM
To: Read, James C
Cc: Hieu Hoang; moses-support@mit.edu; Arnold, Doug
Subject: Re: [Moses-support] Major bug found in Moses

Hi James,

Well, it's pretty straightforward: The decoder's job is to find the
hypothesis with the maximum model score. That's why everybody builds
models which assign high model score to high-quality translations.
Unfortunately, you missed this last point in your own work.

Cheers,
Matthias


On Fri, 2015-06-19 at 14:15 +0000, Read, James C wrote:
> I'm gonna try once more. This is what he said:
>
> "the decoder's job is NOT to find the high quality translation"
>
> The next time I have a panel of potential investors in front of me
> I'm gonna pass that line by them and see how it goes down. I stress
> the words HIGH QUALITY TRANSLATION.
>
> Please promise me that the next time you put in a bid for funding you
> will guarantee your prospective funders that under no circumstances
> will you attempt to design a system which searches for HIGH QUALITY
> TRANSLATION.
>
> James
>
> ________________________________________
> From: Matthias Huck <mhuck@inf.ed.ac.uk>
> Sent: Friday, June 19, 2015 5:08 PM
> To: Read, James C
> Cc: Hieu Hoang; moses-support@mit.edu; Arnold, Doug
> Subject: Re: [Moses-support] Major bug found in Moses
>
> Hi James,
>
> Yes, he just said that.
>
> The decoder's job is to find the hypothesis with the maximum model
> score. That's one reason why your work is flawed. You did not care at
> all whether your model score correlates with BLEU or not.
>
> Cheers,
> Matthias
>
>
> On Fri, 2015-06-19 at 13:24 +0000, Read, James C wrote:
> > I quote:
> >
> >
> > "the decoder's job is NOT to find the high quality translation"
> >
> >
> >
> > Did you REALLY just say that?
> >
> >
> > James
> >
> >
> >
> >
> > ______________________________________________________________________
> > From: Hieu Hoang <hieuhoang@gmail.com>
> > Sent: Wednesday, June 17, 2015 9:00 PM
> > To: Read, James C
> > Cc: Kenneth Heafield; moses-support@mit.edu; Arnold, Doug
> > Subject: Re: [Moses-support] Major bug found in Moses
> >
> > the decoder's job is NOT to find the high quality translation (as
> > measured by bleu). It's job is to find translations with high model
> > score.
> >
> >
> > you need the tuning to make sure high quality translation correlates
> > with high model score. If you don't tune, it's pot luck what quality
> > you get.
> >
> >
> > You should tune with the features you use
> >
> >
> > Hieu Hoang
> > Researcher
> >
> > New York University, Abu Dhabi
> >
> > http://www.hoang.co.uk/hieu
> >
> >
> > On 17 June 2015 at 21:52, Read, James C <jcread@essex.ac.uk> wrote:
> > The analogy doesn't seem to be helping me understand just how
> > exactly it is a desirable quality of a TM to
> >
> > a) completely break down if no LM is used (thank you for
> > showing that such is not always the case)
> > b) be dependent on a tuning step to help it find the higher
> > scoring translations
> >
> > What you seem to be essentially saying is that the TM cannot
> > find the higher scoring translations because I didn't pretune
> > the system to do so. And I am supposed to accept that such is
> > a desirable quality of a system whose very job is to find the
> > higher scoring translations.
> >
> > Further, I am still unclear which features you prequire a
> > system to be tuned on. At the very least it seems that I have
> > discovered the selection process that tuning seems to be
> > making up for in some unspecified and altogether opaque way.
> >
> > James
> >
> >
> > ________________________________________
> > From: Hieu Hoang <hieuhoang@gmail.com>
> > Sent: Wednesday, June 17, 2015 8:34 PM
> > To: Read, James C; Kenneth Heafield; moses-support@mit.edu
> > Cc: Arnold, Doug
> > Subject: Re: [Moses-support] Major bug found in Moses
> >
> > 4 BLEU is nothing to sniff at :) I was answering Ken's tangent
> > aspersion
> > that LM are needed for tuning.
> >
> > I have some sympathy for you. You're looking at ways to
> > improve
> > translation by reducing the search space. I've bashed my head
> > against
> > this wall for a while as well without much success.
> >
> > However, as everyone is telling you, you haven't understood
> > the role of
> > tuning. Without tuning, you're pointing your lab rat to some
> > random part
> > of the search space, instead of away from the furry animal
> > with whiskers
> > and towards the yellow cheesy thing
> >
> > On 17/06/2015 20:45, Read, James C wrote:
> > > Doesn't look like the LM is contributing all that much then
> > does it?
> > >
> > > James
> > >
> > > ________________________________________
> > > From: moses-support-bounces@mit.edu
> > <moses-support-bounces@mit.edu> on behalf of Hieu Hoang
> > <hieuhoang@gmail.com>
> > > Sent: Wednesday, June 17, 2015 7:35 PM
> > > To: Kenneth Heafield; moses-support@mit.edu
> > > Subject: Re: [Moses-support] Major bug found in Moses
> > >
> > > On 17/06/2015 20:13, Kenneth Heafield wrote:
> > >> I'll bite.
> > >>
> > >> The moses.ini files ship with bogus feature weights. One
> > is required to
> > >> tune the system to discover good weights for their system.
> > You did not
> > >> tune. The results of an untuned system are meaningless.
> > >>
> > >> So for example if the feature weights are all zeros, then
> > the scores are
> > >> all zero. The system will arbitrarily pick some awful
> > translation from
> > >> a large space of translations.
> > >>
> > >> The filter looks at one feature p(target | source). So now
> > you've
> > >> constrained the awful untuned model to a slightly better
> > region of the
> > >> search space.
> > >>
> > >> In other words, all you've done is a poor approximation to
> > manually
> > >> setting the weight to 1.0 on p(target | source) and the
> > rest to 0.
> > >>
> > >> The problem isn't that you are running without a language
> > model (though
> > >> we generally do not care what happens without one). The
> > problem is that
> > >> you did not tune the feature weights.
> > >>
> > >> Moreover, as Marcin is pointing out, I wouldn't necessarily
> > expect
> > >> tuning to work without an LM.
> > > Tuning does work without a LM. The results aren't half bad.
> > fr-en
> > > europarl (pb):
> > > with LM: 22.84
> > > retuned without LM: 18.33
> > >> On 06/17/15 11:56, Read, James C wrote:
> > >>> Actually the approximation I expect to be:
> > >>>
> > >>> p(e|f)=p(f|e)
> > >>>
> > >>> Why would you expect this to give poor results if the TM
> > is well trained? Surely the results of my filtering
> > experiments provve otherwise.
> > >>>
> > >>> James
> > >>>
> > >>> ________________________________________
> > >>> From: moses-support-bounces@mit.edu
> > <moses-support-bounces@mit.edu> on behalf of Rico Sennrich
> > <rico.sennrich@gmx.ch>
> > >>> Sent: Wednesday, June 17, 2015 5:32 PM
> > >>> To: moses-support@mit.edu
> > >>> Subject: Re: [Moses-support] Major bug found in Moses
> > >>>
> > >>> Read, James C <jcread@...> writes:
> > >>>
> > >>>> I have been unable to find a logical explanation for this
> > behaviour other
> > >>> than to conclude that there must be some kind of bug in
> > Moses which causes a
> > >>> TM only run of Moses to perform poorly in finding the most
> > likely
> > >>> translations according to the TM when
> > >>>> there are less likely phrase pairs included in the
> > race.
> > >>> I may have overlooked something, but you seem to have
> > removed the language
> > >>> model from your config, and used default weights. your
> > default model will
> > >>> thus (roughly) implement the following model:
> > >>>
> > >>> p(e|f) = p(e|f)*p(f|e)
> > >>>
> > >>> which is obviously wrong, and will give you poor results.
> > This is not a bug
> > >>> in the code, but a poor choice of models and weights.
> > Standard steps in SMT
> > >>> (like tuning the model weights on a development set, and
> > including a
> > >>> language model) will give you the desired results.
> > >>>
> > >>> _______________________________________________
> > >>> Moses-support mailing list
> > >>> Moses-support@mit.edu
> > >>> http://mailman.mit.edu/mailman/listinfo/moses-support
> > >>>
> > >>> _______________________________________________
> > >>> Moses-support mailing list
> > >>> Moses-support@mit.edu
> > >>> http://mailman.mit.edu/mailman/listinfo/moses-support
> > >> _______________________________________________
> > >> Moses-support mailing list
> > >> Moses-support@mit.edu
> > >> http://mailman.mit.edu/mailman/listinfo/moses-support
> > >>
> > > --
> > > Hieu Hoang
> > > Researcher
> > > New York University, Abu Dhabi
> > > http://www.hoang.co.uk/hieu
> > >
> > > _______________________________________________
> > > Moses-support mailing list
> > > Moses-support@mit.edu
> > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > > .
> > >
> >
> > --
> > Hieu Hoang
> > Researcher
> > New York University, Abu Dhabi
> > http://www.hoang.co.uk/hieu
> >
> >
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>



--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 104, Issue 81
**********************************************

Tweet