Moses-support Digest, Vol 104, Issue 62

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Major bug found in Moses (Matthias Huck)
2. Re: Dependencies in EMS/Experiment.perl (Philipp Koehn)
3. Re: Major bug found in Moses (Rico Sennrich)


----------------------------------------------------------------------

Message: 1
Date: Fri, 19 Jun 2015 20:45:58 +0100
From: Matthias Huck <mhuck@inf.ed.ac.uk>
Subject: Re: [Moses-support] Major bug found in Moses
To: "Read, James C" <jcread@essex.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>, "Arnold, Doug"
<doug@essex.ac.uk>
Message-ID: <1434743158.30904.1203.camel@portedgar>
Content-Type: text/plain; charset="UTF-8"

Hi James,

Well, it's pretty straightforward: The decoder's job is to find the
hypothesis with the maximum model score. That's why everybody builds
models which assign high model score to high-quality translations.
Unfortunately, you missed this last point in your own work.

Cheers,
Matthias


On Fri, 2015-06-19 at 14:15 +0000, Read, James C wrote:
> I'm gonna try once more. This is what he said:
>
> "the decoder's job is NOT to find the high quality translation"
>
> The next time I have a panel of potential investors in front of me
> I'm gonna pass that line by them and see how it goes down. I stress
> the words HIGH QUALITY TRANSLATION.
>
> Please promise me that the next time you put in a bid for funding you
> will guarantee your prospective funders that under no circumstances
> will you attempt to design a system which searches for HIGH QUALITY
> TRANSLATION.
>
> James
>
> ________________________________________
> From: Matthias Huck <mhuck@inf.ed.ac.uk>
> Sent: Friday, June 19, 2015 5:08 PM
> To: Read, James C
> Cc: Hieu Hoang; moses-support@mit.edu; Arnold, Doug
> Subject: Re: [Moses-support] Major bug found in Moses
>
> Hi James,
>
> Yes, he just said that.
>
> The decoder's job is to find the hypothesis with the maximum model
> score. That's one reason why your work is flawed. You did not care at
> all whether your model score correlates with BLEU or not.
>
> Cheers,
> Matthias
>
>
> On Fri, 2015-06-19 at 13:24 +0000, Read, James C wrote:
> > I quote:
> >
> >
> > "the decoder's job is NOT to find the high quality translation"
> >
> >
> >
> > Did you REALLY just say that?
> >
> >
> > James
> >
> >
> >
> >
> > ______________________________________________________________________
> > From: Hieu Hoang <hieuhoang@gmail.com>
> > Sent: Wednesday, June 17, 2015 9:00 PM
> > To: Read, James C
> > Cc: Kenneth Heafield; moses-support@mit.edu; Arnold, Doug
> > Subject: Re: [Moses-support] Major bug found in Moses
> >
> > the decoder's job is NOT to find the high quality translation (as
> > measured by bleu). It's job is to find translations with high model
> > score.
> >
> >
> > you need the tuning to make sure high quality translation correlates
> > with high model score. If you don't tune, it's pot luck what quality
> > you get.
> >
> >
> > You should tune with the features you use
> >
> >
> > Hieu Hoang
> > Researcher
> >
> > New York University, Abu Dhabi
> >
> > http://www.hoang.co.uk/hieu
> >
> >
> > On 17 June 2015 at 21:52, Read, James C <jcread@essex.ac.uk> wrote:
> > The analogy doesn't seem to be helping me understand just how
> > exactly it is a desirable quality of a TM to
> >
> > a) completely break down if no LM is used (thank you for
> > showing that such is not always the case)
> > b) be dependent on a tuning step to help it find the higher
> > scoring translations
> >
> > What you seem to be essentially saying is that the TM cannot
> > find the higher scoring translations because I didn't pretune
> > the system to do so. And I am supposed to accept that such is
> > a desirable quality of a system whose very job is to find the
> > higher scoring translations.
> >
> > Further, I am still unclear which features you prequire a
> > system to be tuned on. At the very least it seems that I have
> > discovered the selection process that tuning seems to be
> > making up for in some unspecified and altogether opaque way.
> >
> > James
> >
> >
> > ________________________________________
> > From: Hieu Hoang <hieuhoang@gmail.com>
> > Sent: Wednesday, June 17, 2015 8:34 PM
> > To: Read, James C; Kenneth Heafield; moses-support@mit.edu
> > Cc: Arnold, Doug
> > Subject: Re: [Moses-support] Major bug found in Moses
> >
> > 4 BLEU is nothing to sniff at :) I was answering Ken's tangent
> > aspersion
> > that LM are needed for tuning.
> >
> > I have some sympathy for you. You're looking at ways to
> > improve
> > translation by reducing the search space. I've bashed my head
> > against
> > this wall for a while as well without much success.
> >
> > However, as everyone is telling you, you haven't understood
> > the role of
> > tuning. Without tuning, you're pointing your lab rat to some
> > random part
> > of the search space, instead of away from the furry animal
> > with whiskers
> > and towards the yellow cheesy thing
> >
> > On 17/06/2015 20:45, Read, James C wrote:
> > > Doesn't look like the LM is contributing all that much then
> > does it?
> > >
> > > James
> > >
> > > ________________________________________
> > > From: moses-support-bounces@mit.edu
> > <moses-support-bounces@mit.edu> on behalf of Hieu Hoang
> > <hieuhoang@gmail.com>
> > > Sent: Wednesday, June 17, 2015 7:35 PM
> > > To: Kenneth Heafield; moses-support@mit.edu
> > > Subject: Re: [Moses-support] Major bug found in Moses
> > >
> > > On 17/06/2015 20:13, Kenneth Heafield wrote:
> > >> I'll bite.
> > >>
> > >> The moses.ini files ship with bogus feature weights. One
> > is required to
> > >> tune the system to discover good weights for their system.
> > You did not
> > >> tune. The results of an untuned system are meaningless.
> > >>
> > >> So for example if the feature weights are all zeros, then
> > the scores are
> > >> all zero. The system will arbitrarily pick some awful
> > translation from
> > >> a large space of translations.
> > >>
> > >> The filter looks at one feature p(target | source). So now
> > you've
> > >> constrained the awful untuned model to a slightly better
> > region of the
> > >> search space.
> > >>
> > >> In other words, all you've done is a poor approximation to
> > manually
> > >> setting the weight to 1.0 on p(target | source) and the
> > rest to 0.
> > >>
> > >> The problem isn't that you are running without a language
> > model (though
> > >> we generally do not care what happens without one). The
> > problem is that
> > >> you did not tune the feature weights.
> > >>
> > >> Moreover, as Marcin is pointing out, I wouldn't necessarily
> > expect
> > >> tuning to work without an LM.
> > > Tuning does work without a LM. The results aren't half bad.
> > fr-en
> > > europarl (pb):
> > > with LM: 22.84
> > > retuned without LM: 18.33
> > >> On 06/17/15 11:56, Read, James C wrote:
> > >>> Actually the approximation I expect to be:
> > >>>
> > >>> p(e|f)=p(f|e)
> > >>>
> > >>> Why would you expect this to give poor results if the TM
> > is well trained? Surely the results of my filtering
> > experiments provve otherwise.
> > >>>
> > >>> James
> > >>>
> > >>> ________________________________________
> > >>> From: moses-support-bounces@mit.edu
> > <moses-support-bounces@mit.edu> on behalf of Rico Sennrich
> > <rico.sennrich@gmx.ch>
> > >>> Sent: Wednesday, June 17, 2015 5:32 PM
> > >>> To: moses-support@mit.edu
> > >>> Subject: Re: [Moses-support] Major bug found in Moses
> > >>>
> > >>> Read, James C <jcread@...> writes:
> > >>>
> > >>>> I have been unable to find a logical explanation for this
> > behaviour other
> > >>> than to conclude that there must be some kind of bug in
> > Moses which causes a
> > >>> TM only run of Moses to perform poorly in finding the most
> > likely
> > >>> translations according to the TM when
> > >>>> there are less likely phrase pairs included in the
> > race.
> > >>> I may have overlooked something, but you seem to have
> > removed the language
> > >>> model from your config, and used default weights. your
> > default model will
> > >>> thus (roughly) implement the following model:
> > >>>
> > >>> p(e|f) = p(e|f)*p(f|e)
> > >>>
> > >>> which is obviously wrong, and will give you poor results.
> > This is not a bug
> > >>> in the code, but a poor choice of models and weights.
> > Standard steps in SMT
> > >>> (like tuning the model weights on a development set, and
> > including a
> > >>> language model) will give you the desired results.
> > >>>
> > >>> _______________________________________________
> > >>> Moses-support mailing list
> > >>> Moses-support@mit.edu
> > >>> http://mailman.mit.edu/mailman/listinfo/moses-support
> > >>>
> > >>> _______________________________________________
> > >>> Moses-support mailing list
> > >>> Moses-support@mit.edu
> > >>> http://mailman.mit.edu/mailman/listinfo/moses-support
> > >> _______________________________________________
> > >> Moses-support mailing list
> > >> Moses-support@mit.edu
> > >> http://mailman.mit.edu/mailman/listinfo/moses-support
> > >>
> > > --
> > > Hieu Hoang
> > > Researcher
> > > New York University, Abu Dhabi
> > > http://www.hoang.co.uk/hieu
> > >
> > > _______________________________________________
> > > Moses-support mailing list
> > > Moses-support@mit.edu
> > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > > .
> > >
> >
> > --
> > Hieu Hoang
> > Researcher
> > New York University, Abu Dhabi
> > http://www.hoang.co.uk/hieu
> >
> >
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>



--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



------------------------------

Message: 2
Date: Fri, 19 Jun 2015 16:49:44 -0400
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] Dependencies in EMS/Experiment.perl
To: Matthias Huck <mhuck@inf.ed.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>, Evgeny Matusov
<ematusov@apptek.com>
Message-ID:
<CAAFADDDMs-evgcce=1khOtjYTRB0gETW9KRAJ6feD-skbYvQsw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

I would have to see the experiment.meta that you defined to get a sense of
what could be done.

In the meantime, what Matthias suggested works for sure - just point to
files that have been generated by steps that you do not want to rerun.

-phi


On Fri, Jun 19, 2015 at 2:42 PM, Matthias Huck <mhuck@inf.ed.ac.uk> wrote:

> Hi Evgeny,
>
> If setting TRAINING:config won't help, then it might get a bit tricky.
> Another thing you can try is setting filtered-config or filtered-dir in
> the [TUNING] section.
>
> The next workaround I can think of is pointing to existing files in all
> the [CORPUS:*] sections by setting tokenized-stem, clean-stem,
> truecased-stem ...
>
> Similarly in the [LM:*] sections with tokenized-corpus and
> truecased-corpus etc., if defining lm and/or binlm doesn't make it skip
> those steps.
>
> Cheers,
> Matthias
>
>
> On Fri, 2015-06-19 at 16:41 +0000, Evgeny Matusov wrote:
> > Hi,
> >
> >
> > to those of you using Experiment.perl for experiments, maybe you can
> > help me solve the following problem:
> >
> >
> > I added a step to filter full segment overlap of evaluation and tuning
> > data with the training data. This steps removes all sentences from
> > each CORPUS which are also found in EVALUATION and TUNING sentences.
> > Thus, one of the CORPUS steps depend on EVALUATION and TUNING.
> >
> >
> > Now, I want to exchange the tuning corpus I am using, picking another
> > one which was already declared in the EVALUATION section. Thus, the
> > filter against which the overlap is checked does not change, and hence
> > the training data does not need to be filtered again, and therefore
> > neither the alignment training nor LM training or anything else should
> > be repeated, just the tuning step should re-start. However,
> > Experiment.perl is not smart enough to realize this. I tried to add
> > "pass-if" or "ignore-if" step on the filter-overlap step that I
> > declared and set a variable to pass it, but this did not help - all
> > steps after it are still executed. Setting TRAINING:config to a valid
> > moses.ini file helps to prevent the alignment training from running,
> > but not the LM training, nor (more importantly), the several
> > cleaning/lowercasing steps that follow the overlap step for each
> > training corpus.
> >
> >
> > Is there an easy way to block everything below tuning from being
> > repeated, even if the tuning data changes?
> >
> >
> > Thanks,
> >
> > Evgeny.
> >
> >
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150619/632a43d9/attachment-0001.htm

------------------------------

Message: 3
Date: Sat, 20 Jun 2015 14:26:16 +0100
From: Rico Sennrich <rico.sennrich@gmx.ch>
Subject: Re: [Moses-support] Major bug found in Moses
To: moses-support@mit.edu
Message-ID: <558569F8.8090209@gmx.ch>
Content-Type: text/plain; charset=windows-1252; format=flowed

On 19/06/15 19:21, Marcin Junczys-Dowmunt wrote:
> So, if anything, Moses is just a very flexible text-rewriting tool.
> Tuning (and data) turns into a translator, GEC tool, POS-tagger,
> Chunker, Semantic Tagger etc.
that's a good point, and the basis of some criticism that can be
levelled at the Moses community: because Moses is so flexible, the
responsibility is on the user to find the right configuration for a
task. I think it is getting harder to find out about all of the
settings/models necessary to reproduce a state-of-the-art system,
especially outside of an established SMT research group. The results is
a high barrier of entry, and frustration on all sides when somebody
performs experiments with default settings.

To stay with the example of phrase table pruning: this is widely used,
and I used count-based pruning, threshold pruning based on p(e|f), and
histogram pruning based on the model score in my WMT submission. Can and
should we make a wider effort to facilitate the reproduction of systems
by disseminating settings or configuration files? This dissemination is
partially done by system description papers, but they cannot cover all
settings [this would make for a very boring paper]. I put some effort
into documenting my WMT submission by releasing EMS configuration files
( https://github.com/rsennrich/wmt2014-scripts/tree/master/example ),
and I would be happy to see this done more often.


------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 104, Issue 62
**********************************************

Tweet