Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Major bug found in Moses (Read, James C)
2. Re: Major bug found in Moses (Read, James C)
3. Re: Major bug found in Moses (Lane Schwartz)
4. Re: Major bug found in Moses (Read, James C)
----------------------------------------------------------------------
Message: 1
Date: Wed, 24 Jun 2015 13:11:42 +0000
From: "Read, James C" <jcread@essex.ac.uk>
Subject: Re: [Moses-support] Major bug found in Moses
To: Rico Sennrich <rico.sennrich@gmx.ch>, "moses-support@mit.edu"
<moses-support@mit.edu>
Message-ID:
<DB3PR06MB071336CB485489ADE51FB1D085AF0@DB3PR06MB0713.eurprd06.prod.outlook.com>
Content-Type: text/plain; charset="iso-8859-1"
Thank you for reading very careful the draft paper I provided a link to and noticing that the Johnson paper is duly cited there. Given that you had already noticed this I shall not proceed to explain the blinding obvious differences between my very simple filter and their filter based on Fisher's exact test.
Other than that it seems painfully clear that the point I meant to make has not been understood entirely. If the default behaviour produces BLEU scores considerably lower than merely selecting the most likely translation of each phrase then evidently there is something very wrong with the default behaviour. If we cannot agree on something as obvious as that then I really can't see this discussion making any productive progress.
James
________________________________________
From: moses-support-bounces@mit.edu <moses-support-bounces@mit.edu> on behalf of Rico Sennrich <rico.sennrich@gmx.ch>
Sent: Friday, June 19, 2015 8:25 PM
To: moses-support@mit.edu
Subject: Re: [Moses-support] Major bug found in Moses
[sorry for the garbled message before]
you are right. The idea is pretty obvious. It roughly corresponds to
'Histogram pruning' in this paper:
Zens, R., Stanton, D., Xu, P. (2012). A Systematic Comparison of Phrase
Table Pruning Technique. In Proceedings of the 2012 Joint Conference on
Empirical Methods in Natural Language Processing and Computational
Natural Language Learning (EMNLP-CoNLL), pp. 972-983.
The idea has been described in the literature before that (for instance,
Johnson et al. (2007) only use the top 30 phrase pairs per source
phrase), and may have been used in practice for even longer. If you read
the paper above, you will find that histogram pruning does not improve
translation quality on a state-of-the-art SMT system, and performs
poorly compared to more advanced pruning techniques.
On 19.06.2015 17:49, Read, James C. wrote:
> So, all I did was filter out the less likely phrase pairs and the BLEU score shot up. Was that such a stroke of genius? Was that not blindingly obvious?
>
>
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
------------------------------
Message: 2
Date: Wed, 24 Jun 2015 13:15:50 +0000
From: "Read, James C" <jcread@essex.ac.uk>
Subject: Re: [Moses-support] Major bug found in Moses
To: amittai axelrod <amittai@umiacs.umd.edu>, Lane Schwartz
<dowobeha@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>, Philipp Koehn
<phi@jhu.edu>
Message-ID:
<DB3PR06MB071326B9E5F9D419DBC46F7E85AF0@DB3PR06MB0713.eurprd06.prod.outlook.com>
Content-Type: text/plain; charset="iso-8859-1"
Thank you for such an invitation. Let's see. Given the choice of
a) reading through thousands of lines of code trying to figure out why the default behaviour performs considerably worse than merely selecting the most likely translation of each phrase or
b) spending much less time implementing a simple system that does just that
which one would you do?
For all know maybe I've already implemented such a system that does just that and not only that improves considerably on such a basic benchmark. But given that on this list we don't seem to be able to accept that there is a problem with the default behaviour of Moses I can only conclude that nobody would be interested in access to the code of such a system.
James
________________________________________
From: amittai axelrod <amittai@umiacs.umd.edu>
Sent: Friday, June 19, 2015 7:52 PM
To: Read, James C; Lane Schwartz
Cc: moses-support@mit.edu; Philipp Koehn
Subject: Re: [Moses-support] Major bug found in Moses
if we don't understand the problem, how can we possibly fix it?
all the relevant code is open source. go for it!
~amittai
On 6/19/15 12:49, Read, James C wrote:
> So, all I did was filter out the less likely phrase pairs and the BLEU
> score shot up. Was that such a stroke of genius? Was that not blindingly
> obvious?
>
>
> Your telling me that redesigning the search algorithm to prefer higher
> scoring phrase pairs is all we need to do to get a best paper at ACL?
>
>
> James
>
>
>
> ------------------------------------------------------------------------
> *From:* Lane Schwartz <dowobeha@gmail.com>
> *Sent:* Friday, June 19, 2015 7:40 PM
> *To:* Read, James C
> *Cc:* Philipp Koehn; Burger, John D.; moses-support@mit.edu
> *Subject:* Re: [Moses-support] Major bug found in Moses
> On Fri, Jun 19, 2015 at 11:28 AM, Read, James C <jcread@essex.ac.uk
> <mailto:jcread@essex.ac.uk>> wrote:
>
> What I take issue with is the en-masse denial that there is a
> problem with the system if it behaves in such a way with no LM + no
> pruning and/or tuning.
>
>
> There is no mass denial taking place.
>
> Regardless of whether or not you tune, the decoder will do its best to
> find translations with the highest model score. That is the expected
> behavior.
>
> What I have tried to tell you, and what other people have tried to tell
> you, is that translations with high model scores are not necessarily
> good translations.
>
> We all want our models to be such that high model scores correspond to
> good translations, and that low model scores correspond with bad
> translations. But unfortunately, our models do not innately have this
> characteristic. We all know this. We also know a good way to deal with
> this shortcoming, namely tuning. Tuning is the process by which we
> attempt to ensure that high model scores correspond to high quality
> translations, and that low model scores correspond to low quality
> translations.
>
> If you can design models that naturally correspond with translation
> quality without tuning, that's great. If you can do that, you've got a
> great shot at winning a Best Paper award at ACL.
>
> In the meantime, you may want to consider an apology for your rude
> behavior and unprofessional attitude.
>
> Goodbye.
> Lane
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
------------------------------
Message: 3
Date: Wed, 24 Jun 2015 08:43:01 -0500
From: Lane Schwartz <dowobeha@gmail.com>
Subject: Re: [Moses-support] Major bug found in Moses
To: "Read, James C" <jcread@essex.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZ=P30=UaPFULVstFfyOokQB+KmRFAmCtbLL+Fg3gwPxYw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
On Wed, Jun 24, 2015 at 8:11 AM, Read, James C <jcread@essex.ac.uk> wrote:
>
>
> Other than that it seems painfully clear that the point I meant to make
> has not been understood entirely. If the default behaviour produces BLEU
> scores considerably lower than merely selecting the most likely translation
> of each phrase then evidently there is something very wrong with the
> default behaviour. If we cannot agree on something as obvious as that then
> I really can't see this discussion making any productive progress.
>
James,
I understand your point. I think that the others who have responded also
understand your point.
We simply disagree with your conclusion.
I encourage you to consider the possibility that if the many experts in
this field who have responded all think that your conclusion is flawed,
then there might be something to that.
I will agree, though, that this is a good time to conclude this discussion.
Sincerely,
Lane Schwartz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150624/78f0b0b9/attachment-0001.htm
------------------------------
Message: 4
Date: Wed, 24 Jun 2015 14:05:49 +0000
From: "Read, James C" <jcread@essex.ac.uk>
Subject: Re: [Moses-support] Major bug found in Moses
To: Lane Schwartz <dowobeha@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<DB3PR06MB07138E6E81C97B73046723F885AF0@DB3PR06MB0713.eurprd06.prod.outlook.com>
Content-Type: text/plain; charset="iso-8859-1"
As the title of this thread makes clear the purpose of reporting the bug was not to invite a discussion about conclusions made in my draft paper. Clearly a community that builds its career around research in SMT is unlikely to agree with those kinds of conclusions. The purpose was to report the flaw in the default behaviour of Moses in the hope that we could all agree that something ought to be done about it.
So far you seem to be the only one who has come even close to acknowledging that there is a problem with Moses default behaviour.
James
________________________________
From: Lane Schwartz <dowobeha@gmail.com>
Sent: Wednesday, June 24, 2015 4:43 PM
To: Read, James C
Cc: Rico Sennrich; moses-support@mit.edu
Subject: Re: [Moses-support] Major bug found in Moses
On Wed, Jun 24, 2015 at 8:11 AM, Read, James C <jcread@essex.ac.uk<mailto:jcread@essex.ac.uk>> wrote:
Other than that it seems painfully clear that the point I meant to make has not been understood entirely. If the default behaviour produces BLEU scores considerably lower than merely selecting the most likely translation of each phrase then evidently there is something very wrong with the default behaviour. If we cannot agree on something as obvious as that then I really can't see this discussion making any productive progress.
James,
I understand your point. I think that the others who have responded also understand your point.
We simply disagree with your conclusion.
I encourage you to consider the possibility that if the many experts in this field who have responded all think that your conclusion is flawed, then there might be something to that.
I will agree, though, that this is a good time to conclude this discussion.
Sincerely,
Lane Schwartz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150624/0ec8b0c6/attachment.htm
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 104, Issue 82
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 104, Issue 82"
Post a Comment