Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Major bug found in Moses (Rico Sennrich)
2. Re: Major bug found in Moses (Read, James C)
3. Re: Major bug found in Moses (Read, James C)
4. Re: Major bug found in Moses (Hieu Hoang)
5. Re: Major bug found in Moses (Read, James C)
----------------------------------------------------------------------
Message: 1
Date: Wed, 17 Jun 2015 16:17:44 +0000 (UTC)
From: Rico Sennrich <rico.sennrich@gmx.ch>
Subject: Re: [Moses-support] Major bug found in Moses
To: moses-support@mit.edu
Message-ID: <loom.20150617T180218-176@post.gmane.org>
Content-Type: text/plain; charset=us-ascii
Read, James C <jcread@...> writes:
>
> Actually the approximation I expect to be:
>
> p(e|f)=p(f|e)
>
> Why would you expect this to give poor results if the TM is well trained?
Surely the results of my filtering
> experiments provve otherwise.
>
> James
I recommend you read the following:
https://en.wikipedia.org/wiki/Confusion_of_the_inverse
you don't explain which score you use for filtering (do you take one of the
scores, their sum, their product, or something else?), but I expect you
(mostly) keep the phrase pairs with a high p(e|f), which is the best thing
to do when you don't have a language model.
------------------------------
Message: 2
Date: Wed, 17 Jun 2015 16:24:02 +0000
From: "Read, James C" <jcread@essex.ac.uk>
Subject: Re: [Moses-support] Major bug found in Moses
To: Kenneth Heafield <moses@kheafield.com>, "moses-support@mit.edu"
<moses-support@mit.edu>
Cc: "Arnold, Doug" <doug@essex.ac.uk>
Message-ID:
<DB3PR06MB0713F22EE2A706E1D5D5392685A60@DB3PR06MB0713.eurprd06.prod.outlook.com>
Content-Type: text/plain; charset="iso-8859-1"
Which features would you like me to tune? The whole purpose of the exercise was to eliminate all variables except the TM and to keep constant those that could not be eliminated so that I could see which types of phrase pairs contribute most to increases in BLEU score in a TM only setup.
Now you are saying I have to tune but tuning won't work without a LM. So how do you expect a researcher to be able to understand how well the TM component of the system is working if you are going to insist that I must include a LM for tuning to work.
Clearly the system is broken. It is designed to work well with a LM and poorly without. When clearly good results can be obtained with a functional TM and well chosen phrase pairs.
James
________________________________________
From: moses-support-bounces@mit.edu <moses-support-bounces@mit.edu> on behalf of Kenneth Heafield <moses@kheafield.com>
Sent: Wednesday, June 17, 2015 7:13 PM
To: moses-support@mit.edu
Subject: Re: [Moses-support] Major bug found in Moses
I'll bite.
The moses.ini files ship with bogus feature weights. One is required to
tune the system to discover good weights for their system. You did not
tune. The results of an untuned system are meaningless.
So for example if the feature weights are all zeros, then the scores are
all zero. The system will arbitrarily pick some awful translation from
a large space of translations.
The filter looks at one feature p(target | source). So now you've
constrained the awful untuned model to a slightly better region of the
search space.
In other words, all you've done is a poor approximation to manually
setting the weight to 1.0 on p(target | source) and the rest to 0.
The problem isn't that you are running without a language model (though
we generally do not care what happens without one). The problem is that
you did not tune the feature weights.
Moreover, as Marcin is pointing out, I wouldn't necessarily expect
tuning to work without an LM.
On 06/17/15 11:56, Read, James C wrote:
> Actually the approximation I expect to be:
>
> p(e|f)=p(f|e)
>
> Why would you expect this to give poor results if the TM is well trained? Surely the results of my filtering experiments provve otherwise.
>
> James
>
> ________________________________________
> From: moses-support-bounces@mit.edu <moses-support-bounces@mit.edu> on behalf of Rico Sennrich <rico.sennrich@gmx.ch>
> Sent: Wednesday, June 17, 2015 5:32 PM
> To: moses-support@mit.edu
> Subject: Re: [Moses-support] Major bug found in Moses
>
> Read, James C <jcread@...> writes:
>
>> I have been unable to find a logical explanation for this behaviour other
> than to conclude that there must be some kind of bug in Moses which causes a
> TM only run of Moses to perform poorly in finding the most likely
> translations according to the TM when
>> there are less likely phrase pairs included in the race.
> I may have overlooked something, but you seem to have removed the language
> model from your config, and used default weights. your default model will
> thus (roughly) implement the following model:
>
> p(e|f) = p(e|f)*p(f|e)
>
> which is obviously wrong, and will give you poor results. This is not a bug
> in the code, but a poor choice of models and weights. Standard steps in SMT
> (like tuning the model weights on a development set, and including a
> language model) will give you the desired results.
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
------------------------------
Message: 3
Date: Wed, 17 Jun 2015 16:25:22 +0000
From: "Read, James C" <jcread@essex.ac.uk>
Subject: Re: [Moses-support] Major bug found in Moses
To: Rico Sennrich <rico.sennrich@gmx.ch>, "moses-support@mit.edu"
<moses-support@mit.edu>
Cc: "Arnold, Doug" <doug@essex.ac.uk>
Message-ID:
<DB3PR06MB071390E70DBCA4364AE74F2B85A60@DB3PR06MB0713.eurprd06.prod.outlook.com>
Content-Type: text/plain; charset="iso-8859-1"
I already answered this question in another post. Apologies for double posting. Here is the code I used for filtering. I filtered based on the fourth score only.
#!/usr/bin/perl -w
#
# Program filters phrase table to leave only phrase pairs
# with probability above a threshold
#
use strict;
use warnings;
use Getopt::Long;
my $phrase;
my $min;
my $phrase_table;
my $filtered_table;
GetOptions( 'min=f' => \$min,
'out=s' => \$filtered_table,
'in=s' => \$phrase_table);
die "ERROR: must give threshold and phrase table input file and output file\n" unless ($min && $phrase_table && $filtered_table);
die "ERROR: file $phrase_table does not exist\n" unless (-e $phrase_table);
open (PHRASETABLE, "<$phrase_table") or die "FATAL: Could not open phrase table $phrase_table\n";;
open (FILTEREDTABLE, ">$filtered_table") or die "FATAL: Could not open phrase table $filtered_table\n";;
while (my $line = <PHRASETABLE>)
{
chomp $line;
my @columns = split ('\|\|\|', $line);
# check that file is a well formatted phrase table
if (scalar @columns < 4)
{
die "ERROR: input file is not a well formatted phrase table. A phrase table must have at least four colums each column separated by |||\n";
}
# get the probability and check it is less than the threshold
my @scores = split /\s+/, $columns[2];
if ($scores[3] > $min)
{
print FILTEREDTABLE $line."\n";;
}
}
________________________________________
From: moses-support-bounces@mit.edu <moses-support-bounces@mit.edu> on behalf of Rico Sennrich <rico.sennrich@gmx.ch>
Sent: Wednesday, June 17, 2015 7:17 PM
To: moses-support@mit.edu
Subject: Re: [Moses-support] Major bug found in Moses
Read, James C <jcread@...> writes:
>
> Actually the approximation I expect to be:
>
> p(e|f)=p(f|e)
>
> Why would you expect this to give poor results if the TM is well trained?
Surely the results of my filtering
> experiments provve otherwise.
>
> James
I recommend you read the following:
https://en.wikipedia.org/wiki/Confusion_of_the_inverse
you don't explain which score you use for filtering (do you take one of the
scores, their sum, their product, or something else?), but I expect you
(mostly) keep the phrase pairs with a high p(e|f), which is the best thing
to do when you don't have a language model.
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
------------------------------
Message: 4
Date: Wed, 17 Jun 2015 20:35:51 +0400
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Major bug found in Moses
To: Kenneth Heafield <moses@kheafield.com>, moses-support@mit.edu
Message-ID: <5581A1E7.70600@gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
On 17/06/2015 20:13, Kenneth Heafield wrote:
> I'll bite.
>
> The moses.ini files ship with bogus feature weights. One is required to
> tune the system to discover good weights for their system. You did not
> tune. The results of an untuned system are meaningless.
>
> So for example if the feature weights are all zeros, then the scores are
> all zero. The system will arbitrarily pick some awful translation from
> a large space of translations.
>
> The filter looks at one feature p(target | source). So now you've
> constrained the awful untuned model to a slightly better region of the
> search space.
>
> In other words, all you've done is a poor approximation to manually
> setting the weight to 1.0 on p(target | source) and the rest to 0.
>
> The problem isn't that you are running without a language model (though
> we generally do not care what happens without one). The problem is that
> you did not tune the feature weights.
>
> Moreover, as Marcin is pointing out, I wouldn't necessarily expect
> tuning to work without an LM.
Tuning does work without a LM. The results aren't half bad. fr-en
europarl (pb):
with LM: 22.84
retuned without LM: 18.33
>
> On 06/17/15 11:56, Read, James C wrote:
>> Actually the approximation I expect to be:
>>
>> p(e|f)=p(f|e)
>>
>> Why would you expect this to give poor results if the TM is well trained? Surely the results of my filtering experiments provve otherwise.
>>
>> James
>>
>> ________________________________________
>> From: moses-support-bounces@mit.edu <moses-support-bounces@mit.edu> on behalf of Rico Sennrich <rico.sennrich@gmx.ch>
>> Sent: Wednesday, June 17, 2015 5:32 PM
>> To: moses-support@mit.edu
>> Subject: Re: [Moses-support] Major bug found in Moses
>>
>> Read, James C <jcread@...> writes:
>>
>>> I have been unable to find a logical explanation for this behaviour other
>> than to conclude that there must be some kind of bug in Moses which causes a
>> TM only run of Moses to perform poorly in finding the most likely
>> translations according to the TM when
>>> there are less likely phrase pairs included in the race.
>> I may have overlooked something, but you seem to have removed the language
>> model from your config, and used default weights. your default model will
>> thus (roughly) implement the following model:
>>
>> p(e|f) = p(e|f)*p(f|e)
>>
>> which is obviously wrong, and will give you poor results. This is not a bug
>> in the code, but a poor choice of models and weights. Standard steps in SMT
>> (like tuning the model weights on a development set, and including a
>> language model) will give you the desired results.
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
--
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu
------------------------------
Message: 5
Date: Wed, 17 Jun 2015 16:45:29 +0000
From: "Read, James C" <jcread@essex.ac.uk>
Subject: Re: [Moses-support] Major bug found in Moses
To: Hieu Hoang <hieuhoang@gmail.com>, Kenneth Heafield
<moses@kheafield.com>, "moses-support@mit.edu"
<moses-support@mit.edu>
Cc: "Arnold, Doug" <doug@essex.ac.uk>
Message-ID:
<DB3PR06MB071386F42D738E66B3663AB185A60@DB3PR06MB0713.eurprd06.prod.outlook.com>
Content-Type: text/plain; charset="iso-8859-1"
Doesn't look like the LM is contributing all that much then does it?
James
________________________________________
From: moses-support-bounces@mit.edu <moses-support-bounces@mit.edu> on behalf of Hieu Hoang <hieuhoang@gmail.com>
Sent: Wednesday, June 17, 2015 7:35 PM
To: Kenneth Heafield; moses-support@mit.edu
Subject: Re: [Moses-support] Major bug found in Moses
On 17/06/2015 20:13, Kenneth Heafield wrote:
> I'll bite.
>
> The moses.ini files ship with bogus feature weights. One is required to
> tune the system to discover good weights for their system. You did not
> tune. The results of an untuned system are meaningless.
>
> So for example if the feature weights are all zeros, then the scores are
> all zero. The system will arbitrarily pick some awful translation from
> a large space of translations.
>
> The filter looks at one feature p(target | source). So now you've
> constrained the awful untuned model to a slightly better region of the
> search space.
>
> In other words, all you've done is a poor approximation to manually
> setting the weight to 1.0 on p(target | source) and the rest to 0.
>
> The problem isn't that you are running without a language model (though
> we generally do not care what happens without one). The problem is that
> you did not tune the feature weights.
>
> Moreover, as Marcin is pointing out, I wouldn't necessarily expect
> tuning to work without an LM.
Tuning does work without a LM. The results aren't half bad. fr-en
europarl (pb):
with LM: 22.84
retuned without LM: 18.33
>
> On 06/17/15 11:56, Read, James C wrote:
>> Actually the approximation I expect to be:
>>
>> p(e|f)=p(f|e)
>>
>> Why would you expect this to give poor results if the TM is well trained? Surely the results of my filtering experiments provve otherwise.
>>
>> James
>>
>> ________________________________________
>> From: moses-support-bounces@mit.edu <moses-support-bounces@mit.edu> on behalf of Rico Sennrich <rico.sennrich@gmx.ch>
>> Sent: Wednesday, June 17, 2015 5:32 PM
>> To: moses-support@mit.edu
>> Subject: Re: [Moses-support] Major bug found in Moses
>>
>> Read, James C <jcread@...> writes:
>>
>>> I have been unable to find a logical explanation for this behaviour other
>> than to conclude that there must be some kind of bug in Moses which causes a
>> TM only run of Moses to perform poorly in finding the most likely
>> translations according to the TM when
>>> there are less likely phrase pairs included in the race.
>> I may have overlooked something, but you seem to have removed the language
>> model from your config, and used default weights. your default model will
>> thus (roughly) implement the following model:
>>
>> p(e|f) = p(e|f)*p(f|e)
>>
>> which is obviously wrong, and will give you poor results. This is not a bug
>> in the code, but a poor choice of models and weights. Standard steps in SMT
>> (like tuning the model weights on a development set, and including a
>> language model) will give you the desired results.
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
--
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 104, Issue 34
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 104, Issue 34"
Post a Comment