Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Reference about tuning the BLEU coefficients (Mikel Forcada)
2. Re: Reference about tuning the BLEU coefficients (Ondrej Bojar)
3. Re: Reference about tuning the BLEU coefficients
(Mikel L. Forcada)
4. Re: Learning the moses.ini v2 format (Hieu Hoang)
----------------------------------------------------------------------
Message: 1
Date: Tue, 3 Feb 2015 07:09:20 +0100
From: Mikel Forcada <mlf@dlsi.ua.es>
Subject: Re: [Moses-support] Reference about tuning the BLEU
coefficients
To: moses-support@mit.edu
Message-ID: <20150203070920.5149cb31@OpenMaTrEx>
Content-Type: text/plain; charset=US-ASCII
Thanks a million, Philipp!
About these tunable metrics, has anything been published? do you have
any references?
Cheers
Mikel
>
> Hi,
>
> while the original BLEU paper mentions co-efficients, there are not
> used (i.e., all are set to 1/4).
>
> There have efforts to create tunable metrics, but these are not
> widely used.
>
> -phi
>
> On Mon, Feb 2, 2015 at 4:34 AM, Mikel L. Forcada <mlf@dlsi.ua.es>
> wrote:
> > Dear list,
> >
> > the correlation of BLEU with manual measurements of quality has
> > extensively been studied (Callison-Burch et al., WMT "Findings"
> > paper). But, would anyone know of any paper where people have
> > actually tuned the BLEU coefficients to approximate some kind of
> > manual measurement of quality? This is hard to search for as most
> > papers talk about tuning the coefficients of something else to BLEU.
> >
> > Thanks a million!
> >
> > All the best,
> >
> > Mikel
> >
> > --
> >
------------------------------
Message: 2
Date: Tue, 03 Feb 2015 08:56:55 +0100
From: Ondrej Bojar <bojar@ufal.mff.cuni.cz>
Subject: Re: [Moses-support] Reference about tuning the BLEU
coefficients
To: Mikel Forcada <mlf@dlsi.ua.es>, moses-support@mit.edu
Message-ID: <e2ac2192-5220-4eb5-affe-2e806f8eca3c@email.android.com>
Content-Type: text/plain; charset=UTF-8
Dear Mikel,
I'd check Daniel Cer's thesis, I'm not sure if he did anything with BLEU coefficients but he definitely convinced himself that there was no better tuning metric than BLEU.
...but we're running the tuning task at WMT this year again, so something new will hopefully show up.
Cheers,
Ondrej.
On February 3, 2015 7:09:20 AM CET, Mikel Forcada <mlf@dlsi.ua.es> wrote:
>Thanks a million, Philipp!
>About these tunable metrics, has anything been published? do you have
>any references?
>Cheers
>Mikel
>
>
>>
>> Hi,
>>
>> while the original BLEU paper mentions co-efficients, there are not
>> used (i.e., all are set to 1/4).
>>
>> There have efforts to create tunable metrics, but these are not
>> widely used.
>>
>> -phi
>>
>> On Mon, Feb 2, 2015 at 4:34 AM, Mikel L. Forcada <mlf@dlsi.ua.es>
>> wrote:
>> > Dear list,
>> >
>> > the correlation of BLEU with manual measurements of quality has
>> > extensively been studied (Callison-Burch et al., WMT "Findings"
>> > paper). But, would anyone know of any paper where people have
>> > actually tuned the BLEU coefficients to approximate some kind of
>> > manual measurement of quality? This is hard to search for as most
>> > papers talk about tuning the coefficients of something else to
>BLEU.
>> >
>> > Thanks a million!
>> >
>> > All the best,
>> >
>> > Mikel
>> >
>> > --
>> >
>_______________________________________________
>Moses-support mailing list
>Moses-support@mit.edu
>http://mailman.mit.edu/mailman/listinfo/moses-support
--
Ondrej Bojar (mailto:obo@cuni.cz / bojar@ufal.mff.cuni.cz)
http://www.cuni.cz/~obo
------------------------------
Message: 3
Date: Tue, 03 Feb 2015 09:44:20 +0100
From: "Mikel L. Forcada" <mlf@dlsi.ua.es>
Subject: Re: [Moses-support] Reference about tuning the BLEU
coefficients
To: Ondrej Bojar <bojar@ufal.mff.cuni.cz>, moses-support@mit.edu
Message-ID: <54D08A64.7040401@dlsi.ua.es>
Content-Type: text/plain; charset=utf-8; format=flowed
Thanks a million Ondrej!
> I'd check Daniel Cer's thesis,
Will do!
> I'm not sure if he did anything with BLEU coefficients but he definitely convinced himself that there was no better tuning metric than BLEU.
>
> ...but we're running the tuning task at WMT this year again, so something new will hopefully show up.
Cool, I assume you refer to this:
http://www.statmt.org/wmt15/tuning-task/
All the best
Mikel
--
Mikel L. Forcada (http://www.dlsi.ua.es/~mlf/)
Departament de Llenguatges i Sistemes Inform?tics
Universitat d'Alacant
E-03071 Alacant, Spain
Phone: +34 96 590 9776
Fax: +34 96 590 9326
------------------------------
Message: 4
Date: Tue, 03 Feb 2015 11:02:21 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Learning the moses.ini v2 format
To: Tom Hoar <tahoar@precisiontranslationtools.com>,
moses-support@mit.edu
Message-ID: <54D0AABD.2030006@gmail.com>
Content-Type: text/plain; charset="windows-1252"
cheer tom
i've add your script to convert v2->v1.
https://github.com/moses-smt/mosesdecoder/commit/78f79632b9a87c72f4ad11005359fcdc57d1c0bc
There's already a script to convert it the other way, so we're keeping
that instead
On 02/02/15 15:20, Tom Hoar wrote:
> Thanks, Hieu.
>
> Here are the two scripts. Add them to the contrib (or other) folder as
> you see fit. They have a -h switch. The moses2-to-ini.py script copies
> everything from moses.ini v2 format into a traditional INI file
> format. The ini-2-moses2.py script restores the moses.ini v2 format.
>
> They change the line order in the [feature] and [weight] sections. The
> overall order of section blocks can change. They preserve the order of
> lines/values inside the old v1 moses.ini format sections, like
> [input-factor], [mapping], etc. Please report any errors.
>
> There two limits. First, the FF name= attribute is not optional
> because relying on sequential orders can be risky. Users should
> declare the name to match the weight value key. Second, they do not
> support use cases where non-tuneable FF does not include weights. If
> you place dummy weights, like you did with UnknownWordPenalty,
> everything is ok.
>
> There's one additional feature. Users can use the --escape-prefix
> command-line option to escape a prefix path in FF "path=" attribute
> values. This will be helpful in moving models. I noticed that the
> clone_moses_model.pl script has not been updated to support the new
> moses.ini file format. Maybe someone would like to use this as a
> starting point.
>
> I find it easier/faster to manipulate values a traditional INI file
> structure. Users can also import classes in the scripts into other
> modules. They have a pretty simple API. I thought others might find
> these useful.
>
> Tom
>
>
> On 02/02/2015 08:56 PM, Hieu Hoang wrote:
>> Hi Tom
>>
>> On 02/02/15 01:33, Tom Hoar wrote:
>>> Much of the v2 moses.ini looks self-explanatory, but I'd like to
>>> confirm my understanding.
>>>
>>> The website (http://www.statmt.org/moses/?n=Moses.FeatureFunctions)
>>> defines three feature/functions without arguments. In the moses.ini
>>> files made by train-model.perl's step 9, there also appears to be a
>>> 4th that requires no argument. Can someone confirm this is the case?
>>> Are there others that could appear without arguments?
>> Yes, PhrasePenalty is a standard feature function (FF), it doesn't
>> need any arguments. It used to be the constant 2.718 in the last
>> score in the phrase-table. But i thought that was silly so move it to
>> it's own feature function.
>>>
>>> [feature]
>>> UnknownWordPenalty
>>> WordPenalty
>>> Distortion
>>> PhrasePenalty * - not listed on the website (are there more)
>>>
>>> Feature/functions in the [feature] section and items in the [weight]
>>> sections appear to be linked. The feature/functions without
>>> arguments have corresponding entries linked by the same option name
>>> with an appended zero in the [weight] section. Since these
>>> feature/functions have arguments, is it safe to say that they can
>>> appear only once in both the [feature] and [weight] sections?
>> You can have multiple feature function of the same type. The most
>> obvious 1 would be having multiple LM, eg.
>> [feature]
>> KENLM path=file1.lm
>> KENLM path=file2.lm
>> Each instance of an FF must have a unique name, if you don't give
>> inamet a , the decoder will name it for you, KENLM0, KENLM1, ....
>> Each instance must have the corresponding weights (unless it's
>> non-tuneable)
>> [weight]
>> KENLM0= 0.4
>> KENLM1= 0.6
>>>
>>> [weight]
>>> UnknownWordPenalty0= 1
>>> WordPenalty0= -1
>>> Distortion0= 0.3
>>> PhrasePenalty0= 0.2
>>>
>>> The feature/functions arguments have corresponding entries liked by
>>> the "name=" argument as the option name in the [weight] section. Are
>>> there cases where there will be entries in the [feature] section
>>> without corresponding entries in the [weight] section or vice-versa?
>> Yes, if the FF is not tuneable, you don't need to give it weight(s).
>> The hardcoded weights will be used. UnknownWordPenalty is a
>> non-tuneable FF and its hardcoded weight is 1, so the line
>> UnknownWordPenalty0= 1
>> isn't strictly necessary.
>>
>> Whether a FF is non-tuneable or not by default is determined in the
>> code. You can also set the non-tuneable property, eg
>> [feature]
>> UnknownWordPenalty tuneable=true
>>
>>>
>>> [feature]
>>> PhraseDictionaryMemory name=*TranslationModel0* num-features=4 ...
>>> KENLM name=*LM0* factor=0 ...
>>>
>>> [weight]
>>> *TranslationModel0*= 0.2 0.2 0.2 0.2
>>> *LM0*= 0.5
>>>
>>> The sections other than [feature] and [weight], such as
>>> [input-factors] and [mapping], appear to preserve the v1 moses.ini
>>> format. Is this true?
>> yep
>>>
>>> The order of lines in the [feature] and [weight] sections is
>>> irrelevant (as many examples have them in different orders). Also,
>>> the order of the arguments on a feature/function line is irrelevant
>>> (examples show them in different orders).
>> yep. However, the relative order of the phrase-tables is relevant.
>> The [mapping] section refers to the index of the phrase-table, eg
>> [mapping]
>> 0 T 0
>> 1 T 1
>> so if you swap the order of the phrase-tables in the [feature]
>> section, they will refer to different phrase-tables
>>
>>>
>>> Finally, is there a connection between the [input-factors] section's
>>> value and the input-factor argument value for PhraseDictionaryMemory
>>> and LexicalReordering feature/functions? Or, are the similar names
>>> and corresponding values only coincidental?
>> Your input sentence can contain multiple factors, eg.
>> surface|POS|lemma, in which case [input-factor] should be
>> [input-factor]
>> 0
>> 1
>> 2
>> However, the phase-table may choose to only use the surface form, in
>> which case
>> [feature]
>> PhraseDictionaryMemory input-factors=0
>>>
>>> My intention is to build two scripts and contribute these scripts to
>>> the Moses project. One will convert the v2 moses.ini file to a
>>> standard form (not associated with the command line syntax) so
>>> people can easily edit the values. The other will convert the
>>> interim form back to the native v2 moses.ini format.
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150203/19f1b0d3/attachment.htm
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 100, Issue 8
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 100, Issue 8"
Post a Comment