Moses-support Digest, Vol 107, Issue 13

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: [Moses-developers] Generation models with Mmsapt (Hieu Hoang)
2. Re: Generating Segment Level BLEU, NIST and METEOR scores
(Graham Neubig)
3. Re: Generating Segment Level BLEU, NIST and METEOR scores
(Cristina)


----------------------------------------------------------------------

Message: 1
Date: Fri, 4 Sep 2015 12:10:21 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] [Moses-developers] Generation models with
Mmsapt
To: Michael Denkowski <michael.j.denkowski@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbinmXfi8zdL6XVn6_OHocaiQxzDY=2seCuMJ0ozk3XXpw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

ok. I'm still getting segfault in Lex Reordering during loading. If you can
provide the ro text file so I can binarize it myself, I can debug it. But
not a priority, I can park the issue for another time

Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

On 4 September 2015 at 03:49, Michael Denkowski <
michael.j.denkowski@gmail.com> wrote:

> Hi Hieu,
>
> Yes, I have everything working together with the caveat about order in the
> moses.ini file (https://github.com/moses-smt/mosesdecoder/pull/124). The
> mmsapt files might be dependent on Boost version so I also included the
> aligned bitext I used to build the model and rebuilt the tarball:
> https://drive.google.com/file/d/0B6trVKD0-obBRHFjMGxRZTJvV1U/view?usp=sharing.
> I made it a pull request instead of just merging it into master so you guys
> could look over the changes since technically the default behavior could
> change if the moses.ini file lists phrase tables before other features.
>
> Best,
> Michael
>
> On Thu, Sep 3, 2015 at 8:44 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
>> i saw your checkins, is it working for you now?
>>
>> Your test data doesn't seem to run for me, the lexical reordering file
>> seems to be corrupt.
>>
>> (if you wanna share test data, can you do it via dropbox/google drive
>> rather than the Moses github)
>>
>>
>> On 03/09/2015 05:13, Michael Denkowski wrote:
>>
>> Sounds good. I added a small test model to my branch:
>> <https://github.com/moses-smt/mosesdecoder/raw/mjdenkowski/mmsapt-factor-test.tar.gz>
>> https://github.com/moses-smt/mosesdecoder/raw/mjdenkowski/mmsapt-factor-test.tar.gz.
>> This translates a sample of fr-en news with a Mmsapt, surface LM, and
>> 400-class LM.
>>
>> --Michael
>>
>> On Wed, Sep 2, 2015 at 2:56 AM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>>
>>> It should work. The function
>>> EvaluateInIsolation()
>>> in the LM is for optimisation reason. eg. if the target phrase is 'a b c
>>> d' and the LM is a trigram, the trigrams 'a b c' and 'b c d' can be
>>> precalculated in EvaluateInIsolation().
>>>
>>> Implementing a pt for factors requires setting up some variables, which
>>> may not have happen yet in mmsapt. if you can send me a small example
>>> model, i'll see what i can do
>>>
>>>
>>> On 01/09/2015 02:11, Ulrich Germann wrote:
>>>
>>> Hi Michael,
>>>
>>> I have no experience with factored models, so I'm speculating here to
>>> some degree. The reason the phrase table calls EvaluateInIsolation is
>>> because all "isolated" phrase scores are considered when pruning. In my
>>> opinion pruning should not happen within the phrase tables (for exactly the
>>> reason that it does not allow feature functions to be agnostic about other
>>> feature functions) but by whatever object calls all the phrase tables and
>>> does the generation. However, for software legacy reasons, that's the way
>>> it is right now, and I'm not likely to address this issue any time soon
>>> myself. The most reasonable fix for this in my opinion is to move pruning
>>> where it belongs --- post all the factor generation stuff.
>>>
>>> Hieu is probably still the person with the best understanding of how
>>> factored phrase table entry generation works, so maybe he can chime in on
>>> this ...
>>>
>>> Cheers - Uli
>>>
>>>
>>> On Mon, Aug 31, 2015 at 11:29 PM, Michael Denkowski <
>>> <michael.j.denkowski@gmail.com>michael.j.denkowski@gmail.com> wrote:
>>>
>>>> Hi Ulrich,
>>>>
>>>> I was looking into using a class-based LM with your dynamic phrase
>>>> table via generation models. I translate factor 0 to 0 with the Mmsapt,
>>>> then generate target factor 1 (word class) with a GM. The class-based LM
>>>> operates on factor 1.
>>>>
>>>> I'm hitting a segfault on what appears to be an order-of-operations
>>>> issue with the PT and LM. In mmsapt.cpp:578, Mmsapt::mkTPhrase makes a
>>>> call to tp->EvaluateInIsolation. This calls all of the models, including
>>>> the LMs. The class LM tries to score factor 1, which doesn't exist yet
>>>> (since generation happens after translation), and it dies. By nature,
>>>> other phrase tables don't have this issue since they can just pull up
>>>> pre-computed scores.
>>>>
>>>> Is scoring with all of the models here a strategic choice to get better
>>>> performance or would it be sufficient to just score with the PT features?
>>>> Thanks!
>>>>
>>>> --Michael
>>>>
>>>
>>>
>>>
>>> --
>>> Ulrich Germann
>>> Senior Researcher
>>> School of Informatics
>>> University of Edinburgh
>>>
>>>
>>> _______________________________________________
>>> Moses-developers mailing listMoses-developers@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-developers
>>>
>>>
>>> --
>>> Hieu Hoang
>>> Researcher
>>> New York University, Abu Dhabihttp://www.hoang.co.uk/hieu
>>>
>>>
>>
>> --
>> Hieu Hoang
>> Researcher
>> New York University, Abu Dhabihttp://www.hoang.co.uk/hieu
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150904/25166c44/attachment-0001.html

------------------------------

Message: 2
Date: Fri, 4 Sep 2015 20:26:38 +0900
From: Graham Neubig <neubig@is.naist.jp>
Subject: Re: [Moses-support] Generating Segment Level BLEU, NIST and
METEOR scores
To: liling tan <alvations@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CADkjOCMMf3KfQ7bXWceZoV0iLEECN1=fpg4gogFQCdQkaP3vyw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Liling, All,

mt-eval allows for sentence-level calculation of BLEU, NIST, RIBES, and WER:
https://github.com/odashi/mteval

It doesn't handle METEOR (which is likely too complicated to be properly
implemented independently of the main branch), but for the others it's a
pretty convenient standalone library.

Graham

On Thu, Sep 3, 2015 at 4:47 AM, liling tan <alvations@gmail.com> wrote:

> Dear Moses devs/users,
>
> Is there any script in moses or other MT libraries that can generate the
> segment level BLEU, NIST and METEOR scores for each sentence in the test
> set?
>
> Best Regards,
> Liling
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150904/43383d2b/attachment-0001.html

------------------------------

Message: 3
Date: Fri, 4 Sep 2015 13:33:15 +0200
From: Cristina <cristinae@cs.upc.edu>
Subject: Re: [Moses-support] Generating Segment Level BLEU, NIST and
METEOR scores
To: liling tan <alvations@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAL0MP8i_f2-kgo21rACqz5+yuns8vOm+Gx2dHEyy3xfcf4k-FQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Liling,

Asiya generates the segment scores for any metric it has as part of the
analysis:

http://nlp.lsi.upc.edu/asiya/

Best,

Cristina


2015-09-02 21:47 GMT+02:00 liling tan <alvations@gmail.com>:

> Dear Moses devs/users,
>
> Is there any script in moses or other MT libraries that can generate the
> segment level BLEU, NIST and METEOR scores for each sentence in the test
> set?
>
> Best Regards,
> Liling
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150904/986dbf9a/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 107, Issue 13
**********************************************

Tweet