Moses-support Digest, Vol 113, Issue 19

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Replacing OOVs with a single OOV token (Lane Schwartz)
2. Re: Replacing OOVs with a single OOV token
(Marcin Junczys-Dowmunt)
3. Re: Replacing OOVs with a single OOV token (Lane Schwartz)
4. Re: Scripts for n-best-list rescoring (Michael Denkowski)


----------------------------------------------------------------------

Message: 1
Date: Tue, 8 Mar 2016 08:08:14 -0600
From: Lane Schwartz <dowobeha@gmail.com>
Subject: [Moses-support] Replacing OOVs with a single OOV token
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZ=E2m1txf38YT-yoEERp2nsy6gM-kiNcfbHczcaHrv7YA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Is there an existing mechanism by which any OOV word would be replaced in
the output with a single unique OOV token, instead of either being passed
through or dropped?

Thanks,
Lane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160308/e1b01713/attachment-0001.html

------------------------------

Message: 2
Date: Tue, 8 Mar 2016 14:11:25 +0000
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Replacing OOVs with a single OOV token
To: moses-support@mit.edu
Message-ID: <56DEDD8D.4030207@amu.edu.pl>
Content-Type: text/plain; charset="windows-1252"

-mark-unknown marks the unknown word in the output, from there you can
easily post-process.

Best,
Marcin

W dniu 08.03.2016 o 14:08, Lane Schwartz pisze:
> Is there an existing mechanism by which any OOV word would be replaced
> in the output with a single unique OOV token, instead of either being
> passed through or dropped?
>
> Thanks,
> Lane
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160308/b5a904c3/attachment-0001.html

------------------------------

Message: 3
Date: Tue, 8 Mar 2016 08:50:53 -0600
From: Lane Schwartz <dowobeha@gmail.com>
Subject: Re: [Moses-support] Replacing OOVs with a single OOV token
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZms25Qk2zP9AiE_fmfrwy3=soYvVjpuHJkXzpggERVtXw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Thanks, Marcin.

That got me what I needed.

On Tue, Mar 8, 2016 at 8:11 AM, Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
wrote:

> -mark-unknown marks the unknown word in the output, from there you can
> easily post-process.
>
> Best,
> Marcin
>
> W dniu 08.03.2016 o 14:08, Lane Schwartz pisze:
>
> Is there an existing mechanism by which any OOV word would be replaced in
> the output with a single unique OOV token, instead of either being passed
> through or dropped?
>
> Thanks,
> Lane
>
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
When a place gets crowded enough to require ID's, social collapse is not
far away. It is time to go elsewhere. The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160308/728fd49d/attachment-0001.html

------------------------------

Message: 4
Date: Tue, 8 Mar 2016 11:02:19 -0500
From: Michael Denkowski <michael.j.denkowski@gmail.com>
Subject: Re: [Moses-support] Scripts for n-best-list rescoring
To: Philipp Koehn <phi@jhu.edu>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CA+-GegKD8qL_ULKB06VRX5GMGt0Q680k3v1XkYOfBQtB95MEjQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I recently checked in the N-best re-scorer I wrote since I couldn't find a
good existing one either:
https://github.com/moses-smt/mosesdecoder/tree/master/scripts/nbest-rescore.
Given an N-best list and references, it uses K-best MIRA to learn
re-ranking weights relatively quickly. It's agnostic to original decoder
and optimizer as long as entries are in the right format. The readme
includes more detailed instructions.

Best,
Michael

On Tue, Mar 8, 2016 at 8:25 AM, Philipp Koehn <phi@jhu.edu> wrote:

> Hi,
>
> there is this mysterious check-in:
>
> Commit: c6314d927d8b7b638eca387f31ccfab7facb6624
>
> https://github.com/moses-smt/mosesdecoder/commit/c6314d927d8b7b638eca387f31ccfab7facb6624
> Author: Michael Denkowski <mdenkows@amazon.com>
> Date: 2016-02-23 (Tue, 23 Feb 2016)
>
> Changed paths:
> A scripts/nbest-rescore/README.md
> A scripts/nbest-rescore/rescore.py
> A scripts/nbest-rescore/topbest.py
> A scripts/nbest-rescore/train.py
>
> -phi
>
> On Tue, Mar 8, 2016 at 8:18 AM, Lane Schwartz <dowobeha@gmail.com> wrote:
>
>> I don't think there is. At my previous lab, I believe we had to build our
>> own in-house script. It would be nice to have one in moses.
>>
>> On Sat, Oct 31, 2015 at 12:56 PM, Marcin Junczys-Dowmunt <
>> junczys@amu.edu.pl> wrote:
>>
>>> Hi,
>>> does moses include scripts for n-best-list rescoring/resorting after a
>>> new feature has been added to the list?
>>>
>>> I guess, this can probably be achieved by running a single parameter
>>> tuning step on the extended n-best-list, but then I still need to fiddle
>>> around with calculating model scores with the new weights etc. Is there
>>> anything public and working with the moses n-best-list format?
>>>
>>> Cheers,
>>> Marcin
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>
>>
>>
>> --
>> When a place gets crowded enough to require ID's, social collapse is not
>> far away. It is time to go elsewhere. The best thing about space travel
>> is that it made it possible to go elsewhere.
>> -- R.A. Heinlein, "Time Enough For Love"
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160308/df1b5f20/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 113, Issue 19
**********************************************

Tweet