Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Skip OOV when computing Language Model score (Philipp Koehn)
2. Re: Skip OOV when computing Language Model score
(Kenneth Heafield)
3. Special Issue of Machine Translation Journal on Neural
Network Approaches to Machine Translation (Qun Liu)
4. Same tuning set for different trainings (???)
5. Fwd: Moses-support post from jasneet.sabharwal@sfu.ca
requires approval (Hieu Hoang)
----------------------------------------------------------------------
Message: 1
Date: Thu, 14 Jan 2016 18:07:17 -0500
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] Skip OOV when computing Language Model
score
To: LUONG NGOC Quang <quangngocluong@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAAFADDCa94HfZU0AciseCK55WzqvSXsaLCMVUMv1R=fgdOjj6g@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
You may get the behavior you want by adding
"oov-feature=1"
to your LM specification line in moses.ini
and also add a second weight with value "0" to the corresponding LM weight
setting.
This will then only use the scores
p(the|<s>)
p(house|<s>,the,<unk>) ---> backoff to p(house)
p(in|<s>,the,<unk>,house,<unk>) ---> backoff to p(in)
-phi
On Thu, Jan 14, 2016 at 8:25 AM, LUONG NGOC Quang <quangngocluong@gmail.com>
wrote:
> Dear All,
>
> I am currently using a SRILM Language Model (LM) in my Moses decoder. Does
> anyone know how can I ask the decoder, at the decoding time, skip all
> out-of-vocabulary words when computing the LM score (instead of doing
> back-off)?
>
> For instance, with the n-gram: "the <unk> house <unk> in", I would like
> the decoder to assign it the probability of the phrase: "the house in"
> (existing in the LM).
>
> Do I need more options/declarations in moses.ini file?
>
> Any help is very much appreciated,
>
> Best,
> Quang
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160114/4aee4d36/attachment-0001.html
------------------------------
Message: 2
Date: Thu, 14 Jan 2016 23:22:54 +0000
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] Skip OOV when computing Language Model
score
To: moses-support@mit.edu
Message-ID: <56982DCE.4070908@kheafield.com>
Content-Type: text/plain; charset=windows-1252
Hi,
I think oov-feature=1 just activates the OOV count feature while
leaving LM score unchanged. So it would still include p(<unk> | in).
One might try setting the OOV feature weight to -weight_LM *
weird_moses_internal_constant * log p(<unk>) in an attempt to cancel out
the log p(<unk>) terms. However that won't work either because:
1) It will still charge backoff penalties, b(the)b(house) in the example.
2) The context will be lost each time so it's p(house) not p(house | the).
If the <unk>s follow a pattern, such as appearing every other word, one
could insert them into the ARPA file though that would waste memory.
I don't think there's any way to accomplish exactly what OP asked for
without coding (though it wouldn't be that hard once one understands how
the LM infrastructure works).
Kenneth
On 01/14/2016 11:07 PM, Philipp Koehn wrote:
> Hi,
>
> You may get the behavior you want by adding
> "oov-feature=1"
> to your LM specification line in moses.ini
> and also add a second weight with value "0" to the corresponding LM
> weight setting.
>
> This will then only use the scores
> p(the|<s>)
> p(house|<s>,the,<unk>) ---> backoff to p(house)
> p(in|<s>,the,<unk>,house,<unk>) ---> backoff to p(in)
>
> -phi
>
> On Thu, Jan 14, 2016 at 8:25 AM, LUONG NGOC Quang
> <quangngocluong@gmail.com <mailto:quangngocluong@gmail.com>> wrote:
>
> Dear All,
>
> I am currently using a SRILM Language Model (LM) in my Moses
> decoder. Does anyone know how can I ask the decoder, at the decoding
> time, skip all out-of-vocabulary words when computing the LM score
> (instead of doing back-off)?
>
> For instance, with the n-gram: "the <unk> house <unk> in", I would
> like the decoder to assign it the probability of the phrase: "the
> house in" (existing in the LM).
>
> Do I need more options/declarations in moses.ini file?
>
> Any help is very much appreciated,
>
> Best,
> Quang
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
------------------------------
Message: 3
Date: Thu, 14 Jan 2016 23:41:43 +0000
From: Qun Liu <liuquncn@gmail.com>
Subject: [Moses-support] Special Issue of Machine Translation Journal
on Neural Network Approaches to Machine Translation
To: moses-support@mit.edu
Message-ID: <56983237.8050506@gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
?******** CFP: Machine Translation Journal ********
Special Issue on Neural Network Approaches to Machine Translation
http://www.springer.com/computer/artificial/journal/10590
Guest editors:
Xiaodong He (Microsoft Research, US)
Qun Liu (ADAPT Centre, Dublin City University, Ireland)
Hermann Ney (RWTH Aachen, Germany)
---------------------------------------------------------------------------
In recent years, Neural Network (NN) has been the driving force behind huge improvements in speech and image processing. This has led to high expectations for NN in NLP and MT. In recent top conferences, a significant proportion of papers in the MT domain have been related to NN. A large amount of recent publications have shown the effectiveness of NN in various aspects of MT.
Accordingly, we are pleased to put out a call for papers for a special issue on this topic, and welcome descriptions of innovative ways of addressing this challenging problem. Topics include but are not restricted to the following:
-- Monolingual Word/Phrase/Sentence Embedding for MT
-- Bilingual Word/Phrase/Sentence Embedding for MT
-- Neural Language Model for MT
-- Neural Translation Model for MT
-- NN for MT Evaluation
-- NN for MT Tuning
-- Neural MT / Encoder-Decoder / Sequence-to-Sequence MT
-- Attention-based Neural MT
-- Novel hybrid MT using NN
-- Applications of NN to 'hard' problems in MT
-- NN for morphologically rich language MT
-- NN for reordering in MT
-- NN for syntax-based MT
-- NN for semantic-based MT
-- NN for Context-aware MT
-- NN for language generation in MT
-- NN for MT output reranking
-- NN for MT system combination
-- NN for MT application
** Note that revised, extended versions of previously accepted papers at leading peer-reviewed conferences are likely to be eligible for inclusion under the MT journal's fast-tracking process. **
---------------------------------------------------------------------------
Submission guidelines:
- Authors should follow the "Instructions for Authors" available on
the MT Journal website:
http://www.springer.com/computer/artificial/journal/10590
- Submissions must be limited to 15 pages (including references)
- Papers should be submitted online directly on the MT journal's
submission website: http://www.editorialmanager.com/coat/default.asp,
indicating this special issue in 'article type'
---------------------------------------------------------------------------
Important dates:
- Paper submission: April 30th 2016.
- Notification to authors: June 30th 2016.
- Camera-ready*: August 31th 2016.
- Publication time*: October-December 2016
* tentative - depending on the number of review rounds required
------------------------------
Message: 4
Date: Fri, 15 Jan 2016 11:01:18 +0200
From: ??? <fountain1128@gmail.com>
Subject: [Moses-support] Same tuning set for different trainings
To: moses-support@mit.edu
Message-ID: <6D97A5B5-190D-4A24-A7A3-780EE87E421A@gmail.com>
Content-Type: text/plain; charset="gb2312"
Dear all,
I?m trying to train the translation systems by adding different data sets. Is it possible to use the same tuning and testing data for each training?
I tried and I got an error. I guess it?s because I used ?tuning. true.en? in the first training, so it can not be run again?
ERROR: Failed to run '/opt/mosesdecoder/scripts/training/filter-model-given-input.pl ./filtered /home/fangting/working2m/train/model/moses.ini /home/fangting/corpus/tuning.true.en'. at /opt/mosesdecoder/scripts/training/mert-moses.pl line 1719.
-Fangting
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160115/de833041/attachment-0001.html
------------------------------
Message: 5
Date: Fri, 15 Jan 2016 11:43:48 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: [Moses-support] Fwd: Moses-support post from
jasneet.sabharwal@sfu.ca requires approval
To: jasneet.sabharwal@sfu.ca, moses-support <moses-support@mit.edu>
Message-ID: <5698DB74.9080403@gmail.com>
Content-Type: text/plain; charset="windows-1252"
please subscribe to the Moses mailing list before posting to it. You can
subscribe here:
http://mailman.mit.edu/mailman/admin/moses-support
To answer you question - the target phrase has a method called
GetAlignTerm()
that contains the alignment for terminals. This comes from the
phrase-table, and ultimately from the word alignment.
-------- Forwarded Message --------
Subject: Moses-support post from jasneet.sabharwal@sfu.ca requires
approval
Date: Wed, 13 Jan 2016 23:36:50 -0500
From: moses-support-owner@mit.edu
To: moses-support-owner@mit.edu
As list administrator, your authorization is requested for the
following mailing list posting:
List: Moses-support@mit.edu
From: jasneet.sabharwal@sfu.ca
Subject: Getting alignments for current hypothesis in phrase based model
Reason: Post by non-member to a members-only list
At your convenience, visit:
http://mailman.mit.edu/mailman/admindb/moses-support
to approve or deny the request.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160115/0721da96/attachment.html
-------------- next part --------------
An embedded message was scrubbed...
From: Jasneet Sabharwal <jasneet.sabharwal@sfu.ca>
Subject: Getting alignments for current hypothesis in phrase based model
Date: Wed, 13 Jan 2016 20:36:46 -0800 (PST)
Size: 4497
Url: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160115/0721da96/attachment.eml
-------------- next part --------------
An embedded message was scrubbed...
From: moses-support-request@mit.edu
Subject: confirm 8d9783eb57df37352d8b60f95027561c969d891e
Date: no date
Size: 631
Url: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160115/0721da96/attachment-0001.eml
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 111, Issue 42
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 111, Issue 42"
Post a Comment