Moses-support Digest, Vol 96, Issue 11

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: perplexity scores (Rico Sennrich)
2. Re: filter the translation of text between tags (Hieu Hoang)
3. Re: perplexity scores (Kenneth Heafield)
4. Re: Factored Model | <s> Error ??? (Marwa Refaie)


----------------------------------------------------------------------

Message: 1
Date: Wed, 08 Oct 2014 10:56:21 +0100
From: Rico Sennrich <rico.sennrich@gmx.ch>
Subject: Re: [Moses-support] perplexity scores
To: koormoosh <koormoosh@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <54350A45.9050508@gmx.ch>
Content-Type: text/plain; charset="utf-8"

oh, you're also using different smoothing, and possibly different
handling of unknown words.

lmplz defaults to SRILM's|| '-interpolate -kndiscount -unk -gt3min 1
-gt4min 1 -gt5min 1'

On 08/10/14 10:05, koormoosh wrote:
> Thanks. Now it's 15 score closer to the KenLM, but still the
> difference is significant, 22. compared to KenLM 9.
> But still the difference is not close enough to be ignored.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141008/ab9fd399/attachment-0001.htm

------------------------------

Message: 2
Date: Wed, 08 Oct 2014 12:45:19 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] filter the translation of text between
tags
To: emna hkiri <emna.hkiri@gmail.com>, moses-support@mit.edu
Message-ID: <543523CF.4030209@gmail.com>
Content-Type: text/plain; charset="windows-1252"

Moses doesn't know anything about your tags. If you only want to
translate the text inside the tags, then only give that text to Moses.

If you are new to moses, please work through the tutorial
http://www.statmt.org/moses/?n=Moses.Baseline

On 07/10/14 17:25, emna hkiri wrote:
> Dear friends
>
> i'm a Moses beginner; i have
> two parallel texts
> the first one is an arabic corpus
> the second is an english corpus; in the english text i add the type
> of the named entity between the tags such us the following example
>
> *<person**> **Bill" Gates ** < /person> *born* < /date>* October 28,
> 1955* < /date>* is an American is the former chief executive and
> chairman of * < Organization>*Microsoft
> <http://en.wikipedia.org/wiki/Microsoft>* < /Organisation>*, the
> world?s largest personal-computersoftware
> <http://en.wikipedia.org/wiki/Software> company, which he co-founded
> with *<person**> *Paul Allen
> <http://en.wikipedia.org/wiki/Paul_Allen>*< /person>*.
>
> the texts are parallel and i add the tags only in the english text and
> i need that moses do only the translation of the text between tags. i.e
> Bill Gates
> October 28, 1955
> Microsoft <http://en.wikipedia.org/wiki/Microsoft>**
> *
> *
> i'don't know how to proceed
> thank you in advance
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141008/f15659c1/attachment-0001.htm

------------------------------

Message: 3
Date: Wed, 08 Oct 2014 08:45:20 -0400
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] perplexity scores
To: koormoosh@gmail.com
Cc: moses-support@mit.edu
Message-ID: <543531E0.1020601@kheafield.com>
Content-Type: text/plain; charset=ISO-8859-1

Hi,

There's documentation at http://kheafield.com/code/kenlm/estimation/ on
how to build (mostly) comparable models. I just updated it to reflect
the new default behavior of interpolating unigrams. Please also make
sure you have Moses 36da8d1 or later, since the policy is that
documentation reflects the state of master.

Also keep in mind that SRILM's perplexity is comparable to "Perplexity
excluding OOVs" line from query. And ARPA files are compatible so
nothing prevents you from using the query program from the other toolkit.

Kenneth

On 10/08/14 05:56, Rico Sennrich wrote:
> oh, you're also using different smoothing, and possibly different
> handling of unknown words.
>
> lmplz defaults to SRILM's|| '-interpolate -kndiscount -unk -gt3min 1
> -gt4min 1 -gt5min 1'
>
> On 08/10/14 10:05, koormoosh wrote:
>> Thanks. Now it's 15 score closer to the KenLM, but still the
>> difference is significant, 22. compared to KenLM 9.
>> But still the difference is not close enough to be ignored.
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

Message: 4
Date: Wed, 8 Oct 2014 16:17:46 +0000
From: Marwa Refaie <basmallah@hotmail.com>
Subject: Re: [Moses-support] Factored Model | <s> Error ???
To: Kenneth Heafield <moses@kheafield.com>, "moses-support@mit.edu"
<moses-support@mit.edu>
Message-ID: <DUB118-W13D3326DEDEED666430605BAA30@phx.gbl>
Content-Type: text/plain; charset="windows-1256"

When i applied moses tokeniser to the text file, all the '|' pipe as factor separator convert to "&#124" .... so can't produce the moses.ini as error ::
ERROR: Couldn't find factor 1 in token "artificial" in
c:/mosesdecoder-master/try/ai/sep/fsmt/data/UNpos.tok.en LINE 1

Any advice please ??


Marwa N. Refaie



> Date: Tue, 7 Oct 2014 21:13:36 -0400
> From: moses@kheafield.com
> To: moses-support@mit.edu
> Subject: Re: [Moses-support] Factored Model / <s> Error ???
>
> Well, does your data contain <s> in a position other than the first
> word? If so you should be escaping it e.g. with the Moses tokenizer.
>
> On 10/07/14 20:32, Marwa Refaie wrote:
> >
> >
> > Thank's for help ,, I fix all what mentioned now I'm stuck in this error ::
> >
> > Start loading text SCFG phrase table. Moses format : [1.000] seconds
> > Reading
> > /cygdrive/c/mosesdecoder-master/try/ai/sep/fsmt/work/model/phrase-table.
> >
> > 0,1-0,1.gz
> > ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80
> >
> > ---85---90---95--100
> > Either your data contains <s> in a position other than the first word or
> > your la
> > nguage model is missing <s>. Did you
> > build your ARPA using IRSTLM and forget to
> > run
> > add-start-end.sh?
> > [1]+ Done
> > c:/mosesdecoder-master/scripts/training/train-mode
> >
> > l.perl -external-bin-dir c:/mosesdecoder-master/tools/bin -root-dir
> > work -corpus
> >
> > c:/mosesdecoder-master/try/ai/sep/fsmt/data/UNpos.lo -f en -e ar
> > -Translation-f
> > actors 0,1-0,1 -lm
> > 0:5:/cygdrive/c:/mosesdecoder-master/try/ai/sep/fsmt/model/su
> >
> > rface.lm -lm
> > 1:5:/cygdrive/c:/mosesdecoder-master/try/ai/sep/fsmt/model/pos.lm &
> >
> > >training.out
> > Aborted (core dumped)
> >
> >
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > As mentioned I build non factored model , it worked well, but when I
> > start use the pos & surface language model for my training data I got
> > these errors ??
> > I used the SRILM.
> >
> >
> >
> >
> >
> > /*Marwa N. Refaie*/
> >
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141008/cc9c013e/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 96, Issue 11
*********************************************

Tweet