Moses-support Digest, Vol 92, Issue 38

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. help (Maria Marpaung)
2. Re: Creating a 2-gram language model (Hieu Hoang)
3. Re: help (Nicola Bertoldi)


----------------------------------------------------------------------

Message: 1
Date: Fri, 20 Jun 2014 12:47:45 +0800
From: Maria Marpaung <maria_marpaung@yahoo.co.id>
Subject: [Moses-support] help
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<1403239665.74077.YahooMailNeo@web190903.mail.sg3.yahoo.com>
Content-Type: text/plain; charset="utf-8"


Hello, I need help.
I'm working on my thesis for form translation system using MOSES MT. I came to the Language model Training. I am having some problems. I have followed some of the steps like following the Moses Baseline. including the following: 1. The language model (LM) is used to ensure fluent output, so it is built
with the target language (i.e Indonesia language in this case). The IRSTLM
documentation gives a full explanation of the command-line options, but the
following will build an appropriate 3-gram language model, removing
singletons, smoothing with improved Kneser-Ney, and adding sentence
boundary symbols: mkdir ~/lm cd ~/lm ~/irstlm/bin/add-start-end.sh \ < ~/corpus/news-commentary-v8.fr-en.true.en \ > news-commentary-v8.fr-en.sb.en export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh \ -i news-commentary-v8.fr-en.sb.en \ -t ./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en ~/irstlm/bin/compile-lm --text news-commentary-v8.fr-en.lm.en.gz \ news-commentary-v8.fr-en.arpa.en First until four commands were executed successfuly. The last one failed.
Here is the result after entering the following command line: maria@maria-Aspire-E1-471:~/lm$ ~/moses/irstlm/bin/compile-lm --text
news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en inpfile: news-commentary-v8.fr-en.lm.en.gz
outfile: news-commentary-v8.fr-en.arpa.en
loading up to the LM level 1000 (if any)
dub: 10000000
Failed to open news-commentary-v8.fr-en.lm.en.gz!


2. I add command "yes" like this:
maria@maria-Aspire-E1-471:~/lm$ ~/moses/irstlm/bin/compile-lm ?text yes
news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en

Warning: Too many arguments



please help me, what should I do?


Best regards!

Maria Marpaung
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140620/61e793c6/attachment-0001.htm

------------------------------

Message: 2
Date: Fri, 20 Jun 2014 06:02:37 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Creating a 2-gram language model
To: Rajkiran Rajkumar <rajkiran2507@gmail.com>, moses-support@mit.edu
Message-ID: <53A3C06D.7020408@gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

if you look inside the code or run the script without any argument, it
tells you this:
-n Order of language model (default 3)


On 19/06/14 07:19, Rajkiran Rajkumar wrote:
> In the baseline translation system, the command -
> "~/irstlm/bin/build-lm.sh \
> -i news-commentary-v8.fr-en.sb.en \
> -t ./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en"
>
> is used to create a "3-gram language model, removing singletons,
> smoothing with improved Kneser-Ney, and adding sentence boundary
> symbols" is what the tutorial says. How can I build a 2-gram language
> model?
>
> And, in general, which is more efficient for a bilingual corpus of
> 160,000 sentences? 2-gram or 3-gram?
>
> Thanks in advance,
> Rajkiran
> College of Engineering Guindy, India
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140620/aa388890/attachment-0001.htm

------------------------------

Message: 3
Date: Fri, 20 Jun 2014 09:56:53 +0200
From: Nicola Bertoldi <bertoldi@fbk.eu>
Subject: Re: [Moses-support] help
To: Maria Marpaung <maria_marpaung@yahoo.co.id>
Cc: moses-support support <moses-support@mit.edu>
Message-ID: <54472063-6182-4343-88D7-3890924860DA@fbk.eu>
Content-Type: text/plain; charset="windows-1252"

Hi Maria
first I would like to ask (for next time)
to write to the IRSTLM mailing list (user-irstlm@list.fbk.eu)
to any problem related to IRSTLM only


Now I am going to reply you privately.


Nicola



On Jun 20, 2014, at 6:47 AM, Maria Marpaung wrote:

> Hello, I need help.
> I'm working on my thesis for form translation system using MOSES MT. I came to the Language model Training. I am having some problems. I have followed some of the steps like following the Moses Baseline. including the following:
>
> 1. The language model (LM) is used to ensure fluent output, so it is built
> with the target language (i.e Indonesia language in this case). The IRSTLM
> documentation gives a full explanation of the command-line options, but the
> following will build an appropriate 3-gram language model, removing
> singletons, smoothing with improved Kneser-Ney, and adding sentence
> boundary symbols:
>
> mkdir ~/lm
> cd ~/lm
> ~/irstlm/bin/add-start-end.sh \
> < ~/corpus/news-commentary-v8.fr-en.true.en \
> > news-commentary-v8.fr-en.sb.en
> export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh \
> -i news-commentary-v8.fr-en.sb.en \
> -t ./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en
> ~/irstlm/bin/compile-lm --text news-commentary-v8.fr-en.lm.en.gz \
> news-commentary-v8.fr-en.arpa.en
>
> First until four commands were executed successfuly. The last one failed.
> Here is the result after entering the following command line:
>
> maria@maria-Aspire-E1-471:~/lm$ ~/moses/irstlm/bin/compile-lm --text
> news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
>
> inpfile: news-commentary-v8.fr-en.lm.en.gz
> outfile: news-commentary-v8.fr-en.arpa.en
> loading up to the LM level 1000 (if any)
> dub: 10000000
> Failed to open news-commentary-v8.fr-en.lm.en.gz!
>
>
> 2. I add command "yes" like this:
> maria@maria-Aspire-E1-471:~/lm$ ~/moses/irstlm/bin/compile-lm ?text yes
> news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
>
> Warning: Too many arguments
>
>
>
> please help me, what should I do?
>
>
> Best regards!
>
> Maria Marpaung
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140620/cd0b9b46/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 92, Issue 38
*********************************************

Tweet