Moses-support Digest, Vol 86, Issue 16

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Warning: Too many arguments while IRSTLM language model
Training (renubalyan)
2. Re: merging two translation models (Rico Sennrich)
3. Re: Warning: Too many arguments while IRSTLM language model
Training (Barry Haddow)


----------------------------------------------------------------------

Message: 1
Date: Thu, 5 Dec 2013 23:02:23 +0530 (IST)
From: renubalyan <renubalyan@cdac.in>
Subject: Re: [Moses-support] Warning: Too many arguments while IRSTLM
language model Training
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<1953985012.12464.1386264743191.JavaMail.open-xchange@webmail.cdac.in>
Content-Type: text/plain; charset="utf-8"

Thanks a lot.

I managed to create the lm using the perl script instead of using steps 1-5.


Regards
Renu

On December 5, 2013 at 10:00 PM Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:

> Sorry, I was wrong and Prashant was correct.
> ./compile-lm --text
> creates the ARPA file.
>
> Perhaps an easier way to create a LM using IRSTLM is to use the Moses wrapper
> script
> scripts/generic/trainlm-irst2.perl
>
> This does steps 1 to 5 for you. Here is an example of how to run it
>
> /home/s0565741/workspace/github/hh/scripts/generic/trainlm-irst2.perl
> -cores 4 -irst-dir /home/s0565741/workspace/bin/irstlm/bin -p 0 -order 5
> -text
> /home/s0565741/workspace/experiment/europarl/en-es/lm/europarl.lowercased.1
> -lm /home/s0565741/workspace/experiment/europarl/en-es/lm/europarl.lm.1
>
>
>
>
>
> On 5 December 2013 15:12, renubalyan <renubalyan@cdac.in
> <mailto:renubalyan@cdac.in> > wrote:
> > > Hi,
> >
> > Thanks for the response.
> >
> > I tried this option too, if I run the command without '--text yes' option
> > then the command runs fine, However I wanted to ask one thing does this
> > give me an arpa file or a binarized one? Because when I run the next command
> > mentioned in the manual:
> >
> > 6. /home/renu/Desktop/mosesdecoder/bin/build_binary
> > news-commentary-v8.fr-en.arpa.en news-commentary-v8.fr-en.blm.en
> >
> > I get the following output:
> >
> > Reading news-commentary-v8.fr-en.arpa.en
> >
> > ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> >
> > ****************************************************************************************************
> > lm/read_arpa.cc:63 in void lm::ReadARPACounts(util::FilePiece&,
> > std::vector<long long unsigned int>&) threw FormatLoadException because
> > `line.size() >= 4 && StringPiece(line.data(), 4) == "blmt"'.
> > This looks like an IRSTLM binary file. Did you forget to pass --text yes
> > to compile-lm? Byte: 40 File: news-commentary-v8.fr-en.arpa.en
> > ERROR
> > The last second line put in bold indicates that the one I am using is a
> > binary file.
> > Does that mean I already have a binary file and I do not need to use
> > step 6 mentioned above (which infact is for converting from arpa to binary
> > file)
> >
> >
> > Thanks
> > Renu
> >
> >
> >
> >
> >
> > On December 5, 2013 at 4:19 PM Hieu Hoang < Hieu.Hoang@ed.ac.uk
> > <mailto:Hieu.Hoang@ed.ac.uk> > wrote:
> >
> > > > > I'm not sure what is
> > > --text yes
> > > this is how the EMS runs IRSTLM compile-lm:
> > > .../compile-lm .../europarl_pos.lm.4 .../europarl_pos.binlm.4
> > >
> > >
> > > On 4 December 2013 15:58, renubalyan <renubalyan@cdac.in
> > > <mailto:renubalyan@cdac.in> > wrote:
> > > > > > > Hi,
> > > >
> > > > I am building the baseline system based on Moses manual
> > > > instructions.
> > > >
> > > > I have installed Moses, GIZA++ and IRSTLM as mentioned in the
> > > > manual.
> > > > The corpus preparation (tokenization, ...cleaning) steps also goes
> > > > well.
> > > >
> > > > However when I move to Language Model Training: I have some
> > > > problems
> > > >
> > > > I am following these steps:
> > > >
> > > > 1. mkdir ~/lm
> > > >
> > > > 2. cd ~/lm
> > > >
> > > > 3. /home/renu/Desktop/irstlm/bin/add-start-end.sh <
> > > > /home/renu/Desktop/corpus/news-commentary-v8.fr-en.true.en>
> > > > news-commentary-v8.fr-en.sb.en
> > > >
> > > > 4. export IRSTLM=/home/renu/Desktop/irstlm;
> > > > /home/renu/Desktop/irstlm/bin/build-lm.sh -i
> > > > news-commentary-v8.fr-en.sb.en -t ./tmp -p -s improved-kneser-ney -o
> > > > news-commentary-v8.fr-en.lm.en
> > > >
> > > > 5. /home/renu/Desktop/irstlm/bin/compile-lm --text yes
> > > > news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
> > > >
> > > > Steps 1-4 work well but step 5 gives me -------(Warning:Too many
> > > > parameters)
> > > >
> > > > I have searched the web for any possible solution but could not
> > > > find any.
> > > >
> > > > I am not able to move ahead, kindly help.
> > > >
> > > > Thanks
> > > > Renu
> > > >
> > > >
> > > > -------------------------------------------------------------------------------------------------------------------------------
> > > > This e-mail is for the sole use of the intended recipient(s) and
> > > > may
> > > > contain confidential and privileged information. If you are not
> > > > the
> > > > intended recipient, please contact the sender by reply e-mail and
> > > > destroy
> > > > all copies and the original message. Any unauthorized review, use,
> > > > disclosure, dissemination, forwarding, printing or copying of this
> > > > email
> > > > is strictly prohibited and appropriate legal action will be taken.
> > > >
> > > > -------------------------------------------------------------------------------------------------------------------------------
> > > >
> > > > _______________________________________________
> > > > Moses-support mailing list
> > > > Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> > > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > > > <http://mailman.mit.edu/mailman/listinfo/moses-support>
> > > > > > >
> > >
> > >
> > > --
> > > Hieu Hoang
> > > Research Associate
> > > University of Edinburgh
> > > http://www.hoang.co.uk/hieu <http://www.hoang.co.uk/hieu>
> > >
> > > > >
> >
> >
> > -------------------------------------------------------------------------------------------------------------------------------
> > This e-mail is for the sole use of the intended recipient(s) and may
> > contain confidential and privileged information. If you are not the
> > intended recipient, please contact the sender by reply e-mail and destroy
> > all copies and the original message. Any unauthorized review, use,
> > disclosure, dissemination, forwarding, printing or copying of this email
> > is strictly prohibited and appropriate legal action will be taken.
> >
> > -------------------------------------------------------------------------------------------------------------------------------
> > >
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu <http://www.hoang.co.uk/hieu>
>
>

-------------------------------------------------------------------------------------------------------------------------------

This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.
-------------------------------------------------------------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131205/007adffe/attachment-0001.htm

------------------------------

Message: 2
Date: Thu, 05 Dec 2013 21:15:39 +0000
From: Rico Sennrich <rico.sennrich@gmx.ch>
Subject: Re: [Moses-support] merging two translation models
To: moses-support@mit.edu
Message-ID: <52A0ECFB.6070307@gmx.ch>
Content-Type: text/plain; charset=KOI8-R; format=flowed

Adding/removing models during decoding is not currently supported. The
code for loading feature functions (including translation models) has
recently been refactored, and I don't know how easy it would be to add
such a functionality in this new framework.

On 04.12.2013 18:00, ??????? ????????? wrote:
> Many thanx to your replies!
>
> One more question concerning these scripts: are there capability to detach translation models to free RAM and attached new TMs while the decoder is running? If not maybe you can provide a roadmap for me to contribute such functionality?
>
> Kind regards!


------------------------------

Message: 3
Date: Thu, 05 Dec 2013 21:19:44 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Warning: Too many arguments while IRSTLM
language model Training
To: renubalyan <renubalyan@cdac.in>, moses-support@mit.edu
Message-ID: <52A0EDF0.3010402@staffmail.ed.ac.uk>
Content-Type: text/plain; charset="iso-8859-1"

Hi

It looks like you are following the Moses baseline instructions
(http://www.statmt.org/moses/?n=Moses.Baseline). It's not explained, but
step 5 should convert the IRSTLM iARPA file produced by step 4 to a
(standard) ARPA file. The following step will then binarise it with KenLM.

The command you ran is

/home/renu/Desktop/irstlm/bin/compile-lm --text yes
news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en

I notice that someone added a "yes" to this command in the
documentation recently (November 13th). Does it work if you don't
include "yes"?

IRSTLM folks - can you clarify? Does the '--text' parameter require a
'yes' argument? The usage for the command suggests it does, but it used
to work without,

cheers - Barry

On 04/12/13 15:58, renubalyan wrote:
> Hi,
> I am building the baseline system based on Moses manual instructions.
> I have installed Moses, GIZA++ and IRSTLM as mentioned in the manual.
> The corpus preparation (tokenization, ...cleaning) steps also goes well.
> However when I move to Language Model Training: I have some problems
> I am following these steps:
> 1. mkdir ~/lm
>
> 2. cd ~/lm
>
> 3. /home/renu/Desktop/irstlm/bin/add-start-end.sh <
> /home/renu/Desktop/corpus/news-commentary-v8.fr-en.true.en>
> news-commentary-v8.fr-en.sb.en
>
> 4. export IRSTLM=/home/renu/Desktop/irstlm;
> /home/renu/Desktop/irstlm/bin/build-lm.sh -i
> news-commentary-v8.fr-en.sb.en -t ./tmp -p -s improved-kneser-ney -o
> news-commentary-v8.fr-en.lm.en
>
> 5. /home/renu/Desktop/irstlm/bin/compile-lm --text yes
> news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
> Steps 1-4 work well but step 5 gives me -------(Warning:Too many
> parameters)
>
> I have searched the web for any possible solution but could not find any.
> I am not able to move ahead, kindly help.
> Thanks
> Renu
>
> -------------------------------------------------------------------------------------------------------------------------------
>
> This e-mail is for the sole use of the intended recipient(s) and may
> contain confidential and privileged information. If you are not the
> intended recipient, please contact the sender by reply e-mail and destroy
> all copies and the original message. Any unauthorized review, use,
> disclosure, dissemination, forwarding, printing or copying of this email
> is strictly prohibited and appropriate legal action will be taken.
> -------------------------------------------------------------------------------------------------------------------------------
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131205/b2ba6455/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 86, Issue 16
*********************************************

Tweet