Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Warning: Too many arguments while IRSTLM language model
Training (Renu Balyan)
2. Re: Warning: Too many arguments while IRSTLM language model
Training (Barry Haddow)
----------------------------------------------------------------------
Message: 1
Date: Fri, 6 Dec 2013 09:57:02 +0530
From: "Renu Balyan" <renubalyan@cdac.in>
Subject: Re: [Moses-support] Warning: Too many arguments while IRSTLM
language model Training
To: "Barry Haddow" <bhaddow@staffmail.ed.ac.uk>,
<moses-support@mit.edu>
Message-ID: <708CA6CAC14E4EA79164E938B9942567@renubalyan>
Content-Type: text/plain; charset="iso-8859-1"
Hi,
Thanks for the response.
Yes, I am using the Moses baseline instructions (http://www.statmt.org/moses/?n=Moses.Baseline).
I tried this option too, if I run the command without '--text yes' option then the command runs fine, However I wanted to ask one thing does this give me an arpa file or a binarized one? Because when I run the next command mentioned in the manual:
6. /home/renu/Desktop/mosesdecoder/bin/build_binary news-commentary-v8.fr-en.arpa.en news-commentary-v8.fr-en.blm.en
I get the following output:
Reading news-commentary-v8.fr-en.arpa.en
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
lm/read_arpa.cc:63 in void lm::ReadARPACounts(util::FilePiece&, std::vector<long long unsigned int>&) threw FormatLoadException because `line.size() >= 4 && StringPiece(line.data(), 4) == "blmt"'.
This looks like an IRSTLM binary file. Did you forget to pass --text yes to compile-lm? Byte: 40 File: news-commentary-v8.fr-en.arpa.en
ERROR
The last second line put in bold indicates that the one I am using is a binary file.
Does that mean I already have a binary file and I do not need to use step 6 mentioned above (which infact is for converting from arpa to binary file)
However, I did not try to run it by keeping '--text' and removing 'yes' from the command. I will try this as well.
Thanks
Renu
----- Original Message -----
From: Barry Haddow
To: renubalyan ; moses-support@mit.edu
Sent: Friday, December 06, 2013 2:49 AM
Subject: Re: [Moses-support] Warning: Too many arguments while IRSTLM language model Training
Hi
It looks like you are following the Moses baseline instructions (http://www.statmt.org/moses/?n=Moses.Baseline). It's not explained, but step 5 should convert the IRSTLM iARPA file produced by step 4 to a (standard) ARPA file. The following step will then binarise it with KenLM.
The command you ran is
/home/renu/Desktop/irstlm/bin/compile-lm --text yes news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
I notice that someone added a "yes" to this command in the documentation recently (November 13th). Does it work if you don't include "yes"?
IRSTLM folks - can you clarify? Does the '--text' parameter require a 'yes' argument? The usage for the command suggests it does, but it used to work without,
cheers - Barry
On 04/12/13 15:58, renubalyan wrote:
Hi,
I am building the baseline system based on Moses manual instructions.
I have installed Moses, GIZA++ and IRSTLM as mentioned in the manual.
The corpus preparation (tokenization, ...cleaning) steps also goes well.
However when I move to Language Model Training: I have some problems
I am following these steps:
1. mkdir ~/lm
2. cd ~/lm
3. /home/renu/Desktop/irstlm/bin/add-start-end.sh < /home/renu/Desktop/corpus/news-commentary-v8.fr-en.true.en> news-commentary-v8.fr-en.sb.en
4. export IRSTLM=/home/renu/Desktop/irstlm; /home/renu/Desktop/irstlm/bin/build-lm.sh -i news-commentary-v8.fr-en.sb.en -t ./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en
5. /home/renu/Desktop/irstlm/bin/compile-lm --text yes news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
Steps 1-4 work well but step 5 gives me -------(Warning:Too many parameters)
I have searched the web for any possible solution but could not find any.
I am not able to move ahead, kindly help.
Thanks
Renu
-------------------------------------------------------------------------------------------------------------------------------
This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.
-------------------------------------------------------------------------------------------------------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
-------------------------------------------------------------------------------------------------------------------------------
This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.
-------------------------------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131206/8b5b8d67/attachment-0001.htm
------------------------------
Message: 2
Date: Fri, 06 Dec 2013 08:36:37 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Warning: Too many arguments while IRSTLM
language model Training
To: Renu Balyan <renubalyan@cdac.in>, moses-support@mit.edu
Message-ID: <52A18C95.10301@staffmail.ed.ac.uk>
Content-Type: text/plain; charset="iso-8859-1"
Hi Renu
If you run without "--text yes" then you will get an IRSTLM binary file,
as the error message shows, so you don't have to run step 6. This file
is different from a KenLM binary file. You can also use the IRSTLM
binary file in Moses, as long as you adjust moses.ini to tell Moses what
type of LM you have.
However, I was suggesting running with just the "--text" argument (i.e.
just remove the "yes") to see if that still works,
cheers - Barry
On 06/12/13 04:27, Renu Balyan wrote:
> Hi,
> Thanks for the response.
> Yes, I am using the Moses baseline instructions
> (http://www.statmt.org/moses/?n=Moses.Baseline).
> I tried this option too, if I run the command without '--text yes'
> option then the command runs fine, However I wanted to ask one thing
> does this give me an arpa file or a binarized one? Because when I run
> the next command mentioned in the manual:
> 6. /home/renu/Desktop/mosesdecoder/bin/build_binary
> news-commentary-v8.fr-en.arpa.en news-commentary-v8.fr-en.blm.en
> *I get the following output:*
>
> Reading news-commentary-v8.fr-en.arpa.en
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>
> ****************************************************************************************************
>
> lm/read_arpa.cc:63 in void lm::ReadARPACounts(util::FilePiece&,
> std::vector<long long unsigned int>&) threw FormatLoadException
> because `line.size() >= 4 && StringPiece(line.data(), 4) == "blmt"'.
> *This looks like an IRSTLM binary file. Did you forget to pass --text
> yes to compile-lm? Byte: 40 File: news-commentary-v8.fr-en.arpa.en*
> ERROR
>
> The last second line put in bold indicates that the one I am using is
> a binary file.
> Does that mean I already have a binary file and I do not need to use
> step 6 mentioned above (which infact is for converting from arpa to
> binary file)
> However, I did not try to run it by keeping '--text' and
> removing 'yes' from the command. I will try this as well.
> Thanks
> Renu
>
> ----- Original Message -----
> *From:* Barry Haddow <mailto:bhaddow@staffmail.ed.ac.uk>
> *To:* renubalyan <mailto:renubalyan@cdac.in> ;
> moses-support@mit.edu <mailto:moses-support@mit.edu>
> *Sent:* Friday, December 06, 2013 2:49 AM
> *Subject:* Re: [Moses-support] Warning: Too many arguments while
> IRSTLM language model Training
>
> Hi
>
> It looks like you are following the Moses baseline instructions
> (http://www.statmt.org/moses/?n=Moses.Baseline). It's not
> explained, but step 5 should convert the IRSTLM iARPA file
> produced by step 4 to a (standard) ARPA file. The following step
> will then binarise it with KenLM.
>
> The command you ran is
>
> /home/renu/Desktop/irstlm/bin/compile-lm --text yes
> news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
>
> I notice that someone added a "yes" to this command in the
> documentation recently (November 13th). Does it work if you don't
> include "yes"?
>
> IRSTLM folks - can you clarify? Does the '--text' parameter
> require a 'yes' argument? The usage for the command suggests it
> does, but it used to work without,
>
> cheers - Barry
>
> On 04/12/13 15:58, renubalyan wrote:
>> Hi,
>> I am building the baseline system based on Moses manual
>> instructions.
>> I have installed Moses, GIZA++ and IRSTLM as mentioned in the
>> manual.
>> The corpus preparation (tokenization, ...cleaning) steps also
>> goes well.
>> However when I move to Language Model Training: I have some problems
>> I am following these steps:
>> 1. mkdir ~/lm
>>
>> 2. cd ~/lm
>>
>> 3. /home/renu/Desktop/irstlm/bin/add-start-end.sh <
>> /home/renu/Desktop/corpus/news-commentary-v8.fr-en.true.en>
>> news-commentary-v8.fr-en.sb.en
>>
>> 4. export IRSTLM=/home/renu/Desktop/irstlm;
>> /home/renu/Desktop/irstlm/bin/build-lm.sh -i
>> news-commentary-v8.fr-en.sb.en -t ./tmp -p -s improved-kneser-ney
>> -o news-commentary-v8.fr-en.lm.en
>>
>> 5. /home/renu/Desktop/irstlm/bin/compile-lm --text yes
>> news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
>> Steps 1-4 work well but step 5 gives me -------(Warning:Too many
>> parameters)
>>
>> I have searched the web for any possible solution but could not
>> find any.
>> I am not able to move ahead, kindly help.
>> Thanks
>> Renu
>>
>> -------------------------------------------------------------------------------------------------------------------------------
>>
>> This e-mail is for the sole use of the intended recipient(s) and may
>> contain confidential and privileged information. If you are not the
>> intended recipient, please contact the sender by reply e-mail and
>> destroy
>> all copies and the original message. Any unauthorized review, use,
>> disclosure, dissemination, forwarding, printing or copying of
>> this email
>> is strictly prohibited and appropriate legal action will be taken.
>> -------------------------------------------------------------------------------------------------------------------------------
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> -------------------------------------------------------------------------------------------------------------------------------
>
> This e-mail is for the sole use of the intended recipient(s) and may
> contain confidential and privileged information. If you are not the
> intended recipient, please contact the sender by reply e-mail and destroy
> all copies and the original message. Any unauthorized review, use,
> disclosure, dissemination, forwarding, printing or copying of this email
> is strictly prohibited and appropriate legal action will be taken.
> -------------------------------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131206/3bdb6907/attachment.htm
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 86, Issue 17
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 86, Issue 17"
Post a Comment