Moses-support Digest, Vol 174, Issue 6

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: reg. moses installation (Namrata Hadimani)
2. Re: reg. moses installation (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Thu, 22 Apr 2021 21:50:13 +0100
From: Namrata Hadimani <namrata.hadimani@mycit.ie>
Subject: Re: [Moses-support] reg. moses installation
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>, ram anirudh cherukupally
<ramanirudh28@gmail.com>
Message-ID:
<CAN3QW0D1n-7tf4imZ25fb-h3j4kHnX0ZvWi6Kok=nUwuCrUjCA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi All,

I am trying to perform Language Model Training using the below command

~/mosesdecoder/bin/lmplz -o 3
<~/corpus/news-commentary-v8.fr-en.true.en >
news-commentary-v8.fr-en.arpa.en


followed similar to the link below :
http://www.statmt.org/moses/?n=Moses.Baseline

But I faced this issue :
=== 1/5 Counting and sorting n-grams ===
Reading /home/namrata/smt/corpus/news-commentary-v8.fr-en.true.en
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
tcmalloc: large alloc 3135389696 bytes == 0x144e000 @
tcmalloc: large alloc 10451279872 bytes == 0xbc272000 @
Unigram tokens 0 types 3
=== 2/5 Calculating and sorting adjusted counts ===
Chain sizes: 1:36 2:4734547456 3:8877277184
tcmalloc: large alloc 8877277184 bytes == 0x144e000 @
tcmalloc: large alloc 4734550016 bytes == 0x32ba4e000 @
terminate called after throwing an instance of
'lm::builder::BadDiscountException'
what(): /home/namrata/smt/mosesdecoder/lm/builder/adjust_counts.cc:53 in
void lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
lm::builder::DiscountConfig&) threw BadDiscountException because `s.n[j] ==
0'.
Could not calculate Kneser-Ney discounts for 1-grams with adjusted count 2
because we didn't observe any 1-grams with adjusted count 1; Is this small
or artificial data?
Try deduplicating the input. To override this error for e.g. a class-based
model, rerun with --discount_fallback

So I tried changing my command to the following as :
~/mosesdecoder/bin/lmplz -o 3 <~/corpus/news-commentary-v8.fr-en.true.en >
news-commentary-v8.fr-en.arpa.en *--**discount_fallback*

After this When i run the command : ~/mosesdecoder/bin/build_binary \

news-commentary-v8.fr-en.arpa.en \
news-commentary-v8.fr-en.blm.en



I am facing error like: lm/vocab.cc:324 in void
lm::ngram::MissingSentenceMarker(const lm::ngram::Config&, const char*)
threw SpecialWordMissingException.
The ARPA file is missing </s> and the model is configured to reject these
models. Run build_binary -s to disable this check. Byte: 66
ERROR

Could anyone help me out with this, please.

Regards,
Namrata Hadimani

On Thu, 22 Apr 2021 at 18:32, Namrata Hadimani <namrata.hadimani@mycit.ie>
wrote:

> Hi Hieu,
>
> Thanks for the help, I am able to successfully compile the moses ToolKit.
>
> Regards,
> Namrata Hadimani
>
> On Thu, 22 Apr 2021 at 17:23, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
>> i've just successfully compiled moses-4.0 on ubuntu 20.04 with boost 1.71.
>>
>> there's something wrong with your boost installation
>> On 4/22/2021 3:09 AM, ram anirudh cherukupally wrote:
>>
>> There is atleast 60 GB space, so I think it is not space issue. Has
>> moses-4.0 been tested for compilation using boost 1.71? Do you recommend
>> using boost 1.64 (as exemplified in Moses manual?)
>> Thank you
>>
>> On Thu, Apr 22, 2021 at 1:15 PM Hieu Hoang <hieuhoang@gmail.com> wrote:
>>
>>> there seems to be a problem with the boost library. Is the disk full?
>>> Perhaps you should re-install boost
>>> On 4/21/2021 10:39 PM, ram anirudh cherukupally wrote:
>>>
>>> Dear Moses-Support,
>>>
>>> Please find the build.log.gz attached as per the instructions when the
>>> build failed.
>>> My system specs:
>>>
>>> OS: Ubuntu 20.04
>>> RAM: 8 GB
>>> libboost-dev version: 1.71.0.0ubuntu2
>>>
>>> Command used for compiling moses: ./bjam -j4
>>>
>>> Thanks and Regards
>>>
>>> --
>>> CH Ram Anirudh
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>> --
>>> Hieu Hoanghttp://statmt.org/hieu
>>>
>>>
>>
>> --
>> CH Ram Anirudh
>>
>> --
>> Hieu Hoanghttp://statmt.org/hieu
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20210422/40f6e7d3/attachment-0001.html

------------------------------

Message: 2
Date: Thu, 22 Apr 2021 13:57:38 -0700
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] reg. moses installation
To: Namrata Hadimani <namrata.hadimani@mycit.ie>
Cc: moses-support <moses-support@mit.edu>, ram anirudh cherukupally
<ramanirudh28@gmail.com>
Message-ID: <7b9f03e0-dbdf-b7b7-37dc-99d3c1e8fec2@gmail.com>
Content-Type: text/plain; charset="utf-8"

cool. it would be good to say why the problem occurred in the first
place and how you fixed it so others can learn from it


On 4/22/2021 10:32 AM, Namrata Hadimani wrote:
> Hi?Hieu,
>
> Thanks for the help, I am able to successfully compile the moses ToolKit.
>
> Regards,
> Namrata Hadimani
>
> On Thu, 22 Apr 2021 at 17:23, Hieu Hoang <hieuhoang@gmail.com
> <mailto:hieuhoang@gmail.com>> wrote:
>
> i've just successfully compiled moses-4.0 on ubuntu 20.04 with
> boost 1.71.
>
> there's something wrong with your boost installation
>
> On 4/22/2021 3:09 AM, ram anirudh cherukupally wrote:
>> There is atleast 60 GB space, so I think it is not space issue.
>> Has moses-4.0 been tested for compilation using boost 1.71? Do
>> you recommend using boost 1.64? (as exemplified in Moses manual?)
>> Thank you
>>
>> On Thu, Apr 22, 2021 at 1:15 PM Hieu Hoang <hieuhoang@gmail.com
>> <mailto:hieuhoang@gmail.com>> wrote:
>>
>> there seems to be a problem with the boost library. Is the
>> disk full? Perhaps you should re-install boost
>>
>> On 4/21/2021 10:39 PM, ram anirudh cherukupally wrote:
>>> Dear Moses-Support,
>>>
>>> Please find the build.log.gz attached as per the
>>> instructions when the build failed.
>>> My system specs:
>>>
>>> OS: Ubuntu 20.04
>>> RAM: 8 GB
>>> libboost-dev version: 1.71.0.0ubuntu2
>>>
>>> Command used for compiling moses: ./bjam -j4
>>>
>>> Thanks and Regards
>>>
>>> --
>>> CH Ram Anirudh
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>
>>
>> --
>> Hieu Hoang
>> http://statmt.org/hieu <http://statmt.org/hieu>
>>
>>
>>
>> --
>> CH Ram Anirudh
>>
> --
> Hieu Hoang
> http://statmt.org/hieu <http://statmt.org/hieu>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
> <http://mailman.mit.edu/mailman/listinfo/moses-support>
>
--
Hieu Hoang
http://statmt.org/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20210422/ee9ef502/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 174, Issue 6
*********************************************

Tweet