Moses-support Digest, Vol 174, Issue 7

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: reg. moses installation (Kenneth Heafield)
2. Fwd: reg. moses installation (Namrata Hadimani)


----------------------------------------------------------------------

Message: 1
Date: Thu, 22 Apr 2021 23:33:43 +0100
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] reg. moses installation
To: <moses-support@mit.edu>
Message-ID: <df8a5661-a9c1-3f35-1cce-ed105ae43aa0@kheafield.com>
Content-Type: text/plain; charset=utf-8

Your training corpus is empty.

cat ~/corpus/news-commentary-v8.fr-en.true.en

On 4/22/21 9:50 PM, Namrata Hadimani wrote:
> Hi All,
>
> I am trying to perform Language Model Training using the below command?
>
> ~/mosesdecoder/bin/lmplz -o 3 <~/corpus/news-commentary-v8.fr-en.true.en > news-commentary-v8.fr-en.arpa.en
>
>
> followed similar to the link below :
> http://www.statmt.org/moses/?n=Moses.Baseline
> <http://www.statmt.org/moses/?n=Moses.Baseline>
>
> But I faced this issue? :
> ?=== 1/5 Counting and sorting n-grams ===
> Reading /home/namrata/smt/corpus/news-commentary-v8.fr-en.true.en
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> tcmalloc: large alloc 3135389696 bytes == 0x144e000 @
> tcmalloc: large alloc 10451279872 bytes == 0xbc272000 @
> Unigram tokens 0 types 3
> === 2/5 Calculating and sorting adjusted counts ===
> Chain sizes: 1:36 2:4734547456 3:8877277184
> tcmalloc: large alloc 8877277184 bytes == 0x144e000 @
> tcmalloc: large alloc 4734550016 bytes == 0x32ba4e000 @
> terminate called after throwing an instance of
> 'lm::builder::BadDiscountException'
> ? what(): ?/home/namrata/smt/mosesdecoder/lm/builder/adjust_counts.cc:53
> in void
> lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
> lm::builder::DiscountConfig&) threw BadDiscountException because `s.n[j]
> == 0'.
> Could not calculate Kneser-Ney discounts for 1-grams with adjusted count
> 2 because we didn't observe any 1-grams with adjusted count 1; Is this
> small or artificial data?
> Try deduplicating the input.? To override this error for e.g. a
> class-based model, rerun with --discount_fallback
>
> So I tried changing my command to the following as
> :??~/mosesdecoder/bin/lmplz -o 3
> <~/corpus/news-commentary-v8.fr-en.true.en >
> news-commentary-v8.fr-en.arpa.en?*--**discount_fallback*?
>
> After this When i run the command :?~/mosesdecoder/bin/build_binary \
>
> news-commentary-v8.fr-en.arpa.en \
> news-commentary-v8.fr-en.blm.en
>
>
>
> I am facing error like:? lm/vocab.cc:324 in void
> lm::ngram::MissingSentenceMarker(const lm::ngram::Config&, const char*)
> threw SpecialWordMissingException.
> The ARPA file is missing </s> and the model is configured to reject
> these models.? Run build_binary -s to disable this check. Byte: 66
> ERROR
>
> Could anyone help me out with this, please.
>
> Regards,
> Namrata Hadimani
>
> On Thu, 22 Apr 2021 at 18:32, Namrata Hadimani
> <namrata.hadimani@mycit.ie <mailto:namrata.hadimani@mycit.ie>> wrote:
>
> Hi?Hieu,
>
> Thanks for the help, I am able to successfully compile the moses
> ToolKit.
>
> Regards,
> Namrata Hadimani
>
> On Thu, 22 Apr 2021 at 17:23, Hieu Hoang <hieuhoang@gmail.com
> <mailto:hieuhoang@gmail.com>> wrote:
>
> i've just successfully compiled moses-4.0 on ubuntu 20.04 with
> boost 1.71.
>
> there's something wrong with your boost installation
>
> On 4/22/2021 3:09 AM, ram anirudh cherukupally wrote:
>> There is atleast 60 GB space, so I think it is not space
>> issue. Has moses-4.0 been tested for compilation using boost
>> 1.71? Do you recommend using boost 1.64? (as exemplified in
>> Moses manual?)
>> Thank you
>>
>> On Thu, Apr 22, 2021 at 1:15 PM Hieu Hoang
>> <hieuhoang@gmail.com <mailto:hieuhoang@gmail.com>> wrote:
>>
>> there seems to be a problem with the boost library. Is the
>> disk full? Perhaps you should re-install boost
>>
>> On 4/21/2021 10:39 PM, ram anirudh cherukupally wrote:
>>> Dear Moses-Support,
>>>
>>> Please find the build.log.gz attached as per the
>>> instructions when the build failed.
>>> My system specs:
>>>
>>> OS: Ubuntu 20.04
>>> RAM: 8 GB
>>> libboost-dev version: 1.71.0.0ubuntu2
>>>
>>> Command used for compiling moses: ./bjam -j4
>>>
>>> Thanks and Regards
>>>
>>> --
>>> CH Ram Anirudh
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>
>>
>> --
>> Hieu Hoang
>> http://statmt.org/hieu <http://statmt.org/hieu>
>>
>>
>>
>> --
>> CH Ram Anirudh
>>
> --
> Hieu Hoang
> http://statmt.org/hieu <http://statmt.org/hieu>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
> <http://mailman.mit.edu/mailman/listinfo/moses-support>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

Message: 2
Date: Fri, 23 Apr 2021 00:40:30 +0100
From: Namrata Hadimani <namrata.hadimani@mycit.ie>
Subject: [Moses-support] Fwd: reg. moses installation
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAN3QW0Cwia6-EW7NgwhdubW+aMVtoCCp2PWb4NNCs+vj2YPSfQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

++

---------- Forwarded message ---------
From: Namrata Hadimani <namrata.hadimani@mycit.ie>
Date: Fri, 23 Apr 2021 at 00:39
Subject: Re: [Moses-support] reg. moses installation
To: Kenneth Heafield <moses@kheafield.com>


Hi Kenneth,

Actually I am facing error in creating the Tokenisation itself. I tried to
run through the tokeniser.perl script but it didn't help me . There is no
log generated for this functionality.
Could you guide me more.

*Below are the scripts I ran :*

~/mosesdecoder/scripts/tokenizer/tokenizer.perl -l en \
< ~/corpus/training/news-commentary-v8.fr-en.en \
> ~/corpus/news-commentary-v8.fr-en.tok.en

*And I am getting this as a response :*

Tokenizer Version 1.1
Language: en
Number of threads: 1

*After this Step, the tokens are not created and the file is Empty. How
will I fix this problem?*

*Thanks in Advance*

Regards,
Namrata Hadimani



On Thu, 22 Apr 2021 at 23:45, Kenneth Heafield <moses@kheafield.com> wrote:

> Your training corpus is empty.
>
> cat ~/corpus/news-commentary-v8.fr-en.true.en
>
> On 4/22/21 9:50 PM, Namrata Hadimani wrote:
> > Hi All,
> >
> > I am trying to perform Language Model Training using the below command
> >
> > ~/mosesdecoder/bin/lmplz -o 3
> <~/corpus/news-commentary-v8.fr-en.true.en >
> news-commentary-v8.fr-en.arpa.en
> >
> >
> > followed similar to the link below :
> > http://www.statmt.org/moses/?n=Moses.Baseline
> > <http://www.statmt.org/moses/?n=Moses.Baseline>
> >
> > But I faced this issue :
> > === 1/5 Counting and sorting n-grams ===
> > Reading /home/namrata/smt/corpus/news-commentary-v8.fr-en.true.en
> >
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> > tcmalloc: large alloc 3135389696 bytes == 0x144e000 @
> > tcmalloc: large alloc 10451279872 bytes == 0xbc272000 @
> > Unigram tokens 0 types 3
> > === 2/5 Calculating and sorting adjusted counts ===
> > Chain sizes: 1:36 2:4734547456 3:8877277184
> > tcmalloc: large alloc 8877277184 bytes == 0x144e000 @
> > tcmalloc: large alloc 4734550016 bytes == 0x32ba4e000 @
> > terminate called after throwing an instance of
> > 'lm::builder::BadDiscountException'
> > what(): /home/namrata/smt/mosesdecoder/lm/builder/adjust_counts.cc:53
> > in void
> > lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
> > lm::builder::DiscountConfig&) threw BadDiscountException because `s.n[j]
> > == 0'.
> > Could not calculate Kneser-Ney discounts for 1-grams with adjusted count
> > 2 because we didn't observe any 1-grams with adjusted count 1; Is this
> > small or artificial data?
> > Try deduplicating the input. To override this error for e.g. a
> > class-based model, rerun with --discount_fallback
> >
> > So I tried changing my command to the following as
> > : ~/mosesdecoder/bin/lmplz -o 3
> > <~/corpus/news-commentary-v8.fr-en.true.en >
> > news-commentary-v8.fr-en.arpa.en *--**discount_fallback*
> >
> > After this When i run the command : ~/mosesdecoder/bin/build_binary \
> >
> > news-commentary-v8.fr-en.arpa.en \
> > news-commentary-v8.fr-en.blm.en
> >
> >
> >
> > I am facing error like: lm/vocab.cc:324 in void
> > lm::ngram::MissingSentenceMarker(const lm::ngram::Config&, const char*)
> > threw SpecialWordMissingException.
> > The ARPA file is missing </s> and the model is configured to reject
> > these models. Run build_binary -s to disable this check. Byte: 66
> > ERROR
> >
> > Could anyone help me out with this, please.
> >
> > Regards,
> > Namrata Hadimani
> >
> > On Thu, 22 Apr 2021 at 18:32, Namrata Hadimani
> > <namrata.hadimani@mycit.ie <mailto:namrata.hadimani@mycit.ie>> wrote:
> >
> > Hi Hieu,
> >
> > Thanks for the help, I am able to successfully compile the moses
> > ToolKit.
> >
> > Regards,
> > Namrata Hadimani
> >
> > On Thu, 22 Apr 2021 at 17:23, Hieu Hoang <hieuhoang@gmail.com
> > <mailto:hieuhoang@gmail.com>> wrote:
> >
> > i've just successfully compiled moses-4.0 on ubuntu 20.04 with
> > boost 1.71.
> >
> > there's something wrong with your boost installation
> >
> > On 4/22/2021 3:09 AM, ram anirudh cherukupally wrote:
> >> There is atleast 60 GB space, so I think it is not space
> >> issue. Has moses-4.0 been tested for compilation using boost
> >> 1.71? Do you recommend using boost 1.64 (as exemplified in
> >> Moses manual?)
> >> Thank you
> >>
> >> On Thu, Apr 22, 2021 at 1:15 PM Hieu Hoang
> >> <hieuhoang@gmail.com <mailto:hieuhoang@gmail.com>> wrote:
> >>
> >> there seems to be a problem with the boost library. Is the
> >> disk full? Perhaps you should re-install boost
> >>
> >> On 4/21/2021 10:39 PM, ram anirudh cherukupally wrote:
> >>> Dear Moses-Support,
> >>>
> >>> Please find the build.log.gz attached as per the
> >>> instructions when the build failed.
> >>> My system specs:
> >>>
> >>> OS: Ubuntu 20.04
> >>> RAM: 8 GB
> >>> libboost-dev version: 1.71.0.0ubuntu2
> >>>
> >>> Command used for compiling moses: ./bjam -j4
> >>>
> >>> Thanks and Regards
> >>>
> >>> --
> >>> CH Ram Anirudh
> >>>
> >>>
> >>> _______________________________________________
> >>> Moses-support mailing list
> >>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> >>> http://mailman.mit.edu/mailman/listinfo/moses-support <
> http://mailman.mit.edu/mailman/listinfo/moses-support>
> >>
> >> --
> >> Hieu Hoang
> >> http://statmt.org/hieu <http://statmt.org/hieu>
> >>
> >>
> >>
> >> --
> >> CH Ram Anirudh
> >>
> > --
> > Hieu Hoang
> > http://statmt.org/hieu <http://statmt.org/hieu>
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> > <http://mailman.mit.edu/mailman/listinfo/moses-support>
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20210422/4ac07621/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 174, Issue 7
*********************************************

Tweet