Moses-support Digest, Vol 174, Issue 8

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: reg. moses installation (Namrata Hadimani)


----------------------------------------------------------------------

Message: 1
Date: Fri, 23 Apr 2021 12:23:36 +0100
From: Namrata Hadimani <namrata.hadimani@mycit.ie>
Subject: Re: [Moses-support] reg. moses installation
To: Kenneth Heafield <moses@kheafield.com>, moses-support
<moses-support@mit.edu>
Message-ID:
<CAN3QW0CC7PmOqp=oiKxRuTk8XVsr5iE71gYmSd8ux=aCT41qtw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Is this some installation related ?
On Fri, 23 Apr 2021 at 12:39 AM, Namrata Hadimani <namrata.hadimani@mycit.ie>
wrote:

> Hi Kenneth,
>
> Actually I am facing error in creating the Tokenisation itself. I tried to
> run through the tokeniser.perl script but it didn't help me . There is no
> log generated for this functionality.
> Could you guide me more.
>
> *Below are the scripts I ran :*
>
> ~/mosesdecoder/scripts/tokenizer/tokenizer.perl -l en \
> < ~/corpus/training/news-commentary-v8.fr-en.en \
> > ~/corpus/news-commentary-v8.fr-en.tok.en
>
> *And I am getting this as a response :*
>
> Tokenizer Version 1.1
> Language: en
> Number of threads: 1
>
> *After this Step, the tokens are not created and the file is Empty. How
> will I fix this problem?*
>
> *Thanks in Advance*
>
> Regards,
> Namrata Hadimani
>
>
>
> On Thu, 22 Apr 2021 at 23:45, Kenneth Heafield <moses@kheafield.com>
> wrote:
>
>> Your training corpus is empty.
>>
>> cat ~/corpus/news-commentary-v8.fr-en.true.en
>>
>> On 4/22/21 9:50 PM, Namrata Hadimani wrote:
>> > Hi All,
>> >
>> > I am trying to perform Language Model Training using the below command
>> >
>> > ~/mosesdecoder/bin/lmplz -o 3
>> <~/corpus/news-commentary-v8.fr-en.true.en >
>> news-commentary-v8.fr-en.arpa.en
>> >
>> >
>> > followed similar to the link below :
>> > http://www.statmt.org/moses/?n=Moses.Baseline
>> > <http://www.statmt.org/moses/?n=Moses.Baseline>
>> >
>> > But I faced this issue :
>> > === 1/5 Counting and sorting n-grams ===
>> > Reading /home/namrata/smt/corpus/news-commentary-v8.fr-en.true.en
>> >
>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>> > tcmalloc: large alloc 3135389696 bytes == 0x144e000 @
>> > tcmalloc: large alloc 10451279872 bytes == 0xbc272000 @
>> > Unigram tokens 0 types 3
>> > === 2/5 Calculating and sorting adjusted counts ===
>> > Chain sizes: 1:36 2:4734547456 3:8877277184
>> > tcmalloc: large alloc 8877277184 bytes == 0x144e000 @
>> > tcmalloc: large alloc 4734550016 bytes == 0x32ba4e000 @
>> > terminate called after throwing an instance of
>> > 'lm::builder::BadDiscountException'
>> > what(): /home/namrata/smt/mosesdecoder/lm/builder/adjust_counts.cc:53
>> > in void
>> > lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
>> > lm::builder::DiscountConfig&) threw BadDiscountException because `s.n[j]
>> > == 0'.
>> > Could not calculate Kneser-Ney discounts for 1-grams with adjusted count
>> > 2 because we didn't observe any 1-grams with adjusted count 1; Is this
>> > small or artificial data?
>> > Try deduplicating the input. To override this error for e.g. a
>> > class-based model, rerun with --discount_fallback
>> >
>> > So I tried changing my command to the following as
>> > : ~/mosesdecoder/bin/lmplz -o 3
>> > <~/corpus/news-commentary-v8.fr-en.true.en >
>> > news-commentary-v8.fr-en.arpa.en *--**discount_fallback*
>> >
>> > After this When i run the command : ~/mosesdecoder/bin/build_binary \
>> >
>> > news-commentary-v8.fr-en.arpa.en \
>> > news-commentary-v8.fr-en.blm.en
>> >
>> >
>> >
>> > I am facing error like: lm/vocab.cc:324 in void
>> > lm::ngram::MissingSentenceMarker(const lm::ngram::Config&, const char*)
>> > threw SpecialWordMissingException.
>> > The ARPA file is missing </s> and the model is configured to reject
>> > these models. Run build_binary -s to disable this check. Byte: 66
>> > ERROR
>> >
>> > Could anyone help me out with this, please.
>> >
>> > Regards,
>> > Namrata Hadimani
>> >
>> > On Thu, 22 Apr 2021 at 18:32, Namrata Hadimani
>> > <namrata.hadimani@mycit.ie <mailto:namrata.hadimani@mycit.ie>> wrote:
>> >
>> > Hi Hieu,
>> >
>> > Thanks for the help, I am able to successfully compile the moses
>> > ToolKit.
>> >
>> > Regards,
>> > Namrata Hadimani
>> >
>> > On Thu, 22 Apr 2021 at 17:23, Hieu Hoang <hieuhoang@gmail.com
>> > <mailto:hieuhoang@gmail.com>> wrote:
>> >
>> > i've just successfully compiled moses-4.0 on ubuntu 20.04 with
>> > boost 1.71.
>> >
>> > there's something wrong with your boost installation
>> >
>> > On 4/22/2021 3:09 AM, ram anirudh cherukupally wrote:
>> >> There is atleast 60 GB space, so I think it is not space
>> >> issue. Has moses-4.0 been tested for compilation using boost
>> >> 1.71? Do you recommend using boost 1.64 (as exemplified in
>> >> Moses manual?)
>> >> Thank you
>> >>
>> >> On Thu, Apr 22, 2021 at 1:15 PM Hieu Hoang
>> >> <hieuhoang@gmail.com <mailto:hieuhoang@gmail.com>> wrote:
>> >>
>> >> there seems to be a problem with the boost library. Is the
>> >> disk full? Perhaps you should re-install boost
>> >>
>> >> On 4/21/2021 10:39 PM, ram anirudh cherukupally wrote:
>> >>> Dear Moses-Support,
>> >>>
>> >>> Please find the build.log.gz attached as per the
>> >>> instructions when the build failed.
>> >>> My system specs:
>> >>>
>> >>> OS: Ubuntu 20.04
>> >>> RAM: 8 GB
>> >>> libboost-dev version: 1.71.0.0ubuntu2
>> >>>
>> >>> Command used for compiling moses: ./bjam -j4
>> >>>
>> >>> Thanks and Regards
>> >>>
>> >>> --
>> >>> CH Ram Anirudh
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Moses-support mailing list
>> >>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> >>> http://mailman.mit.edu/mailman/listinfo/moses-support <
>> http://mailman.mit.edu/mailman/listinfo/moses-support>
>> >>
>> >> --
>> >> Hieu Hoang
>> >> http://statmt.org/hieu <http://statmt.org/hieu>
>> >>
>> >>
>> >>
>> >> --
>> >> CH Ram Anirudh
>> >>
>> > --
>> > Hieu Hoang
>> > http://statmt.org/hieu <http://statmt.org/hieu>
>> >
>> > _______________________________________________
>> > Moses-support mailing list
>> > Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> > http://mailman.mit.edu/mailman/listinfo/moses-support
>> > <http://mailman.mit.edu/mailman/listinfo/moses-support>
>> >
>> >
>> > _______________________________________________
>> > Moses-support mailing list
>> > Moses-support@mit.edu
>> > http://mailman.mit.edu/mailman/listinfo/moses-support
>> >
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20210423/e94d24a6/attachment-0001.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 174, Issue 8
*********************************************

Tweet