Moses-support Digest, Vol 111, Issue 40

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Error on lmplz (Kenneth Heafield)
2. Re: Error on lmplz (Lane Schwartz)


----------------------------------------------------------------------

Message: 1
Date: Wed, 13 Jan 2016 17:44:39 +0000
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] Error on lmplz
To: Lane Schwartz <dowobeha@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <56968D07.2030309@kheafield.com>
Content-Type: text/plain; charset=utf-8

Works as intended.

On 01/13/2016 02:59 PM, Lane Schwartz wrote:
> Thanks, Kenneth. Here's what I get now.
>
> $ ~/mosesdecoder.multisource.git/bin/lmplz -o 2 <<< "that is what
> happens ? cssd has nothing more or voldemort or pastries in prague ."
> === 1/5 Counting and sorting n-grams ===
> Reading /tmp/sh-thd-1452698150 (deleted)
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> tcmalloc: large alloc 29442056192 bytes == 0x1c74000 @
> tcmalloc: large alloc 78512136192 bytes == 0x6de346000 @
> ****************************************************************************************************
> Unigram tokens 16 types 18
> === 2/5 Calculating and sorting adjusted counts ===
> Chain sizes: 1:216 2:107979354931
> tcmalloc: large alloc 107979358208 bytes == 0x192a648000 @
> terminate called after throwing an instance of
> 'lm::builder::BadDiscountException'
> what():
> /home/lanes/mosesdecoder.multisource.git/lm/builder/adjust_counts.cc:53
> in void
> lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
> lm::builder::DiscountConfig&) threw BadDiscountException because
> `s.n[j] == 0'.
> Could not calculate Kneser-Ney discounts for 1-grams with adjusted
> count 4 because we didn't observe any 1-grams with adjusted count 3;
> Is this small or artificial data?
> Try deduplicating the input. To override this error for e.g. a
> class-based model, rerun with --discount_fallback
> Aborted (core dumped)
>
>
>
> On Tue, Jan 12, 2016 at 5:40 PM, Kenneth Heafield <moses@kheafield.com
> <mailto:moses@kheafield.com>> wrote:
>
> Pushed the fix from kenlm master in October to Moses master.
>
> On 01/12/2016 10:34 PM, Lane Schwartz wrote:
> > Steps to reproduce this error:
> >
> > $ ~/mosesdecoder.git/bin/lmplz -o 2 <<< "that is what happens ? cssd
> > has nothing more or voldemort or pastries in prague ."
> > === 1/5 Counting and sorting n-grams ===
> > Reading /tmp/sh-thd-107574999377 (deleted)
> > ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> > tcmalloc: large alloc 29442056192 bytes == 0x2ae2000 @
> > tcmalloc: large alloc 78512136192 bytes == 0x6df1b4000 @
> > ****************************************************************************************************
> > Unigram tokens 16 types 18
> > === 2/5 Calculating and sorting adjusted counts ===
> > Chain sizes: 1:216 2:107979354931
> > tcmalloc: large alloc 107979358208 bytes == 0x192b4b6000 @
> > lmplz: ./util/fixed_array.hh:104: T&
> > util::FixedArray<T>::operator[](std::size_t) [with T =
> > lm::NGramStream<lm::builder::BuildingPayload>; std::size_t = long
> > unsigned int]: Assertion `i < size()' failed.
> >
> >
> >
> >
> > On Wed, Sep 30, 2015 at 11:41 AM, Kenneth Heafield <moses@kheafield.com <mailto:moses@kheafield.com>
> > <mailto:moses@kheafield.com <mailto:moses@kheafield.com>>> wrote:
> >
> > That's bad. Would you mind sending me privately a minimal
> example of
> > the data that reproduces the problem?
> >
> > Kenneth
> >
> > On 09/30/2015 04:29 PM, Alex Martinez wrote:
> > > Hello,
> > > today I've pulled moses code and recompiled and some
> experiments (EMS)
> > > that were already working are failing on the LM training
> step with the
> > > following error:
> > >
> > > Executing: /opt/moses/bin/lmplz --text
> > > /home/alexmc/devel/toydata/process/lm/nc=pos.factored.1
> --order 5
> > --arpa
> > > /home/alexmc/devel/toydata/process/lm/nc=pos.lm.1
> --discount_fallback
> > > === 1/5 Counting and sorting n-grams ===
> > > Reading /mnt/a62/devel/toydata/process/lm/nc=pos.factored.1
> > >
> >
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> > > tcmalloc: large alloc 4753956864 bytes == 0x1f7c000 @
> > > tcmalloc: large alloc 22185107456 bytes == 0x11d536000 @
> > >
> >
> ****************************************************************************************************
> > > Unigram tokens 2433135 types 47
> > > === 2/5 Calculating and sorting adjusted counts ===
> > > Chain sizes: 1:564 2:2630656000 3:4932480000 4:7891967488
> > 5:11509120000
> > > tcmalloc: large alloc 11509121024 bytes == 0x1f7c000 @
> > > tcmalloc: large alloc 2630656000 bytes == 0x2aff70000 @
> > > tcmalloc: large alloc 4932485120 bytes == 0x34cc3a000 @
> > > tcmalloc: large alloc 7891968000 bytes == 0x64933c000 @
> > > lmplz: ./util/fixed_array.hh:104: T&
> > > util::FixedArray<T>::operator[](std::size_t) [with T =
> > > lm::NGramStream<lm::builder::BuildingPayload>; std::size_t =
> long
> > > unsigned int]: Assertion `i < size()' failed.
> > >
> > > I'm runing a Linux server with Ubuntu 15.04
> > >
> > > Any help will be appreciated
> > >
> > > Alex Mart?nez
> > >
> > >
> > > _______________________________________________
> > > Moses-support mailing list
> > > Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> <mailto:Moses-support@mit.edu <mailto:Moses-support@mit.edu>>
> > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> <mailto:Moses-support@mit.edu <mailto:Moses-support@mit.edu>>
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> >
> >
> >
> > --
> > When a place gets crowded enough to require ID's, social collapse
> is not
> > far away. It is time to go elsewhere. The best thing about space
> travel
> > is that it made it possible to go elsewhere.
> > -- R.A. Heinlein, "Time Enough For Love"
>
>
>
>
> --
> When a place gets crowded enough to require ID's, social collapse is not
> far away. It is time to go elsewhere. The best thing about space travel
> is that it made it possible to go elsewhere.
> -- R.A. Heinlein, "Time Enough For Love"


------------------------------

Message: 2
Date: Wed, 13 Jan 2016 11:59:33 -0600
From: Lane Schwartz <dowobeha@gmail.com>
Subject: Re: [Moses-support] Error on lmplz
To: Kenneth Heafield <moses@kheafield.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZnx7=2N8RdD+L3Bky5ThgeKSgsDk2=7C6aM+izLb4nRiQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Thanks!

On Wed, Jan 13, 2016 at 11:44 AM, Kenneth Heafield <moses@kheafield.com>
wrote:

> Works as intended.
>
> On 01/13/2016 02:59 PM, Lane Schwartz wrote:
> > Thanks, Kenneth. Here's what I get now.
> >
> > $ ~/mosesdecoder.multisource.git/bin/lmplz -o 2 <<< "that is what
> > happens ? cssd has nothing more or voldemort or pastries in prague ."
> > === 1/5 Counting and sorting n-grams ===
> > Reading /tmp/sh-thd-1452698150 (deleted)
> >
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> > tcmalloc: large alloc 29442056192 bytes == 0x1c74000 @
> > tcmalloc: large alloc 78512136192 bytes == 0x6de346000 @
> >
> ****************************************************************************************************
> > Unigram tokens 16 types 18
> > === 2/5 Calculating and sorting adjusted counts ===
> > Chain sizes: 1:216 2:107979354931
> > tcmalloc: large alloc 107979358208 bytes == 0x192a648000 @
> > terminate called after throwing an instance of
> > 'lm::builder::BadDiscountException'
> > what():
> >
> /home/lanes/mosesdecoder.multisource.git/lm/builder/adjust_counts.cc:53
> > in void
> > lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
> > lm::builder::DiscountConfig&) threw BadDiscountException because
> > `s.n[j] == 0'.
> > Could not calculate Kneser-Ney discounts for 1-grams with adjusted
> > count 4 because we didn't observe any 1-grams with adjusted count 3;
> > Is this small or artificial data?
> > Try deduplicating the input. To override this error for e.g. a
> > class-based model, rerun with --discount_fallback
> > Aborted (core dumped)
> >
> >
> >
> > On Tue, Jan 12, 2016 at 5:40 PM, Kenneth Heafield <moses@kheafield.com
> > <mailto:moses@kheafield.com>> wrote:
> >
> > Pushed the fix from kenlm master in October to Moses master.
> >
> > On 01/12/2016 10:34 PM, Lane Schwartz wrote:
> > > Steps to reproduce this error:
> > >
> > > $ ~/mosesdecoder.git/bin/lmplz -o 2 <<< "that is what happens
> ? cssd
> > > has nothing more or voldemort or pastries in prague ."
> > > === 1/5 Counting and sorting n-grams ===
> > > Reading /tmp/sh-thd-107574999377 (deleted)
> > >
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> > > tcmalloc: large alloc 29442056192 bytes == 0x2ae2000 @
> > > tcmalloc: large alloc 78512136192 bytes == 0x6df1b4000 @
> > >
> ****************************************************************************************************
> > > Unigram tokens 16 types 18
> > > === 2/5 Calculating and sorting adjusted counts ===
> > > Chain sizes: 1:216 2:107979354931
> > > tcmalloc: large alloc 107979358208 bytes == 0x192b4b6000 @
> > > lmplz: ./util/fixed_array.hh:104: T&
> > > util::FixedArray<T>::operator[](std::size_t) [with T =
> > > lm::NGramStream<lm::builder::BuildingPayload>; std::size_t =
> long
> > > unsigned int]: Assertion `i < size()' failed.
> > >
> > >
> > >
> > >
> > > On Wed, Sep 30, 2015 at 11:41 AM, Kenneth Heafield <
> moses@kheafield.com <mailto:moses@kheafield.com>
> > > <mailto:moses@kheafield.com <mailto:moses@kheafield.com>>> wrote:
> > >
> > > That's bad. Would you mind sending me privately a minimal
> > example of
> > > the data that reproduces the problem?
> > >
> > > Kenneth
> > >
> > > On 09/30/2015 04:29 PM, Alex Martinez wrote:
> > > > Hello,
> > > > today I've pulled moses code and recompiled and some
> > experiments (EMS)
> > > > that were already working are failing on the LM training
> > step with the
> > > > following error:
> > > >
> > > > Executing: /opt/moses/bin/lmplz --text
> > > > /home/alexmc/devel/toydata/process/lm/nc=pos.factored.1
> > --order 5
> > > --arpa
> > > > /home/alexmc/devel/toydata/process/lm/nc=pos.lm.1
> > --discount_fallback
> > > > === 1/5 Counting and sorting n-grams ===
> > > > Reading /mnt/a62/devel/toydata/process/lm/nc=pos.factored.1
> > > >
> > >
> >
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> > > > tcmalloc: large alloc 4753956864 bytes == 0x1f7c000 @
> > > > tcmalloc: large alloc 22185107456 bytes == 0x11d536000 @
> > > >
> > >
> >
> ****************************************************************************************************
> > > > Unigram tokens 2433135 types 47
> > > > === 2/5 Calculating and sorting adjusted counts ===
> > > > Chain sizes: 1:564 2:2630656000 3:4932480000 4:7891967488
> > > 5:11509120000
> > > > tcmalloc: large alloc 11509121024 bytes == 0x1f7c000 @
> > > > tcmalloc: large alloc 2630656000 bytes == 0x2aff70000 @
> > > > tcmalloc: large alloc 4932485120 bytes == 0x34cc3a000 @
> > > > tcmalloc: large alloc 7891968000 bytes == 0x64933c000 @
> > > > lmplz: ./util/fixed_array.hh:104: T&
> > > > util::FixedArray<T>::operator[](std::size_t) [with T =
> > > > lm::NGramStream<lm::builder::BuildingPayload>; std::size_t =
> > long
> > > > unsigned int]: Assertion `i < size()' failed.
> > > >
> > > > I'm runing a Linux server with Ubuntu 15.04
> > > >
> > > > Any help will be appreciated
> > > >
> > > > Alex Mart?nez
> > > >
> > > >
> > > > _______________________________________________
> > > > Moses-support mailing list
> > > > Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> > <mailto:Moses-support@mit.edu <mailto:Moses-support@mit.edu>>
> > > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > > >
> > > _______________________________________________
> > > Moses-support mailing list
> > > Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> > <mailto:Moses-support@mit.edu <mailto:Moses-support@mit.edu>>
> > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > >
> > >
> > >
> > >
> > > --
> > > When a place gets crowded enough to require ID's, social collapse
> > is not
> > > far away. It is time to go elsewhere. The best thing about space
> > travel
> > > is that it made it possible to go elsewhere.
> > > -- R.A. Heinlein, "Time Enough For Love"
> >
> >
> >
> >
> > --
> > When a place gets crowded enough to require ID's, social collapse is not
> > far away. It is time to go elsewhere. The best thing about space travel
> > is that it made it possible to go elsewhere.
> > -- R.A. Heinlein, "Time Enough For Love"
>



--
When a place gets crowded enough to require ID's, social collapse is not
far away. It is time to go elsewhere. The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160113/649536e7/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 111, Issue 40
**********************************************

Tweet