Moses-support Digest, Vol 113, Issue 61

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: KenLM loading error (Lane Schwartz)
2. Re: KenLM loading error (Lane Schwartz)


----------------------------------------------------------------------

Message: 1
Date: Tue, 22 Mar 2016 13:30:26 -0500
From: Lane Schwartz <dowobeha@gmail.com>
Subject: Re: [Moses-support] KenLM loading error
To: Kenneth Heafield <moses@kheafield.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZkZHLdpDZy-Z_YeNmtC8waa_67Wp-+EbjEpPtPUF_F=PQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Here's the backtrace. It's weird. I get this error under certain
conditions, but not under others.

Files are attached.



On Tue, Mar 22, 2016 at 4:20 AM, Kenneth Heafield <moses@kheafield.com>
wrote:

> Hi,
>
> I don't see a problem with this in principle. The error means
> that the
> calling code provided an out-of-range word id. Can I have a backtrace
> after compiling with debug?
>
> Kenneth
>
> On 03/21/2016 11:34 PM, Lane Schwartz wrote:
> > Hi,
> >
> > I have a tiny LM that is giving me some problems. The LM wasn't trained
> > with any of the usual toolkits, so it may violate some assumptions that
> > the KenLM code in Moses makes.
> >
> > I'm able to successfully query the LM using bin/query, but when I use
> > the LM within Moses I get the following error:
> >
> > Reading dd_lm0.arpa
> >
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> >
> ****************************************************************************************************
> > mosesserver: lm/model.cc:250: lm::FullScoreReturn
> > lm::ngram::detail::GenericModel<Search,
> > VocabularyT>::ScoreExceptBackoff(const WordIndex*, const WordIndex*,
> > lm::WordIndex, lm::ngram::State&) const [with Search =
> > lm::ngram::detail::HashedSearch<lm::ngram::BackoffValue>;
> > VocabularyT = lm::ngram::ProbingVocabulary; lm::WordIndex = unsigned
> > int]: Assertion `new_word < vocab_.Bound()' failed.
> >
> >
> > And here's the LM:
> >
> > \data\
> > ngram 1=22
> > ngram 2=21
> >
> >
> > \1-grams:
> > 0.-99
> > 0:-99
> > 0</s>-99
> > -99<s>-99
> > -99<unk>
> > -4.30102999566a-99
> > -5.00005affect
> > 0certain-99
> > -4.30102999566have-99
> > -4.30102999566impact-99
> > 0is-99
> > -4.30102999566negative-99
> > -5.00005negatively
> > 0new-99
> > -4.30102999566on-99
> > 0one-99
> > 0provisions-99
> > 0the-99
> > 0these-99
> > 0thing-99
> > 0turnout-99
> > 0will-99
> >
> >
> > \2-grams:
> > 0. </s>
> > 0: these
> > 0<s> one
> > -4.30102999566a negative
> > -5.00005affect negatively
> > 0certain :
> > -4.30102999566have a
> > -4.30102999566impact on
> > 0is certain
> > -4.30102999566negative impact
> > -5.00005negatively the
> > 0new provisions
> > -4.30102999566on the
> > 0one thing
> > 0provisions will
> > 0the turnout
> > 0these new
> > 0thing is
> > 0turnout .
> > -5.00005will affect
> > -4.30102999566will have
> > \end\
> >
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
When a place gets crowded enough to require ID's, social collapse is not
far away. It is time to go elsewhere. The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160322/03e37777/attachment-0001.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: extra-lm.fr.log
Type: application/octet-stream
Size: 6715 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20160322/03e37777/attachment-0002.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dd_lm0.arpa
Type: application/octet-stream
Size: 756 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20160322/03e37777/attachment-0003.obj

------------------------------

Message: 2
Date: Tue, 22 Mar 2016 14:14:25 -0500
From: Lane Schwartz <dowobeha@gmail.com>
Subject: Re: [Moses-support] KenLM loading error
To: Kenneth Heafield <moses@kheafield.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZk_kEPW5odKiHbtn8Q-d9P7CW5iSj0RRpNn97yHOCJO-Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

The error is happening when I use my ReloadingLM class, which inherits from
LanguageModelKen. The problem only appears when I start with one model and
then reload a different model.

When the model is reloaded, the code calls LanguageModelKen::LoadModel
again, resetting m_ngram.


On Tue, Mar 22, 2016 at 1:30 PM, Lane Schwartz <dowobeha@gmail.com> wrote:

> Here's the backtrace. It's weird. I get this error under certain
> conditions, but not under others.
>
> Files are attached.
>
>
>
> On Tue, Mar 22, 2016 at 4:20 AM, Kenneth Heafield <moses@kheafield.com>
> wrote:
>
>> Hi,
>>
>> I don't see a problem with this in principle. The error means
>> that the
>> calling code provided an out-of-range word id. Can I have a backtrace
>> after compiling with debug?
>>
>> Kenneth
>>
>> On 03/21/2016 11:34 PM, Lane Schwartz wrote:
>> > Hi,
>> >
>> > I have a tiny LM that is giving me some problems. The LM wasn't trained
>> > with any of the usual toolkits, so it may violate some assumptions that
>> > the KenLM code in Moses makes.
>> >
>> > I'm able to successfully query the LM using bin/query, but when I use
>> > the LM within Moses I get the following error:
>> >
>> > Reading dd_lm0.arpa
>> >
>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>> >
>> ****************************************************************************************************
>> > mosesserver: lm/model.cc:250: lm::FullScoreReturn
>> > lm::ngram::detail::GenericModel<Search,
>> > VocabularyT>::ScoreExceptBackoff(const WordIndex*, const WordIndex*,
>> > lm::WordIndex, lm::ngram::State&) const [with Search =
>> > lm::ngram::detail::HashedSearch<lm::ngram::BackoffValue>;
>> > VocabularyT = lm::ngram::ProbingVocabulary; lm::WordIndex = unsigned
>> > int]: Assertion `new_word < vocab_.Bound()' failed.
>> >
>> >
>> > And here's the LM:
>> >
>> > \data\
>> > ngram 1=22
>> > ngram 2=21
>> >
>> >
>> > \1-grams:
>> > 0.-99
>> > 0:-99
>> > 0</s>-99
>> > -99<s>-99
>> > -99<unk>
>> > -4.30102999566a-99
>> > -5.00005affect
>> > 0certain-99
>> > -4.30102999566have-99
>> > -4.30102999566impact-99
>> > 0is-99
>> > -4.30102999566negative-99
>> > -5.00005negatively
>> > 0new-99
>> > -4.30102999566on-99
>> > 0one-99
>> > 0provisions-99
>> > 0the-99
>> > 0these-99
>> > 0thing-99
>> > 0turnout-99
>> > 0will-99
>> >
>> >
>> > \2-grams:
>> > 0. </s>
>> > 0: these
>> > 0<s> one
>> > -4.30102999566a negative
>> > -5.00005affect negatively
>> > 0certain :
>> > -4.30102999566have a
>> > -4.30102999566impact on
>> > 0is certain
>> > -4.30102999566negative impact
>> > -5.00005negatively the
>> > 0new provisions
>> > -4.30102999566on the
>> > 0one thing
>> > 0provisions will
>> > 0the turnout
>> > 0these new
>> > 0thing is
>> > 0turnout .
>> > -5.00005will affect
>> > -4.30102999566will have
>> > \end\
>> >
>> >
>> >
>> > _______________________________________________
>> > Moses-support mailing list
>> > Moses-support@mit.edu
>> > http://mailman.mit.edu/mailman/listinfo/moses-support
>> >
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
>
> --
> When a place gets crowded enough to require ID's, social collapse is not
> far away. It is time to go elsewhere. The best thing about space travel
> is that it made it possible to go elsewhere.
> -- R.A. Heinlein, "Time Enough For Love"
>



--
When a place gets crowded enough to require ID's, social collapse is not
far away. It is time to go elsewhere. The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160322/758ec9c5/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 113, Issue 61
**********************************************

Tweet