Moses-support Digest, Vol 127, Issue 26

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Segfault from moses while tuning (Per Starb?ck )
2. Re: Segfault from moses while tuning (Hieu Hoang)
3. Re: Segfault from moses while tuning (Per Starb?ck )
4. Re: Segfault from moses while tuning (Hieu Hoang)
5. Re: Segfault from moses while tuning (Kenneth Heafield)


----------------------------------------------------------------------

Message: 1
Date: Wed, 17 May 2017 13:25:59 +0200
From: starback@stp.lingfil.uu.se (Per Starb?ck )
Subject: Re: [Moses-support] Segfault from moses while tuning
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <sa5y3tvwwqg.fsf@numerus.lingfil.uu.se>
Content-Type: text/plain

Hieu Hoang writes:

> thanks. It may not be the phrase-based/hiero mixup as I had suggested.
>
> Maybe the system ran out of RAM.

I don't think so. It doesn't grow much at all when running, and this is
a rather small test just done to show me the problem.

I think now I should have included not only the backtrace from moses but
also the end of its output before crashing:
======================================================================
[...]
Initializing LexicalReordering..
line=Distortion
FeatureFunction: Distortion0 start: 13 end: 13
line=KENLM lazyken=0 name=LM0 factor=0 path=.../lm/news-commentrary-v8.de-en.blm.en order=3
FeatureFunction: LM0 start: 14 end: 14
Loading UnknownWordPenalty0
Loading WordPenalty0
Loading PhrasePenalty0
Loading LexicalReordering0
Loading table into memory...done.
Loading Distortion0
Loading LM0
Loading TranslationModel0
Start loading text phrase table. Moses format : [6.706] seconds
Reading .../mert-work/filtered/phrase-table.0-0.1.1.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
======================================================================

> Before you run tuning, you should also binarize the phrase-table and
> lexicalised reordering model, ...

She followed the tutorial at
http://www.statmt.org/moses/?n=Moses.Baseline (but with much less
training data) when preparing the non-working example to show me.
If doing that results in a segfault there must be something wrong
somewhere. Isn't the backtrace useful?


------------------------------

Message: 2
Date: Wed, 17 May 2017 12:32:59 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Segfault from moses while tuning
To: Per Starb?ck <starback@stp.lingfil.uu.se>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbg9F5JuNYEJJoG_ZxOe81J0Vx-GPVMH6oLKsHCjxWoXTw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

How much RAM is used is dependent on the phrase-table & reordering model
size, not the test set size.

Since it blows up at during loading of the phrase-table, I would think it
ran out of ram. I would run the script again and use top to monitor
carefully how much RAM is being used just before it crashes.

The only other thing I can think of is there's some gunk in the
phrase-table, but that's unlikely if you are using the downloaded
phrase-table as the data has been cleaned with the usual scripts.

* Looking for MT/NLP opportunities *
Hieu Hoang
http://moses-smt.org/


On 17 May 2017 at 12:25, Per Starb?ck <starback@stp.lingfil.uu.se> wrote:

> Hieu Hoang writes:
>
> > thanks. It may not be the phrase-based/hiero mixup as I had suggested.
> >
> > Maybe the system ran out of RAM.
>
> I don't think so. It doesn't grow much at all when running, and this is
> a rather small test just done to show me the problem.
>
> I think now I should have included not only the backtrace from moses but
> also the end of its output before crashing:
> ======================================================================
> [...]
> Initializing LexicalReordering..
> line=Distortion
> FeatureFunction: Distortion0 start: 13 end: 13
> line=KENLM lazyken=0 name=LM0 factor=0 path=.../lm/news-commentrary-v8.de-en.blm.en
> order=3
> FeatureFunction: LM0 start: 14 end: 14
> Loading UnknownWordPenalty0
> Loading WordPenalty0
> Loading PhrasePenalty0
> Loading LexicalReordering0
> Loading table into memory...done.
> Loading Distortion0
> Loading LM0
> Loading TranslationModel0
> Start loading text phrase table. Moses format : [6.706] seconds
> Reading .../mert-work/filtered/phrase-table.0-0.1.1.gz
> ----5---10---15---20---25---30---35---40---45---50---55---
> 60---65---70---75---80---85---90---95--100
> ======================================================================
>
> > Before you run tuning, you should also binarize the phrase-table and
> > lexicalised reordering model, ...
>
> She followed the tutorial at
> http://www.statmt.org/moses/?n=Moses.Baseline (but with much less
> training data) when preparing the non-working example to show me.
> If doing that results in a segfault there must be something wrong
> somewhere. Isn't the backtrace useful?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170517/cc784a96/attachment-0001.html

------------------------------

Message: 3
Date: Wed, 17 May 2017 14:11:03 +0200
From: starback@stp.lingfil.uu.se (Per Starb?ck )
Subject: Re: [Moses-support] Segfault from moses while tuning
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <sa5o9urwunc.fsf@numerus.lingfil.uu.se>
Content-Type: text/plain

Hieu Hoang writes:

> The only other thing I can think of is there's some gunk in the phrase-table,

Thanks, that seems to be it! Now I've experimented with including and
excluding various parts of the phrase-table file, and
it seems like lines like

Konzert ||| [would] ||| 0.2 1 0.2 0.0833333 ||| 0-0 ||| 5 5 1 ||| |||

with brackets are problematic. I removed all lines containing "[" and
then I get output and no segfault. (There are still lines containing
"]", but that doesn't seem to be a problem.)

Is that a bogus line? That is should I know focus on how that
phrase-table file was created, and not on what happens when I try to use
it?


------------------------------

Message: 4
Date: Wed, 17 May 2017 13:26:15 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Segfault from moses while tuning
To: Per Starb?ck <starback@stp.lingfil.uu.se>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbhyUEZF8mYDxMBfnxFYpzv1vEKTKJU2OpYTrHGZeKDCLQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

ah yes, I think the phrase-table was created in the version when [ and ]
weren't reserved characters but now they are. So you have to use the
executables in that version (v3?), or escape them

* Looking for MT/NLP opportunities *
Hieu Hoang
http://moses-smt.org/


On 17 May 2017 at 13:11, Per Starb?ck <starback@stp.lingfil.uu.se> wrote:

> Hieu Hoang writes:
>
> > The only other thing I can think of is there's some gunk in the
> phrase-table,
>
> Thanks, that seems to be it! Now I've experimented with including and
> excluding various parts of the phrase-table file, and
> it seems like lines like
>
> Konzert ||| [would] ||| 0.2 1 0.2 0.0833333 ||| 0-0 ||| 5 5 1 ||| |||
>
> with brackets are problematic. I removed all lines containing "[" and
> then I get output and no segfault. (There are still lines containing
> "]", but that doesn't seem to be a problem.)
>
> Is that a bogus line? That is should I know focus on how that
> phrase-table file was created, and not on what happens when I try to use
> it?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170517/a990f51d/attachment-0001.html

------------------------------

Message: 5
Date: Wed, 17 May 2017 13:29:19 +0100
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] Segfault from moses while tuning
To: moses-support@mit.edu
Message-ID: <2ff5e65e-3098-6efd-ad39-1199c158d2c0@kheafield.com>
Content-Type: text/plain; charset=windows-1252; format=flowed

Can we have a better error message than "Segmentation fault" when that
happens?

On 05/17/2017 01:26 PM, Hieu Hoang wrote:
> ah yes, I think the phrase-table was created in the version when [ and ]
> weren't reserved characters but now they are. So you have to use the
> executables in that version (v3?), or escape them
>
> * Looking for MT/NLP opportunities *
> Hieu Hoang
> http://moses-smt.org/
>
>
> On 17 May 2017 at 13:11, Per Starb?ck <starback@stp.lingfil.uu.se
> <mailto:starback@stp.lingfil.uu.se>> wrote:
>
> Hieu Hoang writes:
>
> > The only other thing I can think of is there's some gunk in the phrase-table,
>
> Thanks, that seems to be it! Now I've experimented with including and
> excluding various parts of the phrase-table file, and
> it seems like lines like
>
> Konzert ||| [would] ||| 0.2 1 0.2 0.0833333 ||| 0-0 ||| 5 5 1 ||| |||
>
> with brackets are problematic. I removed all lines containing "[" and
> then I get output and no segfault. (There are still lines containing
> "]", but that doesn't seem to be a problem.)
>
> Is that a bogus line? That is should I know focus on how that
> phrase-table file was created, and not on what happens when I try to use
> it?
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 127, Issue 26
**********************************************

Tweet