Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Lexical reordering fails with zlib (Varden Wang)
2. Re: Lexical reordering fails with zlib (Matthias Huck)
3. Re: Lexical reordering fails with zlib (Matthias Huck)
4. Re: how to train with berkeley word aligner (Tom Hoar)
----------------------------------------------------------------------
Message: 1
Date: Thu, 17 Dec 2015 09:18:11 -0800
From: Varden Wang <vardenw@uw.edu>
Subject: Re: [Moses-support] Lexical reordering fails with zlib
To: moses-support@mit.edu
Message-ID:
<CANf2V=tMeFapeEgWgy1Um5VOLdGgd_31Z317C1f1FjSTVLP98g@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
I seem to be having a very very similar issue. I have the exact same
lib package as Guy (but I upgraded from lib 1.2.3.4). I'm using the
commit SHA e211d752f6bc680094520482f190d0f805405c6c of the
mosesdecoder. The funny thing is that I trained on the very same setup
on different data sets without encountering this problem.
My error:
Executing: /usr/local/google/home/varden/MOSES/mosesdecoder/scripts/../bin/lexical-reordering-score
/usr/local/google/home/varden/MOSES/FRENCH_moses_HELP_CONTENT_v1/train/model/extract.o.sorted.gz
0.5 /usr/local/google/home/varden/MOSES/FRENCH_moses_HELP_CONTENT_v1/train/model/reordering-table.
--model "wbe msd wbe-msd-bidirectional-fe"
Lexical Reordering Scorer
scores lexical reordering models of several types (hierarchical,
phrase-based and word-based-extraction
terminate called after throwing an instance of 'util::GZException'
what(): zlib encountered invalid distances set code -3
ERROR: Execution of:
/usr/local/google/home/varden/MOSES/mosesdecoder/scripts/../bin/lexical-reordering-score
/usr/local/google/home/varden/MOSES/FRENCH_moses_HELP_CONTENT_v1/train/model/extract.o.sorted.gz
0.5 /usr/local/google/home/varden/MOSES/FRENCH_moses_HELP_CONTENT_v1/train/model/reordering-table.
--model "wbe msd wbe-msd-bidirectional-fe"
died with signal 6, with coredump
Thanks,
Varden
On Mon, Dec 7, 2015 at 9:01 AM, <moses-support-request@mit.edu> wrote:
> Send Moses-support mailing list submissions to
> moses-support@mit.edu
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.mit.edu/mailman/listinfo/moses-support
> or, via email, send a message with subject or body 'help' to
> moses-support-request@mit.edu
>
> You can reach the person managing the list at
> moses-support-owner@mit.edu
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Moses-support digest..."
>
>
> Today's Topics:
>
> 1. Lexical reordering fails with zlib (Guy)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Mon, 7 Dec 2015 06:14:47 +0000 (UTC)
> From: Guy <guyarg@yahoo.com>
> Subject: [Moses-support] Lexical reordering fails with zlib
> To: moses-support@mit.edu
> Message-ID: <loom.20151207T071423-100@post.gmane.org>
> Content-Type: text/plain; charset=us-ascii
>
> Hello everyone,
>
> I've just recently started to work with Moses and managed to build a couple
> of models without problems... until now.
>
> I was training a new system and I got this error when executing
> lexical-reordering-score:
>
> ../mosesdecoder/scripts/../bin/lexical-reordering-score
> /local/scratch/train/model/extract.o.sorted.gz 0.5
> /local/scratch/train/model/reordering-table. --model "wbe msd
> wbe-msd-bidirectional-fe"
> Lexical Reordering Scorer
> scores lexical reordering models of several types (hierarchical,
> phrase-based and word-based-extraction
> terminate called after throwing an instance of 'util::GZException'
> what(): util/read_compressed.cc:163 in virtual std::size_t
> util::{anonymous}::GZip::Read(void*, std::size_t, util::ReadCompressed&)
> threw GZException'.
> zlib encountered invalid distances set code -3
> Aborted
>
> I found an old post
> (http://permalink.gmane.org/gmane.comp.nlp.moses.user/10151) saying this was
> due to an apparent bug in zlib 1.2.3.4 on Ubuntu 12.04 and that upgrading to
> zlib 1.2.8 solves the problem. However, I already have zlib 1.2.8 (but on
> Ubuntu 14.04) and I still get this error. In case it helps, package name is
> zlib1g:amd64 version 1:1.2.8.dfsg-1ubuntu).
>
> It's a bit strange that I didn't stumble upon this problem when I trained
> previous systems.
>
> Any ideas on what to do?
>
> Thank you very much,
> Guy
>
>
>
>
> ------------------------------
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> End of Moses-support Digest, Vol 110, Issue 16
> **********************************************
------------------------------
Message: 2
Date: Thu, 17 Dec 2015 17:44:58 +0000
From: Matthias Huck <mhuck@inf.ed.ac.uk>
Subject: Re: [Moses-support] Lexical reordering fails with zlib
To: Varden Wang <vardenw@uw.edu>
Cc: moses-support@mit.edu
Message-ID: <1450374298.22442.21.camel@portedgar>
Content-Type: text/plain; charset="UTF-8"
Hi,
It's a problem that apparently occurs very rarely, and as Guy mentioned,
we were so far assuming that it's caused by a zlib bug.
However, the zlib bug was (to my knowledge) fixed in zlib v1.2.8.
This seems to be the bug fix:
https://github.com/madler/zlib/commit/51370f365607fe14a6a7a1a27b3bd29d788f5e5b
I've only encountered the issue once (and I'm training systems
frequently). When I came across it, I executed the same command with a
Moses compile on a different machine that was running an older version
of OpenSuse, rather than Ubuntu 12.04. The problem did not exist on the
old system.
My guess is that it really is a zlib bug, but it would be worrying if
switching to zlib v1.2.8 doesn't resolve it.
Cheers,
Matthias
On Thu, 2015-12-17 at 09:18 -0800, Varden Wang wrote:
> I seem to be having a very very similar issue. I have the exact same
> lib package as Guy (but I upgraded from lib 1.2.3.4). I'm using the
> commit SHA e211d752f6bc680094520482f190d0f805405c6c of the
> mosesdecoder. The funny thing is that I trained on the very same setup
> on different data sets without encountering this problem.
>
> My error:
>
> Executing: /usr/local/google/home/varden/MOSES/mosesdecoder/scripts/../bin/lexical-reordering-score
> /usr/local/google/home/varden/MOSES/FRENCH_moses_HELP_CONTENT_v1/train/model/extract.o.sorted.gz
> 0.5 /usr/local/google/home/varden/MOSES/FRENCH_moses_HELP_CONTENT_v1/train/model/reordering-table.
> --model "wbe msd wbe-msd-bidirectional-fe"
>
> Lexical Reordering Scorer
>
> scores lexical reordering models of several types (hierarchical,
> phrase-based and word-based-extraction
>
> terminate called after throwing an instance of 'util::GZException'
>
> what(): zlib encountered invalid distances set code -3
>
> ERROR: Execution of:
> /usr/local/google/home/varden/MOSES/mosesdecoder/scripts/../bin/lexical-reordering-score
> /usr/local/google/home/varden/MOSES/FRENCH_moses_HELP_CONTENT_v1/train/model/extract.o.sorted.gz
> 0.5 /usr/local/google/home/varden/MOSES/FRENCH_moses_HELP_CONTENT_v1/train/model/reordering-table.
> --model "wbe msd wbe-msd-bidirectional-fe"
>
> died with signal 6, with coredump
>
> Thanks,
>
> Varden
>
> On Mon, Dec 7, 2015 at 9:01 AM, <moses-support-request@mit.edu> wrote:
> > Send Moses-support mailing list submissions to
> > moses-support@mit.edu
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> > or, via email, send a message with subject or body 'help' to
> > moses-support-request@mit.edu
> >
> > You can reach the person managing the list at
> > moses-support-owner@mit.edu
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Moses-support digest..."
> >
> >
> > Today's Topics:
> >
> > 1. Lexical reordering fails with zlib (Guy)
> >
> >
> > ----------------------------------------------------------------------
> >
> > Message: 1
> > Date: Mon, 7 Dec 2015 06:14:47 +0000 (UTC)
> > From: Guy <guyarg@yahoo.com>
> > Subject: [Moses-support] Lexical reordering fails with zlib
> > To: moses-support@mit.edu
> > Message-ID: <loom.20151207T071423-100@post.gmane.org>
> > Content-Type: text/plain; charset=us-ascii
> >
> > Hello everyone,
> >
> > I've just recently started to work with Moses and managed to build a couple
> > of models without problems... until now.
> >
> > I was training a new system and I got this error when executing
> > lexical-reordering-score:
> >
> > ../mosesdecoder/scripts/../bin/lexical-reordering-score
> > /local/scratch/train/model/extract.o.sorted.gz 0.5
> > /local/scratch/train/model/reordering-table. --model "wbe msd
> > wbe-msd-bidirectional-fe"
> > Lexical Reordering Scorer
> > scores lexical reordering models of several types (hierarchical,
> > phrase-based and word-based-extraction
> > terminate called after throwing an instance of 'util::GZException'
> > what(): util/read_compressed.cc:163 in virtual std::size_t
> > util::{anonymous}::GZip::Read(void*, std::size_t, util::ReadCompressed&)
> > threw GZException'.
> > zlib encountered invalid distances set code -3
> > Aborted
> >
> > I found an old post
> > (http://permalink.gmane.org/gmane.comp.nlp.moses.user/10151) saying this was
> > due to an apparent bug in zlib 1.2.3.4 on Ubuntu 12.04 and that upgrading to
> > zlib 1.2.8 solves the problem. However, I already have zlib 1.2.8 (but on
> > Ubuntu 14.04) and I still get this error. In case it helps, package name is
> > zlib1g:amd64 version 1:1.2.8.dfsg-1ubuntu).
> >
> > It's a bit strange that I didn't stumble upon this problem when I trained
> > previous systems.
> >
> > Any ideas on what to do?
> >
> > Thank you very much,
> > Guy
> >
> >
> >
> >
> > ------------------------------
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> >
> > End of Moses-support Digest, Vol 110, Issue 16
> > **********************************************
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
------------------------------
Message: 3
Date: Thu, 17 Dec 2015 22:56:06 +0000
From: Matthias Huck <mhuck@inf.ed.ac.uk>
Subject: Re: [Moses-support] Lexical reordering fails with zlib
To: Varden Wang <vardenw@uw.edu>
Cc: moses-support@mit.edu
Message-ID: <1450392966.1957.39.camel@inf.ed.ac.uk>
Content-Type: text/plain; charset="UTF-8"
Hi,
As an addendum:
You can try a manual workaround. Run gunzip on extract.o.sorted.gz and
do lexical-reordering-score on the resulting plain text file.
It might be inconvenient but would hopefully solve the issue.
Cheers,
Matthias
On Thu, 2015-12-17 at 17:44 +0000, Matthias Huck wrote:
> Hi,
>
> It's a problem that apparently occurs very rarely, and as Guy mentioned,
> we were so far assuming that it's caused by a zlib bug.
>
> However, the zlib bug was (to my knowledge) fixed in zlib v1.2.8.
> This seems to be the bug fix:
> https://github.com/madler/zlib/commit/51370f365607fe14a6a7a1a27b3bd29d788f5e5b
>
> I've only encountered the issue once (and I'm training systems
> frequently). When I came across it, I executed the same command with a
> Moses compile on a different machine that was running an older version
> of OpenSuse, rather than Ubuntu 12.04. The problem did not exist on the
> old system.
>
> My guess is that it really is a zlib bug, but it would be worrying if
> switching to zlib v1.2.8 doesn't resolve it.
>
> Cheers,
> Matthias
>
>
> On Thu, 2015-12-17 at 09:18 -0800, Varden Wang wrote:
> > I seem to be having a very very similar issue. I have the exact same
> > lib package as Guy (but I upgraded from lib 1.2.3.4). I'm using the
> > commit SHA e211d752f6bc680094520482f190d0f805405c6c of the
> > mosesdecoder. The funny thing is that I trained on the very same setup
> > on different data sets without encountering this problem.
> >
> > My error:
> >
> > Executing: /usr/local/google/home/varden/MOSES/mosesdecoder/scripts/../bin/lexical-reordering-score
> > /usr/local/google/home/varden/MOSES/FRENCH_moses_HELP_CONTENT_v1/train/model/extract.o.sorted.gz
> > 0.5 /usr/local/google/home/varden/MOSES/FRENCH_moses_HELP_CONTENT_v1/train/model/reordering-table.
> > --model "wbe msd wbe-msd-bidirectional-fe"
> >
> > Lexical Reordering Scorer
> >
> > scores lexical reordering models of several types (hierarchical,
> > phrase-based and word-based-extraction
> >
> > terminate called after throwing an instance of 'util::GZException'
> >
> > what(): zlib encountered invalid distances set code -3
> >
> > ERROR: Execution of:
> > /usr/local/google/home/varden/MOSES/mosesdecoder/scripts/../bin/lexical-reordering-score
> > /usr/local/google/home/varden/MOSES/FRENCH_moses_HELP_CONTENT_v1/train/model/extract.o.sorted.gz
> > 0.5 /usr/local/google/home/varden/MOSES/FRENCH_moses_HELP_CONTENT_v1/train/model/reordering-table.
> > --model "wbe msd wbe-msd-bidirectional-fe"
> >
> > died with signal 6, with coredump
> >
> > Thanks,
> >
> > Varden
> >
> > On Mon, Dec 7, 2015 at 9:01 AM, <moses-support-request@mit.edu> wrote:
> > > Send Moses-support mailing list submissions to
> > > moses-support@mit.edu
> > >
> > > To subscribe or unsubscribe via the World Wide Web, visit
> > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > > or, via email, send a message with subject or body 'help' to
> > > moses-support-request@mit.edu
> > >
> > > You can reach the person managing the list at
> > > moses-support-owner@mit.edu
> > >
> > > When replying, please edit your Subject line so it is more specific
> > > than "Re: Contents of Moses-support digest..."
> > >
> > >
> > > Today's Topics:
> > >
> > > 1. Lexical reordering fails with zlib (Guy)
> > >
> > >
> > > ----------------------------------------------------------------------
> > >
> > > Message: 1
> > > Date: Mon, 7 Dec 2015 06:14:47 +0000 (UTC)
> > > From: Guy <guyarg@yahoo.com>
> > > Subject: [Moses-support] Lexical reordering fails with zlib
> > > To: moses-support@mit.edu
> > > Message-ID: <loom.20151207T071423-100@post.gmane.org>
> > > Content-Type: text/plain; charset=us-ascii
> > >
> > > Hello everyone,
> > >
> > > I've just recently started to work with Moses and managed to build a couple
> > > of models without problems... until now.
> > >
> > > I was training a new system and I got this error when executing
> > > lexical-reordering-score:
> > >
> > > ../mosesdecoder/scripts/../bin/lexical-reordering-score
> > > /local/scratch/train/model/extract.o.sorted.gz 0.5
> > > /local/scratch/train/model/reordering-table. --model "wbe msd
> > > wbe-msd-bidirectional-fe"
> > > Lexical Reordering Scorer
> > > scores lexical reordering models of several types (hierarchical,
> > > phrase-based and word-based-extraction
> > > terminate called after throwing an instance of 'util::GZException'
> > > what(): util/read_compressed.cc:163 in virtual std::size_t
> > > util::{anonymous}::GZip::Read(void*, std::size_t, util::ReadCompressed&)
> > > threw GZException'.
> > > zlib encountered invalid distances set code -3
> > > Aborted
> > >
> > > I found an old post
> > > (http://permalink.gmane.org/gmane.comp.nlp.moses.user/10151) saying this was
> > > due to an apparent bug in zlib 1.2.3.4 on Ubuntu 12.04 and that upgrading to
> > > zlib 1.2.8 solves the problem. However, I already have zlib 1.2.8 (but on
> > > Ubuntu 14.04) and I still get this error. In case it helps, package name is
> > > zlib1g:amd64 version 1:1.2.8.dfsg-1ubuntu).
> > >
> > > It's a bit strange that I didn't stumble upon this problem when I trained
> > > previous systems.
> > >
> > > Any ideas on what to do?
> > >
> > > Thank you very much,
> > > Guy
> > >
> > >
> > >
> > >
> > > ------------------------------
> > >
> > > _______________________________________________
> > > Moses-support mailing list
> > > Moses-support@mit.edu
> > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > >
> > >
> > > End of Moses-support Digest, Vol 110, Issue 16
> > > **********************************************
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
>
>
>
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
------------------------------
Message: 4
Date: Fri, 18 Dec 2015 08:20:18 +0700
From: Tom Hoar <tahoar@precisiontranslationtools.com>
Subject: Re: [Moses-support] how to train with berkeley word aligner
To: moses-support@mit.edu
Message-ID: <56735F52.9090601@precisiontranslationtools.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Once you create the alignment file from Berkeley, you pick-up
train-model.perl training at step 4. So, add the --do-steps=4-9 argument
to your command line. You might also want to use the --alignment-file or
--alignment-stem arguments to point directly to your Berlekey alignment
file.
On 12/18/2015 12:01 AM, moses-support-request@mit.edu wrote:
> Date: Wed, 16 Dec 2015 22:37:11 +0900
> From: Andrew<ravenyj@hotmail.com>
> Subject: [Moses-support] how to train with berkeley word aligner
> To:"moses-support@mit.edu" <moses-support@mit.edu>
> Message-ID:<BAY181-W632B592198A2DC8E62D5D4B2EF0@phx.gbl>
> Content-Type: text/plain; charset="iso-2022-jp"
>
> I'm replicating the steps described in Baseline System page,and am about to run the following command,except I want to run it with pre-compiled berkeley word aligner rather than mgiza or giza++ (since their installations have been troublesome)Should I fix the command below or code in somewhere?In any case, how should it be fixed?
>
> mkdir ~/working
> cd ~/working
> nohup nice ~/mosesdecoder/scripts/training/train-model.perl -root-dir train \
> -corpus ~/corpus/news-commentary-v8.fr-en.clean \
> -f fr -e en -alignment grow-diag-final-and -reordering msd-bidirectional-fe \
> -lm 0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8 \
> -external-bin-dir ~/mosesdecoder/tools >& training.out &
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 110, Issue 31
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 110, Issue 31"
Post a Comment