Moses-support Digest, Vol 110, Issue 4

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Exit code: 127 ERROR: Can't generate symmetrized
alignment file (Philipp Koehn)
2. Re: Exit code: 127 ERROR: Can't generate symmetrized
alignment file (Barry Haddow)
3. Re: System requiremnts for Moses (Philipp Koehn)


----------------------------------------------------------------------

Message: 1
Date: Wed, 2 Dec 2015 16:22:58 -0500
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] Exit code: 127 ERROR: Can't generate
symmetrized alignment file
To: "Read, James C" <jcread@essex.ac.uk>
Cc: Moses Support <moses-support@mit.edu>
Message-ID:
<CAAFADDDBE+D4EUN9fzkbF4_+MtWxfTS2BrAunn=_GmLArLS+SA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

it looks like that symal was not compiled - it should be in $MOSES/bin

Can you check what went wrong during compilation?

-phi

On Wed, Dec 2, 2015 at 10:21 AM, Read, James C <jcread@essex.ac.uk> wrote:

> nohup nice
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/train-model.perl
> -root-dir phrase_table -corpus
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/europarl-v7.it-en.1-0010.00001000
> -f it -e en -alignment grow-diag-final-and -reordering msd-bidirectional-fe
> -lm 0:3:/media/bigdata/jcread/llv/lm:8 -external-bin-dir
> /media/bigdata/jcread/3rd_party_software/bin >& training.out &
>
>
>
> Runs well for a while and then bombs out with following output and Error
> 127
>
>
>
> (3) generate word alignment @ Wed Dec 2 01:56:06 GMT 2015
> Combining forward and inverted alignment from files:
>
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/phrase_table/giza.it-en/it-en.A3.final.{bz2,gz}
>
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/phrase_table/giza.en-it/en-it.A3.final.{bz2,gz}
> Executing: mkdir -p
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/phrase_table/model
> Executing:
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/
> giza2bal.pl -d "gzip -cd
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/phrase_table/giza.en-it/en-it.A3.final.gz"
> -i "gzip -cd
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/phrase_table/giza.it-en/it-en.A3.final.gz"
> |/media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/../bin/symal
> -alignment="grow" -diagonal="yes" -final="yes" -both="yes" >
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/phrase_table/model/aligned.grow-diag-final-and
> sh: 1:
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/../bin/symal:
> not found
> Exit code: 127
> ERROR: Can't generate symmetrized alignment file
>
>
> It seems this problem with the script has been encountered before:
>
>
> http://comments.gmane.org/gmane.comp.nlp.moses.user/10489
>
>
> I'm not sure I understand the accepted solution.
>
>
> "Use absolute paths to all the scripts, and make sure your parallel files
> have the same names but the extension"
>
>
> The command I issued uses only absolute paths. Is this referring to
> modifications in the training script itself?
>
>
> James
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151202/533ffb52/attachment-0001.html

------------------------------

Message: 2
Date: Wed, 2 Dec 2015 21:35:12 +0000
From: Barry Haddow <bhaddow@inf.ed.ac.uk>
Subject: Re: [Moses-support] Exit code: 127 ERROR: Can't generate
symmetrized alignment file
To: "Read, James C" <jcread@essex.ac.uk>, Moses Support
<moses-support@mit.edu>
Message-ID: <565F6410.8060202@inf.ed.ac.uk>
Content-Type: text/plain; charset="windows-1252"

Hi James

> sh: 1:
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/../bin/symal:
> not found


The script expects to be able to navigate the file system and find the
binaries. If you've built Moses with a "--prefix" option then it won't
be able to find the binaries. If you are running the script from the
source tree, then make sure the binaries are in the directory "bin" .

In answer to your other question, varying the pre-processing pipeline is
definitely possible (how do you think Moses deals with Chinese?) There
are certain data formatting requirements, as Philipp pointed out, but
other than that you have a lot of freedom.

cheers - Barry


On 02/12/15 15:21, Read, James C wrote:
>
> nohup nice
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/train-model.perl
> -root-dir phrase_table -corpus
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/europarl-v7.it-en.1-0010.00001000
> -f it -e en -alignment grow-diag-final-and -reordering
> msd-bidirectional-fe -lm 0:3:/media/bigdata/jcread/llv/lm:8
> -external-bin-dir /media/bigdata/jcread/3rd_party_software/bin >&
> training.out &
>
>
>
> Runs well for a while and then bombs out with following output and
> Error 127
>
>
>
> (3) generate word alignment @ Wed Dec 2 01:56:06 GMT 2015
> Combining forward and inverted alignment from files:
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/phrase_table/giza.it-en/it-en.A3.final.{bz2,gz}
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/phrase_table/giza.en-it/en-it.A3.final.{bz2,gz}
> Executing: mkdir -p
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/phrase_table/model
> Executing:
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/training/giza2bal.pl
> -d "gzip -cd
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/phrase_table/giza.en-it/en-it.A3.final.gz"
> -i "gzip -cd
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/phrase_table/giza.it-en/it-en.A3.final.gz"
> |/media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/../bin/symal
> -alignment="grow" -diagonal="yes" -final="yes" -both="yes" >
> /media/bigdata/jcread/llv/data/europarlv7/prealigned/tokenized_truecased_cleaned/1-0010/00001000/phrase_table/model/aligned.grow-diag-final-and
> sh: 1:
> /media/bigdata/jcread/3rd_party_software/mosesdecoder/scripts/../bin/symal:
> not found
> Exit code: 127
> ERROR: Can't generate symmetrized alignment file
>
>
> It seems this problem with the script has been encountered before:
>
>
> http://comments.gmane.org/gmane.comp.nlp.moses.user/10489
>
>
> I'm not sure I understand the accepted solution.
>
>
> "Use absolute paths to all the scripts, and make sure your parallel
> files have the same names but the extension"
>
>
> The command I issued uses only absolute paths. Is this referring to
> modifications in the training script itself?
>
>
> James
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151202/9e367640/attachment-0001.html
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151202/9e367640/attachment-0001.pl

------------------------------

Message: 3
Date: Wed, 2 Dec 2015 16:42:38 -0500
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] System requiremnts for Moses
To: "Hegde, Sujay" <Sujay.Hegde@xerox.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDAQaqtCCY+ZEfTCoTybPWXgaoC0Uf7ihnCxcX00sDSSjA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

the machine you have is certainly sufficient even for large models.

If you are running two language pairs in parallel and run into RAM
problems, you may want to look into ways to compress the model files
(phrase table, reordering table, language model) using either more
efficient data structures (e.g., various KENLM options), or pruning the
models.

-phi


On Tue, Dec 1, 2015 at 5:08 AM, Hegde, Sujay <Sujay.Hegde@xerox.com> wrote:

> Dear Moses Admin,
>
>
>
> We are using Moses decoder for commercial environment.
>
>
>
> We have 132GB RAM, 1TB disk and quadcore * Virtual
> Machine* with CentOs OS.
>
>
>
> We have 2 language pairs installed, and when running both
> the models together the Translation hangs(Takes a LONG time).
>
> It is fine when we run only one language model.
>
>
>
> Is there any Specific System requirements needed for moses?
>
> Please let me know
>
>
>
> Thanks and Regards,
>
> Sujay,
>
> Xerox Business Services, Bangalore, India
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151202/89ba0eec/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 110, Issue 4
*********************************************

Tweet