Moses-support Digest, Vol 141, Issue 13

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Issues running MGiza on AWS machine (Tom Hoar)
2. Re: Issues running MGiza on AWS machine (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Wed, 1 Aug 2018 08:09:33 +0700
From: Tom Hoar <tahoar@slate.rocks>
Subject: Re: [Moses-support] Issues running MGiza on AWS machine
To: moses-support@mit.edu
Message-ID: <d0ef4b55-a9f8-94aa-6217-9b07770dd178@slate.rocks>
Content-Type: text/plain; charset="utf-8"

Hi James,

Since train-model.perl fails at step 4 fails with the MGIZA binaries you
build on your AWS machine, but succeeds when you copy MGIZA binaries
that you built on your local Ubuntu 16.04 machine, do the build logs
show a missing dependency?

My next question, why don't you just use the binaries that work? It
seems like the AWS machine's Ubuntu distro is missing dependencies and
the MGIZA++ build failed. Check those build logs.

If you want to troubleshoot deeper, you need to backtrack from step 4.
The train-model.perl step 4 uses the output from step 3, i.e. the word
alignment file. Check if that word alignment file is corrupted.

Then check step 3, its inputs are the GIZA alignment files output in
step 2. This step uses the symal binary executable. Make sure you're
using the Moses version of symal, not the one in the MGIZA library.
http://article.gmane.org/gmane.comp.nlp.moses.user/11544
http://moses-support.mit.narkive.com/KpKC2TQn/which-symal

Backtracking to step 2, log lines with the following text messages
should cause the mgiza executable to terminate but it doesn't. The
parallel forks in train-model.perl mask the failure, processing
continues and you experience ambiguous failures downstream.

ERROR: A SOURCE or TARGET sentence has a zero-length sentence.
ERROR! DUPLICATED ENTRY
WARNING: The following sentence pair has source/target sentence
length ration more than

There are rarely errors in Step 1, but if you are experiencing a compile
error on AWS, those MGIZA binaries in step 1 could be the cause.

Also, the C++ binary executables are not the only things that change
when you use use the alternate build. If you also copied the
merge_alignment.py, this could be a problem in train-model.perl step 2.
Make sure the AWS build has this in the right place and that it runs on
the AWS distro's Python interpreter.

Tom



On 7/31/2018 11:01 PM, moses-support-request@mit.edu wrote:
> Date: Tue, 31 Jul 2018 11:41:01 +0100
> From: James Baker<james.d.baker@gmail.com>
> Subject: [Moses-support] Issues running MGiza on AWS machine
> To:moses-support@mit.edu
> Message-ID:
> <CAOa=L2woDwDETeeq7RAs0LxmnbAA_Q7qH=kF8aPAH7ivBk0qgA@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi,
>
> I'm having some peculiar issues with MGiza++. Using MGiza and Moses, I've
> successfully built some translation models on my Ubuntu 16.04 desktop
> machine. I'd now like to do the same thing, but on a machine hosted in AWS.
>
> I'm using the same operating system, and as far as I can tell all my
> versions are identical. The build of MGiza++ runs fine, reports no errors,
> and produces output the same as on my desktop machine. However, when I try
> to build the models, I get a whole load of errors and the resultant models
> are empty (64 bytes for the reordering model, 0 bytes for the translation
> model - the language model builds fine).
>
> The first "errors" I can see in the log seem to occur on stage 4 of the
> Moses training script (train-model.perl):
>
> (4) generate lexical translation table 0-0 @ Tue Jul 31 10:22:58 UTC 2018
> (/opt/model-builder/training/data.ru
> ,/opt/model-builder/training/data.en,/opt/model-builder/training/model/lex)
> !Argument "anna" isn't numeric in numeric ge (>=) at
> /opt/model-builder/mosesdecoder/scripts/training/LexicalTranslationModel.pm
> line 112, <A> line 1.
> Use of uninitialized value $ei in numeric ge (>=) at
> /opt/model-builder/mosesdecoder/scripts/training/LexicalTranslationModel.pm
> line 112, <A> line 1.
> Use of uninitialized value $ei in hash element at
> /opt/model-builder/mosesdecoder/scripts/training/LexicalTranslationModel.pm
> line 118, <A> line 1.
> Use of uninitialized value $ei in array element at
> /opt/model-builder/mosesdecoder/scripts/training/LexicalTranslationModel.pm
> line 121, <A> line 1.
> Use of uninitialized value $ei in array element at
> /opt/model-builder/mosesdecoder/scripts/training/LexicalTranslationModel.pm
> line 123, <A> line 1.
> ...
>
> There are a large number of errors of that nature, and following those
> errors there are additional errors but I suspect these are caused by the
> fact that this stage is failing.
>
> It's possible that there are earlier problems, but I'm not really sure what
> to be looking for in the logs (for instance - there are some lines warning
> about alignments in Model2 being 0 - is that an issue?).
>
> If I replace the MGiza binaries built on the AWS machine with the binaries
> built on my desktop, it runs fine - so I know it's an issue with MGiza and
> presumably something to do with my build. The commands I'm running to build
> and install are as follows
>
> git clonehttps://github.com/moses-smt/mgiza.git
> cd mgiza/mgizapp
> cmake .
> make
> make install
> cp bin/* ../../mosesdecoder/bin
> cp scripts/merge_alignment.py ../../mosesdecoder/bin
>
> As I mentioned previously, these commands work fine on my desktop machine
> which should be a very similar (if not identical) set up.
>
> Does anyone have any ideas as to what might be causing the problem (or,
> more importantly, what I can do to fix it)?
>
> Thanks in advance,
> James

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20180731/28ce8361/attachment-0001.html

------------------------------

Message: 2
Date: Wed, 1 Aug 2018 12:03:47 +1000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Issues running MGiza on AWS machine
To: James Baker <james.d.baker@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbhRZeg1BKLxNvNMb1fgCjaxZdZau0YFN1fDuhztW7siHQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

it's difficult to tell but I would say the mgiza executables isn't the
problem. It's probably to do with running out of disk space or memory.

the snt2coooc executable in mgiza uses a lot of memory so may have been
killed by the OS. The phrase table creation requires a lot of disk space to
sort intermediate files.

I would monitor those 2 things

Hieu Hoang
http://statmt.org/hieu

On 31 July 2018 at 20:41, James Baker <james.d.baker@gmail.com> wrote:

> Hi,
>
> I'm having some peculiar issues with MGiza++. Using MGiza and Moses, I've
> successfully built some translation models on my Ubuntu 16.04 desktop
> machine. I'd now like to do the same thing, but on a machine hosted in AWS.
>
> I'm using the same operating system, and as far as I can tell all my
> versions are identical. The build of MGiza++ runs fine, reports no errors,
> and produces output the same as on my desktop machine. However, when I try
> to build the models, I get a whole load of errors and the resultant models
> are empty (64 bytes for the reordering model, 0 bytes for the translation
> model - the language model builds fine).
>
> The first "errors" I can see in the log seem to occur on stage 4 of the
> Moses training script (train-model.perl):
>
> (4) generate lexical translation table 0-0 @ Tue Jul 31 10:22:58 UTC
> 2018
> (/opt/model-builder/training/data.ru,/opt/model-builder/
> training/data.en,/opt/model-builder/training/model/lex)
> !Argument "anna" isn't numeric in numeric ge (>=) at /opt/model-builder/
> mosesdecoder/scripts/training/LexicalTranslationModel.pm line 112, <A>
> line 1.
> Use of uninitialized value $ei in numeric ge (>=) at /opt/model-builder/
> mosesdecoder/scripts/training/LexicalTranslationModel.pm line 112, <A>
> line 1.
> Use of uninitialized value $ei in hash element at /opt/model-builder/
> mosesdecoder/scripts/training/LexicalTranslationModel.pm line 118, <A>
> line 1.
> Use of uninitialized value $ei in array element at /opt/model-builder/
> mosesdecoder/scripts/training/LexicalTranslationModel.pm line 121, <A>
> line 1.
> Use of uninitialized value $ei in array element at /opt/model-builder/
> mosesdecoder/scripts/training/LexicalTranslationModel.pm line 123, <A>
> line 1.
> ...
>
> There are a large number of errors of that nature, and following those
> errors there are additional errors but I suspect these are caused by the
> fact that this stage is failing.
>
> It's possible that there are earlier problems, but I'm not really sure
> what to be looking for in the logs (for instance - there are some lines
> warning about alignments in Model2 being 0 - is that an issue?).
>
> If I replace the MGiza binaries built on the AWS machine with the binaries
> built on my desktop, it runs fine - so I know it's an issue with MGiza and
> presumably something to do with my build. The commands I'm running to build
> and install are as follows
>
> git clone https://github.com/moses-smt/mgiza.git
> cd mgiza/mgizapp
> cmake .
> make
> make install
> cp bin/* ../../mosesdecoder/bin
> cp scripts/merge_alignment.py ../../mosesdecoder/bin
>
> As I mentioned previously, these commands work fine on my desktop machine
> which should be a very similar (if not identical) set up.
>
> Does anyone have any ideas as to what might be causing the problem (or,
> more importantly, what I can do to fix it)?
>
> Thanks in advance,
> James
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20180731/50221943/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 141, Issue 13
**********************************************

Tweet