Moses-support Digest, Vol 142, Issue 1

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Issues running MGiza on AWS machine (James Baker)
2. Re: Issues running MGiza on AWS machine (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Wed, 1 Aug 2018 08:38:44 +0100
From: James Baker <james.d.baker@gmail.com>
Subject: Re: [Moses-support] Issues running MGiza on AWS machine
To: hieuhoang@gmail.com
Cc: moses-support@mit.edu
Message-ID:
<CAOa=L2xd3CyqfBO9x=Yjzpev5=tjir1TQFWBND3T9EGVFjOMxg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Thanks Hieu,

I'll give that a go this morning and keep an eye on the disk space and RAM,
although I would be surprised if that was the problem (I've got <3GB of
training data, 64GB of RAM, and 100GB of disk space). It also wouldn't
explain why binaries built on a different machine work, but binaries built
on the same machine don't.

Any other ideas for things I should be checking?

Cheers,
James

On Wed, 1 Aug 2018 at 03:03, Hieu Hoang <hieuhoang@gmail.com> wrote:

> it's difficult to tell but I would say the mgiza executables isn't the
> problem. It's probably to do with running out of disk space or memory.
>
> the snt2coooc executable in mgiza uses a lot of memory so may have been
> killed by the OS. The phrase table creation requires a lot of disk space to
> sort intermediate files.
>
> I would monitor those 2 things
>
> Hieu Hoang
> http://statmt.org/hieu
>
> On 31 July 2018 at 20:41, James Baker <james.d.baker@gmail.com> wrote:
>
>> Hi,
>>
>> I'm having some peculiar issues with MGiza++. Using MGiza and Moses, I've
>> successfully built some translation models on my Ubuntu 16.04 desktop
>> machine. I'd now like to do the same thing, but on a machine hosted in AWS.
>>
>> I'm using the same operating system, and as far as I can tell all my
>> versions are identical. The build of MGiza++ runs fine, reports no errors,
>> and produces output the same as on my desktop machine. However, when I try
>> to build the models, I get a whole load of errors and the resultant models
>> are empty (64 bytes for the reordering model, 0 bytes for the translation
>> model - the language model builds fine).
>>
>> The first "errors" I can see in the log seem to occur on stage 4 of the
>> Moses training script (train-model.perl):
>>
>> (4) generate lexical translation table 0-0 @ Tue Jul 31 10:22:58 UTC
>> 2018
>> (/opt/model-builder/training/data.ru
>> ,/opt/model-builder/training/data.en,/opt/model-builder/training/model/lex)
>> !Argument "anna" isn't numeric in numeric ge (>=) at
>> /opt/model-builder/mosesdecoder/scripts/training/LexicalTranslationModel.pm
>> line 112, <A> line 1.
>> Use of uninitialized value $ei in numeric ge (>=) at
>> /opt/model-builder/mosesdecoder/scripts/training/LexicalTranslationModel.pm
>> line 112, <A> line 1.
>> Use of uninitialized value $ei in hash element at
>> /opt/model-builder/mosesdecoder/scripts/training/LexicalTranslationModel.pm
>> line 118, <A> line 1.
>> Use of uninitialized value $ei in array element at
>> /opt/model-builder/mosesdecoder/scripts/training/LexicalTranslationModel.pm
>> line 121, <A> line 1.
>> Use of uninitialized value $ei in array element at
>> /opt/model-builder/mosesdecoder/scripts/training/LexicalTranslationModel.pm
>> line 123, <A> line 1.
>> ...
>>
>> There are a large number of errors of that nature, and following those
>> errors there are additional errors but I suspect these are caused by the
>> fact that this stage is failing.
>>
>> It's possible that there are earlier problems, but I'm not really sure
>> what to be looking for in the logs (for instance - there are some lines
>> warning about alignments in Model2 being 0 - is that an issue?).
>>
>> If I replace the MGiza binaries built on the AWS machine with the
>> binaries built on my desktop, it runs fine - so I know it's an issue with
>> MGiza and presumably something to do with my build. The commands I'm
>> running to build and install are as follows
>>
>> git clone https://github.com/moses-smt/mgiza.git
>> cd mgiza/mgizapp
>> cmake .
>> make
>> make install
>> cp bin/* ../../mosesdecoder/bin
>> cp scripts/merge_alignment.py ../../mosesdecoder/bin
>>
>> As I mentioned previously, these commands work fine on my desktop machine
>> which should be a very similar (if not identical) set up.
>>
>> Does anyone have any ideas as to what might be causing the problem (or,
>> more importantly, what I can do to fix it)?
>>
>> Thanks in advance,
>> James
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20180801/7480b443/attachment-0001.html

------------------------------

Message: 2
Date: Wed, 1 Aug 2018 17:56:56 +1000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Issues running MGiza on AWS machine
To: James Baker <james.d.baker@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbiYwNkq=axpuF6MUz_bmXLbDLyV6EUfhmXT-KL0mGNWyQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

redirect stdout and stderr into a file and grep for 'error'

that usually turns up something

Hieu Hoang
http://statmt.org/hieu

On 1 August 2018 at 17:38, James Baker <james.d.baker@gmail.com> wrote:

> Thanks Hieu,
>
> I'll give that a go this morning and keep an eye on the disk space and
> RAM, although I would be surprised if that was the problem (I've got <3GB
> of training data, 64GB of RAM, and 100GB of disk space). It also wouldn't
> explain why binaries built on a different machine work, but binaries built
> on the same machine don't.
>
> Any other ideas for things I should be checking?
>
> Cheers,
> James
>
> On Wed, 1 Aug 2018 at 03:03, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
>> it's difficult to tell but I would say the mgiza executables isn't the
>> problem. It's probably to do with running out of disk space or memory.
>>
>> the snt2coooc executable in mgiza uses a lot of memory so may have been
>> killed by the OS. The phrase table creation requires a lot of disk space to
>> sort intermediate files.
>>
>> I would monitor those 2 things
>>
>> Hieu Hoang
>> http://statmt.org/hieu
>>
>> On 31 July 2018 at 20:41, James Baker <james.d.baker@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm having some peculiar issues with MGiza++. Using MGiza and Moses,
>>> I've successfully built some translation models on my Ubuntu 16.04 desktop
>>> machine. I'd now like to do the same thing, but on a machine hosted in AWS.
>>>
>>> I'm using the same operating system, and as far as I can tell all my
>>> versions are identical. The build of MGiza++ runs fine, reports no errors,
>>> and produces output the same as on my desktop machine. However, when I try
>>> to build the models, I get a whole load of errors and the resultant models
>>> are empty (64 bytes for the reordering model, 0 bytes for the translation
>>> model - the language model builds fine).
>>>
>>> The first "errors" I can see in the log seem to occur on stage 4 of the
>>> Moses training script (train-model.perl):
>>>
>>> (4) generate lexical translation table 0-0 @ Tue Jul 31 10:22:58 UTC
>>> 2018
>>> (/opt/model-builder/training/data.ru,/opt/model-builder/
>>> training/data.en,/opt/model-builder/training/model/lex)
>>> !Argument "anna" isn't numeric in numeric ge (>=) at
>>> /opt/model-builder/mosesdecoder/scripts/training/LexicalTranslationModel.pm
>>> line 112, <A> line 1.
>>> Use of uninitialized value $ei in numeric ge (>=) at
>>> /opt/model-builder/mosesdecoder/scripts/training/LexicalTranslationModel.pm
>>> line 112, <A> line 1.
>>> Use of uninitialized value $ei in hash element at /opt/model-builder/
>>> mosesdecoder/scripts/training/LexicalTranslationModel.pm line 118, <A>
>>> line 1.
>>> Use of uninitialized value $ei in array element at /opt/model-builder/
>>> mosesdecoder/scripts/training/LexicalTranslationModel.pm line 121, <A>
>>> line 1.
>>> Use of uninitialized value $ei in array element at /opt/model-builder/
>>> mosesdecoder/scripts/training/LexicalTranslationModel.pm line 123, <A>
>>> line 1.
>>> ...
>>>
>>> There are a large number of errors of that nature, and following those
>>> errors there are additional errors but I suspect these are caused by the
>>> fact that this stage is failing.
>>>
>>> It's possible that there are earlier problems, but I'm not really sure
>>> what to be looking for in the logs (for instance - there are some lines
>>> warning about alignments in Model2 being 0 - is that an issue?).
>>>
>>> If I replace the MGiza binaries built on the AWS machine with the
>>> binaries built on my desktop, it runs fine - so I know it's an issue with
>>> MGiza and presumably something to do with my build. The commands I'm
>>> running to build and install are as follows
>>>
>>> git clone https://github.com/moses-smt/mgiza.git
>>> cd mgiza/mgizapp
>>> cmake .
>>> make
>>> make install
>>> cp bin/* ../../mosesdecoder/bin
>>> cp scripts/merge_alignment.py ../../mosesdecoder/bin
>>>
>>> As I mentioned previously, these commands work fine on my desktop
>>> machine which should be a very similar (if not identical) set up.
>>>
>>> Does anyone have any ideas as to what might be causing the problem (or,
>>> more importantly, what I can do to fix it)?
>>>
>>> Thanks in advance,
>>> James
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20180801/7915058e/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 142, Issue 1
*********************************************

Tweet