Moses-support Digest, Vol 113, Issue 57

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: ERROR: Lexical reordering scoring failed at,
/home/berry/mosesdecoder/scripts/training/train-model.perl line
1922. (Tom Hoar)
2. reordering issue (Vincent Nguyen)
3. Is there any way to do continues training ? (dai xin)
4. Re: Is there any way to do continues training ? (Marwa N Refaie)


----------------------------------------------------------------------

Message: 1
Date: Mon, 21 Mar 2016 10:08:00 +0700
From: Tom Hoar <tahoar@precisiontranslationtools.com>
Subject: Re: [Moses-support] ERROR: Lexical reordering scoring failed
at, /home/berry/mosesdecoder/scripts/training/train-model.perl line
1922.
To: moses-support@mit.edu
Message-ID: <56EF6590.1030307@precisiontranslationtools.com>
Content-Type: text/plain; charset="windows-1252"

Birendra,

There are several possible causes for your errors.

First possibility: Retrace the log output from train-model.perl to Step
2. Look for lines with either of these two messages. Both of these
messages means GIZA++ encountered a fatal error, but it does not throw
an error or stop. To fix this, make sure you run Moses'
clean-corpus-n.perl script on the parallel corpus:

"WARNING: The following sentence pair has source/target sentence
length ration more than"
"ERROR: A SOURCE or TARGET sentence has a zero-length sentence."

Second possibility: You ran out of hard drive space during step 5. This
is the step that creates the extract.0000000.gz and
extract.0000000.inv.gz files you report below. There's no easy way to
test for this possibility. Just rebuild your system with a bigger hard
drive and start from scratch.

Third possibility: Your corpus contains characters that Moses reserves
for its use. These include the five reserved XML characters &><'" plus
three more |][ To fix this, simply escape these characters in your
training corpus.

Violations in any/all of the above do not cause a terminal failure in
train-model.perl. Instead, train-model.perl continues processing your
entire corpus and creates an empty gzip output file. Downstream steps
(steps 4, 5 & 6) in train-model.perl then log errors because they can
not read data from the empty gzip files. These downstream errors
eventually cause a terminal failure that you experienced.

Tom



On 3/20/2016 11:00 PM, moses-support-request@mit.edu wrote:
> Date: Sun, 20 Mar 2016 02:05:45 +0530
> From: BIRENDRA CHAUHAN SINGH<birendrachauhan3@gmail.com>
> Subject: [Moses-support] ERROR: Lexical reordering scoring failed at
> /home/berry/mosesdecoder/scripts/training/train-model.perl line 1922.
> To:moses-support@mit.edu
> Message-ID:
> <CAKogDtzeYisqx_9HORapzBX4QFwet0xN1yuj2-M_OBEcksiXgg@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> on running below command for training the baseline model.I have attached
> the output result file below.It shows all the error.
>
> nohup nice ~/mosesdecoder/scripts/training/train-model.perl -root-dir train \
> -corpus ~/corpus/news-commentary-v8.fr-en.clean \
> -f fr -e en -alignment grow-diag-final-and -reordering msd-bidirectional-fe \
> -lm 0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8 \
> -external-bin-dir ~/mosesdecoder/tools >& training.out &
>
> Am getting erros like :
>
> gzip: gzip: /home/berry/working/train/model/tmp.25178/extract.0000000.inv.gz:
> No such file or directory
> /home/berry/working/train/model/tmp.25178/extract.0000000.gz: No such
> file or directory
>
> ERROR: Scoring of phrases failed at
> /home/berry/mosesdecoder/scripts/training/train-model.perl line 1784.
>
> /home/berry/mosesdecoder/scripts/../bin/consolidate: error while
> loading shared libraries: libboost_serialization.so.1.59.0: cannot
> open shared object file: No such file or directory
>
> ERROR: Lexical reordering scoring failed at
> /home/berry/mosesdecoder/scripts/training/train-model.perl line 1922.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:http://mailman.mit.edu/mailman/private/moses-support/attachments/20160319/23941c51/attachment-0001.html
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: training.out
> Type: application/octet-stream
> Size: 16013 bytes
> Desc: not available
> Url :http://mailman.mit.edu/mailman/private/moses-support/attachments/20160319/23941c51/attachment-0001.obj
>
> ------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160320/20aff096/attachment-0001.html

------------------------------

Message: 2
Date: Mon, 21 Mar 2016 09:38:41 +0100
From: Vincent Nguyen <vnguyen@neuf.fr>
Subject: [Moses-support] reordering issue
To: moses-support <moses-support@mit.edu>
Message-ID: <56EFB311.5050200@neuf.fr>
Content-Type: text/plain; charset="utf-8"

Hi,

I have been fighting with some reordering issues.
I have tried both LM interpolation and OSM but with no luck.

Here is an example
Source English :
Canada remains very active within the Working Group, and our law
enforcement officials also participate in the Working Group?s informal
law enforcement meetings on a biannual basis.

French translated :
Le Canada reste tr?s active au sein du Groupe de travail sur
l'application de la loi, et nos fonctionnaires participent aussi au
*Groupe de travail informel de r?unions bisannuelles sur l'application
de la loi.

*instead of "r?unions informelles bisannuelles sur l'application de la loi"

This is just an example but the same kind of inversion is very frequent.

Any clue ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160321/6707ff25/attachment-0001.html

------------------------------

Message: 3
Date: Mon, 21 Mar 2016 14:20:36 +0100
From: dai xin <wingsuestc@gmail.com>
Subject: [Moses-support] Is there any way to do continues training ?
To: moses-support@mit.edu
Message-ID:
<CADEDxC8X95pnvR6cTwA3JHBgRN8O0yTXJP94FekVycNhHNnjuA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

is there anyone has experience on doing continues training on Moses?

I am now developing a machine translation system in my company based on
Moses. For now, I have done model training on company's corpus and used the
model to do translation. We have more continues strings which can be used
to train our model and also language expert evaluating and doing correction
to the result. We want to do the correction and add more strings to our
model.

As far as I know, I can add new strings to the previous corpus and do the
training for another time. It is, however, very time consuming and maybe
not the best way for industrial situation. So I am looking for some ideas
for the continues training.

Best regards,

Many thanks in advance!

Xin Dai
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160321/abc483e2/attachment-0001.html

------------------------------

Message: 4
Date: Mon, 21 Mar 2016 15:25:15 +0200
From: Marwa N Refaie <basmallah@hotmail.com>
Subject: Re: [Moses-support] Is there any way to do continues training
?
To: dai xin <wingsuestc@gmail.com>
Cc: moses-support@mit.edu
Message-ID: <BLU436-SMTP54016113AE05D40D854546BA8F0@phx.gbl>
Content-Type: text/plain; charset="utf-8"

Incremental training. ..
Check in Moses website

Marwa N Refaie



On 21 Mar 2016 15:23, at 15:23, dai xin <wingsuestc@gmail.com> wrote:
>Hi,
>
>is there anyone has experience on doing continues training on Moses?
>
>I am now developing a machine translation system in my company based on
>Moses. For now, I have done model training on company's corpus and used
>the
>model to do translation. We have more continues strings which can be
>used
>to train our model and also language expert evaluating and doing
>correction
>to the result. We want to do the correction and add more strings to our
>model.
>
>As far as I know, I can add new strings to the previous corpus and do
>the
>training for another time. It is, however, very time consuming and
>maybe
>not the best way for industrial situation. So I am looking for some
>ideas
>for the continues training.
>
>Best regards,
>
>Many thanks in advance!
>
>Xin Dai
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Moses-support mailing list
>Moses-support@mit.edu
>http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160321/5c510a23/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 113, Issue 57
**********************************************

Tweet