Home » Uncategories » Moses-support Digest, Vol 89, Issue 24

Moses-support Digest, Vol 89, Issue 24

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: manual: 3.2.1 Train an unfactored model (Peter Kleiweg)
2. Re: train-model log file? (Massinissa Ahmim)
3. Re: manual: 3.2.1 Train an unfactored model (Tom Hoar)
4. Problem while running mert-moses.pl (Allan Jie)


----------------------------------------------------------------------

Message: 1
Date: Tue, 11 Mar 2014 19:46:32 +0100 (CET)
From: Peter Kleiweg <p.c.j.kleiweg@rug.nl>
Subject: Re: [Moses-support] manual: 3.2.1 Train an unfactored model
To: moses-support@mit.edu
Message-ID: <alpine.DEB.2.00.1403111944110.17849@pebbe>
Content-Type: TEXT/PLAIN; charset=US-ASCII

Massinissa Ahmim schreef op de 11e dag van de lentemaand van het jaar 2014:

> Hi Peter,
>
> There is obviously an issue with your training data, have you used
> the proj-syndicate corpora? if so please paste here your training command
> I'd be interested to see that.

I have data downloaded from here:
http://www.statmt.org/moses/download/factored-corpus.tgz


This is the training command:

train-model.perl \
--root-dir unfactored \
--corpus factored-corpus/proj-syndicate \
--f de --e en \
--lm 0:3:/net/aistaff/kleiweg/moses/test/factored-corpus/surface.lm:0 \
--external-bin-dir /net/aps/64/opt/moses/giza-pp/bin \
> training.out \
2> training.err &


> Regards
>
> M
>
>
> 2014-03-11 19:25 GMT+01:00 Peter Kleiweg <p.c.j.kleiweg@rug.nl>:
>
> >
> > Hi,
> >
> > >From the manual: http://www.statmt.org/moses/manual/manual.pdf
> >
> > I have run the example from part 3.2.1, page 64, to create
> > unfactored/model/moses.ini
> >
> > At the bottom of page 67 is this example:
> >
> > moses -f unfactored/model/moses.ini < in -d 0.2
> >
> > The file 'in' contains:
> >
> > menschen beschreibt putin .
> >
> > (The -d 0.2 causes an error in all examples, so I leave that out)
> >
> >
> > I get an error:
> >
> > Exception: moses/Word.cpp:112 in void
> > Moses::Word::CreateFromString(Moses::FactorDirection, const
> > std::vector<long unsigned int>&, const StringPiece&, bool) threw
> > StrayFactorException because `fit'.
> > You have configured 1 factors but the word !|!|. contains factor
> > delimiter | too many times.
> >
> > What is wrong, and how do I fix this?
> >
> > All other examples in section 3.2 work OK.
> >
> >
> > I have moses built with both IRSTLM and SRILM.




--
Peter Kleiweg
http://pkleiweg.home.xs4all.nl/


------------------------------

Message: 2
Date: Tue, 11 Mar 2014 19:46:35 +0100
From: Massinissa Ahmim <massinissa.ahmim@linguacustodia.com>
Subject: Re: [Moses-support] train-model log file?
To: Viktor Pless <viktor.pless@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CANN0mWY9QKEB8NaUVXcA8bV8od8X+CdMp5wnw2KqT430YTb7yw@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi Victor

Yes, all you have to do is to add &> training.out after to your training
command

Regards

Massinissa


2014-03-11 19:39 GMT+01:00 Viktor Pless <viktor.pless@gmail.com>:

> Hi, does the train-model.perl script (or more specifically, mgiza) create
> a log file?
>
> thanks,
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--

[image: Description : Description : lingua_custodia_final full logo]

*The Translation Trustee*

*1, Place Charles de Gaulle*

*78180 Montigny-le-Bretonneux*

*Tel : +33 1 30 44 04 23 Mobile : +33 7 61 44 40 84*

*Email :* *massinissa.ahmim@linguacustodia.com
<massinissa.ahmim@linguacustodia.com>*

*Website :* *www.linguacustodia.com <http://www.linguacustodia.com/> -
www.thetranslationtrustee.com <http://www.thetranslationtrustee.com>*

? Pensez ? l'environnement, n'imprimez ce courriel que si n?cessaire.

Please do not print this email unless it is absolutely necessary. Spread
environmental awareness.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140311/3f8bc96f/attachment-0001.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 4421 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140311/3f8bc96f/attachment-0001.jpg

------------------------------

Message: 3
Date: Wed, 12 Mar 2014 09:23:23 +0700
From: Tom Hoar <tahoar@precisiontranslationtools.com>
Subject: Re: [Moses-support] manual: 3.2.1 Train an unfactored model
To: moses-support@mit.edu
Message-ID: <531FC51B.1060900@precisiontranslationtools.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Shouldn't the moses option come before the "<" redirection like this?

moses -d 0.2 -f unfactored/model/moses.ini < in


On 03/12/2014 01:46 AM, Peter Kleiweg wrote:
> moses -f unfactored/model/moses.ini < in -d 0.2



------------------------------

Message: 4
Date: Wed, 12 Mar 2014 13:33:39 +0800
From: Allan Jie <allanmcgrady@gmail.com>
Subject: [Moses-support] Problem while running mert-moses.pl
To: moses-support@MIT.EDU
Message-ID:
<CANm261Mv9bjjc_Y5whsd9Zj7BvspunFsiFvwgH9mviETvxHDyg@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Dear all,

I try to run tuning using this command.
nohup nice /home/wei_lu/tools/mosesdecoder/scripts/training/mert-moses.pl \
/home/wei_lu/experiment/data/dev/dev2010.lowercased.indexed.zh
/home/wei_lu/experiment/data/dev/dev2010.lowercased.en \
/home/wei_lu/tools/mosesdecoder/bin/moses
/home/wei_lu/pku_seg/model/moses.ini --working-dir
/home/wei_lu/pku_seg/mert-work-index --mertdir
/home/wei_lu/tools/mosesdecoder/bin/ -decoder-flags="-threads 4"

But the result I got fail, here is the log.

*Using SCRIPTS_ROOTDIR: /home/wei_lu/tools/mosesdecoder/scripts*
*filtering the phrase tables... Wed Mar 12 12:18:43 SGT 2014*
*exec:
/home/wei_lu/tools/mosesdecoder/scripts/training/filter-model-given-input.pl
<http://filter-model-given-input.pl> ./filtered
/home/wei_lu/pku_seg/model/moses.ini
/home/wei_lu/experiment/data/dev/dev2010.lowercased.indexed.zh*
*Executing:
/home/wei_lu/tools/mosesdecoder/scripts/training/filter-model-given-input.pl
<http://filter-model-given-input.pl> ./filtered
/home/wei_lu/pku_seg/model/moses.ini
/home/wei_lu/experiment/data/dev/dev2010.lowercased.indexed.zh >
filterphrases.out 2> filterphrases.err*
*Asking moses for feature names and values from filtered/moses.ini*
*Executing: /home/wei_lu/tools/mosesdecoder/bin/moses -threads 4 -config
filtered/moses.ini -inputtype 0 -show-weights > ./features.list*
*Defined parameters (per moses.ini or switch):*
* config: filtered/moses.ini *
* distortion-limit: 6 *
* feature: UnknownWordPenalty WordPenalty PhrasePenalty
PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=6
path=/home/wei_lu/pku_seg/mert-work-index/filtered/phrase-table.0-0.1.1.gz
input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0
num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0
output-factor=0
path=/home/wei_lu/pku_seg/mert-work-index/filtered/reordering-table.wbe-msd-bidirectional-fe
Distortion KENLM lazyken=0 name=LM0 factor=0
path=/home/wei_lu/experiment/lm/allan_lm.lm order=3 *
* input-factors: 0 *
* inputtype: 0 *
* mapping: 0 T 0 *
* show-weights: *
* threads: 4 *
* weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3
0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5 *
*/home/wei_lu/tools/mosesdecoder/bin*
*line=UnknownWordPenalty*
*FeatureFunction: UnknownWordPenalty0 start: 0 end: 0*
*line=WordPenalty*
*FeatureFunction: WordPenalty0 start: 1 end: 1*
*line=PhrasePenalty*
*FeatureFunction: PhrasePenalty0 start: 2 end: 2*
*line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20
num-features=6
path=/home/wei_lu/pku_seg/mert-work-index/filtered/phrase-table.0-0.1.1.gz
input-factor=0 output-factor=0*
*FeatureFunction: TranslationModel0 start: 3 end: 8*
*line=LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=/home/wei_lu/pku_seg/mert-work-index/filtered/reordering-table.wbe-msd-bidirectional-fe*
*FeatureFunction: LexicalReordering0 start: 9 end: 14*
*Initializing LexicalReordering..*
*line=Distortion*
*FeatureFunction: Distortion0 start: 15 end: 15*
*line=KENLM lazyken=0 name=LM0 factor=0
path=/home/wei_lu/experiment/lm/allan_lm.lm order=3*
*FeatureFunction: LM0 start: 16 end: 16*
*Loading the LM will be faster if you build a binary file.*
*Reading /home/wei_lu/experiment/lm/allan_lm.lm*
*----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100*
******************************************************************************************************
*Loading table into memory...done.*
*Start loading text SCFG phrase table. Moses format : [1507.000] seconds*
*Reading
/home/wei_lu/pku_seg/mert-work-index/filtered/phrase-table.0-0.1.1.gz*
*----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100*
*****sh: line 1: 49997 Killed
/home/wei_lu/tools/mosesdecoder/bin/moses -threads 4 -config
filtered/moses.ini -inputtype 0 -show-weights > ./features.list*
*Exit code: 137*
*Failed to run moses with the config filtered/moses.ini at
/home/wei_lu/tools/mosesdecoder/scripts/training/mert-moses.pl
<http://mert-moses.pl> line 1271*.

I have no idea why this happened.
And then I try to run a small dataset(only one sentence in the tuning
instance and relevant phrase-table), that works fine.

I don't know what's this problem.

Best Regards,
Allan

--
Research student, final-year undergraduate
Extreme Scale Network Computing and service laboratory(
http://www.cloud-uestc.cn/cloud-uestc-EN/index.html),
School of Computer Science & Engineering,
University of Electronic Science and Technology of China(
http://www.oice.uestc.edu.cn/en/).
Website: http://www.allanjie.ml/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140312/00b2db18/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 89, Issue 24
*********************************************

0 Response to "Moses-support Digest, Vol 89, Issue 24"

Post a Comment

Subscribe to: Post Comments (Atom)