Moses-support Digest, Vol 122, Issue 33

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Some question about adding features to Moses ( WULongski )
2. Re: issue while Training Translation Model (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Thu, 22 Dec 2016 13:26:17 +0800
From: " WULongski " <1251817914@qq.com>
Subject: [Moses-support] Some question about adding features to Moses
To: " moses-support " <moses-support@mit.edu>
Message-ID: <tencent_3B6697966E138213147828F4@qq.com>
Content-Type: text/plain; charset="gb18030"

Hi,

I have two questions:

1?

After reading the adding feature section of the Moses(http://www.statmt.org/moses/?n=Moses.FeatureFunctions), it says "However, new features are usually more complicated. For instance, it requires reading in a file and representing it with a data structure and more complex computations. See moses/LM/SRI.h and moses/LM/SRI.cpp for something more involved."

while my moses.ini is looking as this:

//////////////////////// moses.ini

UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryCompact name=TranslationModel0 num-features=4 path=/nobackup/f1/asr/wulong/Moses/mosesdecoder/ALLDATApinyin/working/filtered-oneword/phrase-table.0-0.1.1 input-factor=0 output-factor=0
LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/nobackup/f1/asr/wulong/Moses/mosesdecoder/ALLDATApinyin/working/filtered-oneword/reordering-table.wbe-msd-bidirectional-fe.0-0.1
Distortion
KENLM name=LM0 factor=0 path=/nobackup/f1/asr/wulong/Moses/mosesdecoder/ALLDATApinyin/lm/train.blm.en order=3

//////////////////////////

the last line is the KENLM , this means that language model estimation is did by the KENLM. But I installed the moses by the way on the Mose website: " Easy Setup on Ubuntu (on other linux systems, you'll need to install packages that provide gcc, make, git, automake, libtool) " in the http://www.statmt.org/moses/?n=Development.GetStarted.

But in the third step of the installing process, it says " 3. Run the following to install a recent version of Boost (the default version on your system might be too old), as well as cmph (for CompactPT), irstlm (language model from FBK, required to pass the regression tests), and xmlrpc-c (for moses server). By default, these will be installed in ./opt in your working directory:make -f contrib/Makefiles/install-dependencies.gmake ". It means that the language model estimation is did by the irstlm . so I am confused why is KENLM in the moses.ini.

2?

"KENLM name=LM0 factor=0 path=/nobackup/f1/asr/wulong/Moses/mosesdecoder/ALLDATApinyin/lm/train.blm.en order=3"

can you explain the parameters of the above line? i know order =3 means 3-gram , path is the arpa table language model .But i do not know the other parameter meaning.



Thank you very much .
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161222/0e5faad8/attachment-0001.html

------------------------------

Message: 2
Date: Thu, 22 Dec 2016 09:37:54 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] issue while Training Translation Model
To: Kamal Deep Garg <kamal.1cse@gmail.com>, moses-support@mit.edu
Message-ID: <66cc9913-51fe-2773-263a-d9463e6fd178@gmail.com>
Content-Type: text/plain; charset="utf-8"


the error displayed is:
malformed XML
you must escape your data (training, tuning, test). If you tokenize with
the moses tokenizer, then escaping is done for you. If you use your own
tokenizer, then use the moses script:
scripts/tokenizer/escape-special-chars.perl

ps. keep the size of your email down - send the text of the error rather
than the screenshot
On 22/12/2016 07:22, Kamal Deep Garg wrote:
> Dear Sir
>
> I am creating Punjabi to English SMT system. I have successfully
> created the LM for English text. when I am creating TM by using the
> following command
>
> nohup nice ~/Desktop/satsahib/moses/scripts/
> training/train-model.perl -root-dir
> train -corpus train.clean -f
> pun -e eng -alignment
> grow-diag-final-and -reordering
> msd-bidirectional-fe
> -lm 0:3:/home/tanuj/Desktop/exp04/mono.blm.eng:8
> -external-bin-dir ~/Desktop/satsahib/moses/tools &>
> training.out &
>
> ................................................
>
> *It just stops here
> *
> *Score v2.1 -- scoring methods for extracted rules*
> *using inverse mode*
> *Loading lexical translation table from
> /scratch/MT/working/train/model/lex.e2f *
> *
> *
> *I am sending u data as well as snapshots of error that i got. PFA.*
> *
> *
> Please help me out i will be highly obliged. thanking you in anticipation
> Yours sincerely
>
> Thank You
>
> Kamal Deep Garg
> 9988493359

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161222/facd7222/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 122, Issue 33
**********************************************

Tweet