Moses-support Digest, Vol 100, Issue 19

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Using factor in the hierarchical model (Hieu Hoang)
2. unknown words in SRILM/Kenlm (koormoosh)
3. Re: Sparse feature error (Barry Haddow)


----------------------------------------------------------------------

Message: 1
Date: Thu, 05 Feb 2015 12:40:32 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Using factor in the hierarchical model
To: Lisa Kim <ekkim214@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <54D364C0.3010402@gmail.com>
Content-Type: text/plain; charset="utf-8"

it looks like your phrase-table has 2 input factors and 2 output factors.

When you binarize your phrase-table, you must run it like this:
CreateOnDiskPt 2 2 ....
not
CreateOnDiskPt 1 1 ....

On 05/02/15 04:42, Lisa Kim wrote:
>
> Thank you, Hoang.
>
> After updating latest version, I have different error during tuning
> phase. :(
>
> For the some issues, I cannot upload the model files and these are
> part of my models and configs.
>
> Do I have some mistakes in the ems configuration ?
>
> Error Message :
>
> Start loading text phrase table. Moses format : [0.014] seconds
> terminate called after throwing an instance of 'util::Exception'
> what():
> moses/TranslationModel/RuleTable/PhraseDictionaryOnDisk.cpp:96 in
> virtual void Moses::PhraseDictionaryOnDisk::InitializeForInput(const
> Moses::InputType&) threw util::Exception because
> `obj->GetMisc("NumSourceFactors") != m_input.size()'.
> On-disk phrase table has 1 source factors.. The ini file specified 2
> source factors
> Aborted (core dumped)
> Exit code: 134
>
> Moses.ini file :
>
> # input factors
> [input-factors]
> 0
> 1
>
> # mapping steps
> [mapping]
> 0 T 0
> 1 T 1
>
> [cube-pruning-pop-limit]
> 1000
>
> [non-terminals]
> X
>
> [search-algorithm]
> 3
>
> [inputtype]
> 3
>
> [max-chart-span]
> 20
> 1000
>
> # feature functions
> [feature]
> UnknownWordPenalty
> WordPenalty
> PhrasePenalty
> PhraseDictionaryMemory name=TranslationModel0 num-features=4
> path=/home3/ekkim/working/E2K_1502/hierarchical_class/model/phrase-table.1.0,1-0,1
> input-factor=0,1 output-factor=0,1
> PhraseDictionaryMemory name=TranslationModel1 num-features=1
> path=/home3/ekkim/working/E2K_1502/hierarchical_class/model/glue-grammar.1
> input-factor=0 output-factor=0
> KENLM lazyken=0 name=LM0 factor=0
> path=/home3/ekkim/working/E2K_1502/hierarchical_class/lm/travel22.binlm.1
> order=5# dense weights for feature functions
>
> KENLM lazyken=0 name=LM1 factor=1
> path=/home3/ekkim/working/E2K_1502/hierarchical_class/lm/travel22_mkcls.binlm.1
> order=5
>
> # dense weights for feature functions
>
> [weight]
> UnknownWordPenalty0= 1
> WordPenalty0= -1
> PhrasePenalty0= 0.2
> TranslationModel0= 0.2 0.2 0.2 0.2
> TranslationModel1= 1.0
> LM0= 0.5
> LM1= 0.5
>
> Part of Phrase Table :
>
> [X][X] any|296 discount|413 [X][X] [X] ||| [X][X] ??|349 ??|176
> ?|393 [X][X] [X] ||| 0.0306047 0.00709669 0.0481453 0.000125329 |||
> 0-4 1-1 1-3 2-1 2-2 3-0 ||| 0.460664 0.292832 0.292832 ||| |||
> [X][X] any|296 discount|413 [X][X] a|327 [X] ||| [X][X] ??|349 ?
> ?|176 ?|393 [X][X] [X] ||| 0.00803946 0.000214715 0.0481453
> 0.000125329 ||| 0-4 1-1 1-3 2-1 2-2 3-0 ||| 0.460664 0.0769231
> 0.0769231 ||| |||
> [X][X] any|296 discount|413 [X][X] use|136 [X] ||| [X][X] ??|349 ?
> ?|176 ?|393 [X][X] [X] ||| 0.00950118 2.99551e-06 0.0481453
> 0.000125329 ||| 0-4 1-1 1-3 2-1 2-2 3-0 ||| 0.460664 0.0909091
> 0.0909091 ||| |||
> [X][X] any|296 discount|413 cards|404 [X] ||| ??|349 ?|161 ?|302
> ??|184 [X][X] [X] ||| 0.0266944 0.00037872 0.0481453 8.14358e-05 |||
> 0-4 1-0 2-0 2-1 3-2 3-3 ||| 0.561111 0.311111 0.311111 ||| |||
> [X][X] any|296 discount|413 card|404 [X] ||| ??|349 ?|161 ?|302 ?
> ?|184 [X][X] [X] ||| 0.0214509 0.00256232 0.0481453 0.00010193 |||
> 0-4 1-0 1-2 2-0 2-1 3-3 ||| 0.561111 0.25 0.25 ||| |||
> [X][X] any|296 discount|413 for|487 [X] ||| ??|349 [X][X] [X] |||
> 0.000905141 0.000836177 0.0202717 0.102804 ||| 0-1 1-0 2-0 3-0 |||
> 17.7303 0.791666 0.333333 ||| |||
>
> Addings in ems cofig file for factor training :
> input-factors = word mkcls
> output-factors = word mkcls
> alignment-factors = "word -> word"
> translation-factors = "word+mkcls -> word+mkcls"
> decoding-steps = "t0"
>
> 2015. 2. 4. ?? 9:07? "Hieu Hoang" <Hieu.Hoang@ed.ac.uk
> <mailto:Hieu.Hoang@ed.ac.uk>> ?? ??:
>
> try using the latest version of moses from github.
>
> If it still fails, please make your model files available for
> download so I can replicate the problem
>
> Hieu Hoang
> Research Associate (until March 2015)
> ** searching for interesting commercial MT position **
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
> On 4 February 2015 at 06:54, Lisa Kim <ekkim214@gmail.com
> <mailto:ekkim214@gmail.com>> wrote:
>
>
> Thank you.
> It is successful to change the LM name without "=" symbol.
>
> However, decoding still fails with following message.
> Could you check the error message?
>
> --------------------
> Start loading text phrase table. Moses format : [0.038] seconds
> terminate called recursively
> terminate called after throwing an instance of 'util::Exception'
> what(): Aborted (core dumped)
> Exit code: 134
> The decoder died. CONFIG WAS -weight-overwrite 'LM1= 0.125000
> PhrasePenalty0= 0.050000 WordPenalty0= -0.250000
> TranslationModel0= 0.050000 0.050000 0.050000 0.050000
> TranslationModel1= 0.250000 LM0= 0.125000'
>
> 2015. 2. 3. ?? 8:12? "Hieu Hoang" <hieuhoang@gmail.com
> <mailto:hieuhoang@gmail.com>> ?? ??:
>
> as ken mentioned, don't use the = sign in the name of your
> LM file. I think it's fixed in the latest version of
> moses. Which version do you have, from when ago?
>
> moses_chart should handle factors. Again, if you version
> is from long ago, I suggest your update
>
> On 02/02/15 10:06, ekkim214 wrote:
>>
>> I want to use some factor (class) in the hierarchical model.
>> I am using the ems script and using KenLM for surface and
>> factor both.
>> The building of LM is successful with settings
>> "-discount_fallback" but
>> I got an error during tuning phase like this.
>>
>> Exception: moses/LM/Ken.cpp:399 in Moses::LanguageModel*
>> Moses::ConstructKenLM(const string&) threw
>> util::Exception because `args.size() != 2'.
>> Incorrect format of KenLM property:
>> path=/home3/ekkim/working/E2K_1501/hierarchical_class/lm/travel22=mkcls.binlm.5
>> Exit code: 1
>>
>> Does not "moses_chart" decoder which is distributed in
>> the package support factored training?
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>

--
Hieu Hoang
Research Associate (until March 2015)
** searching for interesting commercial MT position **
University of Edinburgh
http://www.hoang.co.uk/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150205/81c1fe45/attachment-0001.htm

------------------------------

Message: 2
Date: Fri, 6 Feb 2015 00:14:10 +1100
From: koormoosh <koormoosh@gmail.com>
Subject: [Moses-support] unknown words in SRILM/Kenlm
To: moses-support@mit.edu
Message-ID:
<CAN3_CDj26io4gEfXT=Sp=dqMedSC3CUabNahYz_pJpw281bubQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

I am trying to figure out how unknown words are being handled in
SRILM/KenLM. I've searched inside the /lm/src directory but the grep
matches are not helpful. I am interested in LM and doing some experiments
with my own implementation of Kneser-Ney, so knowing how unknown words are
handled is important to get roughly equal results with SRILM or KenLM. Any
comments? A pointer to a class is appreciated the most.

* please note that I am not looking for a solution to handle unknown words,
as I already have a solution for it. I want to know exactly how unknown
words are being handled in SRILM.

thank you
-Koormoosh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150206/cd77c578/attachment-0001.htm

------------------------------

Message: 3
Date: Thu, 05 Feb 2015 13:39:57 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Sparse feature error
To: jian zhang <zhangj@computing.dcu.ie>, moses-support@mit.edu
Message-ID: <54D372AD.9070204@staffmail.ed.ac.uk>
Content-Type: text/plain; charset=UTF-8; format=flowed

Hi Jian

The line looks OK. Is the number of dense features correct? Could you
post run4.features.dat and run4.scores.dat somewhere?

cheers - Barry

On 04/02/15 16:32, jian zhang wrote:
> Hi all,
>
> My tuning with sparse features throws an error like:
>
> kbmira with c=0.01 decay=0.999 no_shuffle=0
> Initialising random seed to 4000
> Found 829 initial sparse features
> .terminate called after throwing an instance of
> 'MosesTuning::FileFormatException'
> what(): Error in line "-0.897812 0 0 0 -0.956697 0 0 0 0 -27.6047 2
> 1 -6 3 -3.99712 -6.39143 -11.4808 -11.4945 PL_s1=2 PL_1,2=2 PL_t2=3
> PL_2,2=1 PL_s2=1 WT_?~.=1 WT_OTHER~the=1 TWI_OTHER=1 " of
> run4.features.dat
>
> Could you please check it, thanks. (I am using Moses 2.0)
>
> Thanks,
>
> Jian
> --
> Jian Zhang
> Centre for Next Generation Localisation (CNGL)
> <http://www.cngl.ie/index.html>
> Dublin City University <http://www.dcu.ie/>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support


--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 100, Issue 19
**********************************************

Tweet