Moses-support Digest, Vol 107, Issue 15

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: [Moses-developers] Generation models with Mmsapt
(Michael Denkowski)
2. PhraseDictionaryCompact Problem (Amir Pouya Agha Sadeghi)


----------------------------------------------------------------------

Message: 1
Date: Fri, 4 Sep 2015 23:56:49 -0400
From: Michael Denkowski <michael.j.denkowski@gmail.com>
Subject: Re: [Moses-support] [Moses-developers] Generation models with
Mmsapt
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CA+-GegK3_G_u0X5+Z_SnhqDEQ83R0=Cu8reedGoze5a5JJDJXQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I added non-binarized versions of all the model files including reordering
to the tarball:
https://drive.google.com/file/d/0B6trVKD0-obBdEV0dFg5RkN4Yjg/view?usp=sharing
.

Best,
Michael

On Fri, Sep 4, 2015 at 7:10 AM, Hieu Hoang <hieuhoang@gmail.com> wrote:

> ok. I'm still getting segfault in Lex Reordering during loading. If you
> can provide the ro text file so I can binarize it myself, I can debug it.
> But not a priority, I can park the issue for another time
>
> Hieu Hoang
> Researcher
> New York University, Abu Dhabi
> http://www.hoang.co.uk/hieu
>
> On 4 September 2015 at 03:49, Michael Denkowski <
> michael.j.denkowski@gmail.com> wrote:
>
>> Hi Hieu,
>>
>> Yes, I have everything working together with the caveat about order in
>> the moses.ini file (https://github.com/moses-smt/mosesdecoder/pull/124).
>> The mmsapt files might be dependent on Boost version so I also included the
>> aligned bitext I used to build the model and rebuilt the tarball:
>> https://drive.google.com/file/d/0B6trVKD0-obBRHFjMGxRZTJvV1U/view?usp=sharing.
>> I made it a pull request instead of just merging it into master so you guys
>> could look over the changes since technically the default behavior could
>> change if the moses.ini file lists phrase tables before other features.
>>
>> Best,
>> Michael
>>
>> On Thu, Sep 3, 2015 at 8:44 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>>
>>> i saw your checkins, is it working for you now?
>>>
>>> Your test data doesn't seem to run for me, the lexical reordering file
>>> seems to be corrupt.
>>>
>>> (if you wanna share test data, can you do it via dropbox/google drive
>>> rather than the Moses github)
>>>
>>>
>>> On 03/09/2015 05:13, Michael Denkowski wrote:
>>>
>>> Sounds good. I added a small test model to my branch:
>>> <https://github.com/moses-smt/mosesdecoder/raw/mjdenkowski/mmsapt-factor-test.tar.gz>
>>> https://github.com/moses-smt/mosesdecoder/raw/mjdenkowski/mmsapt-factor-test.tar.gz.
>>> This translates a sample of fr-en news with a Mmsapt, surface LM, and
>>> 400-class LM.
>>>
>>> --Michael
>>>
>>> On Wed, Sep 2, 2015 at 2:56 AM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>>>
>>>> It should work. The function
>>>> EvaluateInIsolation()
>>>> in the LM is for optimisation reason. eg. if the target phrase is 'a b
>>>> c d' and the LM is a trigram, the trigrams 'a b c' and 'b c d' can be
>>>> precalculated in EvaluateInIsolation().
>>>>
>>>> Implementing a pt for factors requires setting up some variables, which
>>>> may not have happen yet in mmsapt. if you can send me a small example
>>>> model, i'll see what i can do
>>>>
>>>>
>>>> On 01/09/2015 02:11, Ulrich Germann wrote:
>>>>
>>>> Hi Michael,
>>>>
>>>> I have no experience with factored models, so I'm speculating here to
>>>> some degree. The reason the phrase table calls EvaluateInIsolation is
>>>> because all "isolated" phrase scores are considered when pruning. In my
>>>> opinion pruning should not happen within the phrase tables (for exactly the
>>>> reason that it does not allow feature functions to be agnostic about other
>>>> feature functions) but by whatever object calls all the phrase tables and
>>>> does the generation. However, for software legacy reasons, that's the way
>>>> it is right now, and I'm not likely to address this issue any time soon
>>>> myself. The most reasonable fix for this in my opinion is to move pruning
>>>> where it belongs --- post all the factor generation stuff.
>>>>
>>>> Hieu is probably still the person with the best understanding of how
>>>> factored phrase table entry generation works, so maybe he can chime in on
>>>> this ...
>>>>
>>>> Cheers - Uli
>>>>
>>>>
>>>> On Mon, Aug 31, 2015 at 11:29 PM, Michael Denkowski <
>>>> <michael.j.denkowski@gmail.com>michael.j.denkowski@gmail.com> wrote:
>>>>
>>>>> Hi Ulrich,
>>>>>
>>>>> I was looking into using a class-based LM with your dynamic phrase
>>>>> table via generation models. I translate factor 0 to 0 with the Mmsapt,
>>>>> then generate target factor 1 (word class) with a GM. The class-based LM
>>>>> operates on factor 1.
>>>>>
>>>>> I'm hitting a segfault on what appears to be an order-of-operations
>>>>> issue with the PT and LM. In mmsapt.cpp:578, Mmsapt::mkTPhrase makes a
>>>>> call to tp->EvaluateInIsolation. This calls all of the models, including
>>>>> the LMs. The class LM tries to score factor 1, which doesn't exist yet
>>>>> (since generation happens after translation), and it dies. By nature,
>>>>> other phrase tables don't have this issue since they can just pull up
>>>>> pre-computed scores.
>>>>>
>>>>> Is scoring with all of the models here a strategic choice to get
>>>>> better performance or would it be sufficient to just score with the PT
>>>>> features? Thanks!
>>>>>
>>>>> --Michael
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Ulrich Germann
>>>> Senior Researcher
>>>> School of Informatics
>>>> University of Edinburgh
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-developers mailing listMoses-developers@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-developers
>>>>
>>>>
>>>> --
>>>> Hieu Hoang
>>>> Researcher
>>>> New York University, Abu Dhabihttp://www.hoang.co.uk/hieu
>>>>
>>>>
>>>
>>> --
>>> Hieu Hoang
>>> Researcher
>>> New York University, Abu Dhabihttp://www.hoang.co.uk/hieu
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150904/75402227/attachment-0001.html

------------------------------

Message: 2
Date: Sat, 5 Sep 2015 11:43:19 +0430
From: Amir Pouya Agha Sadeghi <amirpouya.a@gmail.com>
Subject: [Moses-support] PhraseDictionaryCompact Problem
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAFUH-rD5nsM8ogOPa=+-ygrFOh3YcRoi5m3Gb4qUge2CXa4P2Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,
I am trying to retrain my models with Moses 3, I have
used processPhraseTableMin to binarise my phrase-table but when I change my
moses.ini and try to run Moses with compact phrase table I get this error.
Any Idea ? I attached my moses.ini to this email.

Defined parameters (per moses.ini or switch):
config:
/Share/local/aghasadeghi/final_prj/Experiments/EXPR_STRUCT/scripts/model/binarised-model/moses-bin.ini
distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty
PhraseDictionaryCompact name=TranslationModel0 num-features=4
path=/Share/local/aghasadeghi/final_prj/Experiments/EXPR_STRUCT/scripts/model/binarised-model/phrase-table
input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0
num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0
output-factor=0
path=/Share/local/aghasadeghi/final_prj/Experiments/EXPR_STRUCT/scripts/model/binarised-model/reordering-table
Distortion KENLM lazyken=0 name=LM0 factor=0
path=/Share/local/aghasadeghi/final_prj/Experiments/EXPR_STRUCT/scripts/../lm/fa.lm
order=4
input-factors: 0
mapping: 0 T 0
weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3
0.3 Distortion0= 0.3 LM0= 0.5
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhrasePenalty
FeatureFunction: PhrasePenalty0 start: 2 end: 2
line=PhraseDictionaryCompact name=TranslationModel0 num-features=4
path=/Share/local/aghasadeghi/final_prj/Experiments/EXPR_STRUCT/scripts/model/binarised-model/phrase-table
input-factor=0 output-factor=0
*Exception: moses/FF/Factory.cpp:349 in void
Moses::FeatureRegistry::Construct(const string&, const string&) threw
UnknownFeatureException because `i == registry_.end()'.*
*Feature name PhraseDictionaryCompact is not registered.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150905/03d851b4/attachment.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: moses-bin.ini
Type: application/octet-stream
Size: 1212 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20150905/03d851b4/attachment.obj

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 107, Issue 15
**********************************************

Tweet