Moses-support Digest, Vol 100, Issue 36

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Using factor in the hierarchical model (Lisa Kim)
2. Re: optimizing lattice InputFeature weight (Jorg Tiedemann)


----------------------------------------------------------------------

Message: 1
Date: Tue, 10 Feb 2015 18:57:18 +0900
From: Lisa Kim <ekkim214@gmail.com>
Subject: Re: [Moses-support] Using factor in the hierarchical model
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAOmDmvYDgea8fP-XFmpEuVi3Mi_BrGHGVp0WbsDbynC1jd=SCw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Thank you.
I did not notice the options for CteateOnDiskPt command.
I have finally succeeded to decode factored hierarchical model without LM
of factor.
However, when I apply additional LM for output factor, decoder died with
segmentation fault.
This is the result of run using gdb debugging.

I hope this will be last question in this thread. :)

--------------------------------
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff5b3b700 (LWP 18577)]
0x000000000043c722 in Moses::Factor::GetId (this=0x0) at ./moses/Factor.h:69
69 return m_id;

(gdb) where
#0 0x000000000043c722 in Moses::Factor::GetId (this=0x0)
at ./moses/Factor.h:69
#1 0x00000000005861a2 in Moses::LanguageModelSRI::GetLmID (this=0x1aa3580,
factor=0x0) at moses/LM/SRI.cpp:132
#2 0x000000000058638e in Moses::LanguageModelSRI::GetValue (this=0x1aa3580,
contextFactor=..., finalState=0x7fffe849d848) at moses/LM/SRI.cpp:160
#3 0x000000000078c36d in
Moses::LanguageModelSingleFactor::GetValueForgotState
(this=0x1aa3580, contextFactor=..., outState=...)
at moses/LM/SingleFactor.cpp:72
#4 0x0000000000789d3b in
Moses::LanguageModelImplementation::GetValueGivenState (this=0x1aa3580,
contextFactor=..., state=...)
at moses/LM/Implementation.cpp:78
#5 0x000000000078afaf in
Moses::LanguageModelImplementation::EvaluateWhenApplied (this=0x1aa3580,
hypo=..., featureID=1, out=0x7fffe832cb38)
at moses/LM/Implementation.cpp:308
#6 0x0000000000645f77 in Moses::ChartHypothesis::EvaluateWhenApplied (
this=0x7fffe832caf0) at moses/ChartHypothesis.cpp:229
#7 0x00000000007d0479 in Moses::RuleCubeItem::CreateHypothesis (
this=0x7fffe80bae90, transOpt=..., manager=...)
at moses/RuleCubeItem.cpp:82
#8 0x000000000068d69d in Moses::RuleCube::RuleCube (this=0x7fffe85ab1b0,
transOpt=..., allChartCells=..., manager=...) at moses/RuleCube.cpp:50
#9 0x000000000067f552 in Moses::ChartCell::Decode (this=0x7ffff0000d90,
transOptList=..., allChartCells=...) at moses/ChartCell.cpp:98
#10 0x00000000006174e9 in Moses::ChartManager::Decode (this=0x7ffff0000990)
at moses/ChartManager.cpp:96
#11 0x000000000043b596 in Moses::TranslationTask::Run (this=0x24543c0)
at moses/TranslationTask.cpp:102
#12 0x00000000005a0a01 in Moses::ThreadPool::Execute (this=0x7fffffffde00)
at moses/ThreadPool.cpp:61
#13 0x00000000005a386a in boost::_mfi::mf0<void,
Moses::ThreadPool>::operator()
(this=0x65bdf38, p=0x7fffffffde00)
at /usr/include/boost/bind/mem_fn_template.hpp:49
#14 0x00000000005a37da in
boost::_bi::list1<boost::_bi::value<Moses::ThreadPool*>
>::operator()<boost::_mfi::mf0<void, Moses::ThreadPool>, boost::_bi::list0>
(
this=0x65bdf48, f=..., a=...) at /usr/include/boost/bind/bind.hpp:253
#15 0x00000000005a3789 in boost::_bi::bind_t<void, boost::_mfi::mf0<void,
Moses::ThreadPool>, boost::_bi::list1<boost::_bi::value<Moses::ThreadPool*>
> >::operator() (this=0x65bdf38) at
/usr/include/boost/bind/bind_template.hpp:20
#16 0x00000000005a374e in
boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf0<void,
Moses::ThreadPool>, boost::_bi::list1<boost::_bi::value<Moses::ThreadPool*>
> > >::run (this=0x65bddb0)
at /usr/include/boost/thread/detail/thread.hpp:61
#17 0x00000000008b20d4 in thread_proxy ()
---Type <return> to continue, or q <return> to quit---
#18 0x00007ffff6fa6e9a in start_thread (arg=0x7ffff5b3b700)
at pthread_create.c:308
#19 0x00007ffff6cd431d in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#20 0x0000000000000000 in ?? ()
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150210/d200e5dc/attachment-0001.htm

------------------------------

Message: 2
Date: Tue, 10 Feb 2015 13:13:43 +0100
From: Jorg Tiedemann <tiedeman@gmail.com>
Subject: Re: [Moses-support] optimizing lattice InputFeature weight
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: Hieu Hoang <hieu.hoang@ed.ac.uk>, moses-support
<moses-support@mit.edu>
Message-ID: <C790312C-8AA8-4226-8374-64F80496A98A@gmail.com>
Content-Type: text/plain; charset="windows-1252"


Thanks Hieu - it seems to work now! Great!

J?rg


J?rg Tiedemann
tiedeman@gmail.com




On Feb 9, 2015, at 12:49 PM, Hieu Hoang wrote:

> i think i've resolved it and committed the change
> https://github.com/moses-smt/mosesdecoder/commit/ce80e53b30f766ab85cb58c4a2d06742b4a4f38b
> the scores for the input path wasn't being set. At the moment, there's different code paths, depending on what type of phrase-table and input type so it's a little confusing. This would hopefully be resolved when we get rid of the binary phrase-table
>
> On 07/02/15 20:35, Jorg Tiedemann wrote:
>>
>> inputtype is set to 2. I really don't know why this doesn't work.
>>
>> Here an example:
>>
>> lattice input:
>>
>> ((('muutoksia',0.5,1),('Muutoksia',0.9,1),),(('yritystukik?yt?nt?ihin',0.9,3),('yritys',0.5,1),),(('tuki',0.5,1),),(('k?yt?nt?ihin',0.5,1),),(('-',0.9,1),),(('Maakunta',0.9,2),('maa',0.5,1),),(('kunta',0.5,1),),(('-',0.9,1),),(('Alueet',0.9,1),('alueet',0.5,1),),(('-',0.9,1),),(('Uutiset',0.9,1),('uutiset',0.5,1),),(('-',0.9,1),),(('Karjalainen',0.9,1),('karjalainen',0.5,1),),)
>>
>> nbest-list:
>> 0 ||| changes in the business practices - - - news - karjalainen ||| Distortion0= -6 LM0= -74.8382 InputFeature0= 0 WordPenalty0= -11 PhrasePenalty0= 11 TranslationModel0= -45.7094 -47.7159 -16.804 -11.8221 ||| -50.4294
>> 0 ||| changes in the business practices - - - news - karjalainen ||| Distortion0= -6 LM0= -74.8382 InputFeature0= 0 WordPenalty0= -11 PhrasePenalty0= 10 TranslationModel0= -45.7472 -47.7159 -16.5762 -11.8221 ||| -50.5914
>> ...
>>
>> The InputFeature0 is always 0
>>
>> Decoding output includes:
>> 0 -- (muutoksia , , -0.6931) (Muutoksia , , -0.1051)
>> 1 -- (yritystukik?yt?nt?ihin , , -0.1053) (yritys , , -0.6931)
>> 2 -- (tuki , , -0.6931)
>> 3 -- (k?yt?nt?ihin , , -0.6931)
>> 4 -- (- , , -0.1051)
>> 5 -- (Maakunta , , -0.1052) (maa , , -0.6931)
>> 6 -- (kunta , , -0.6931)
>> 7 -- (- , , -0.1051)
>> 8 -- (Alueet , , -0.1051) (alueet , , -0.6931)
>> 9 -- (- , , -0.1051)
>> 10 -- (Uutiset , , -0.1051) (uutiset , , -0.6931)
>> 11 -- (- , , -0.1051)
>> 12 -- (Karjalainen , , -0.1051) (karjalainen , , -0.6931)
>>
>> What I don't understand is why there is an empty field in the output above.
>>
>> And the config file sets the weight for InputFeature0 to 1:
>>
>> [feature]
>> InputFeature num-features=1 num-input-features=1 real-word-count=0
>> ....
>> # dense weights for feature functions
>> [weight]
>> InputFeature0= 1
>>
>> Strange ...
>>
>> J?rg
>>
>>
>> J?rg Tiedemann
>> tiedeman@gmail.com
>>
>>
>>
>>
>> On Feb 7, 2015, at 9:21 PM, Hieu Hoang wrote:
>>
>>> there's no reason why it shouldn't work.
>>>
>>> the only thing i can think of is that the input type hasn't been set to lattice. In the moses.ini, there should be something like
>>> [inputtype]
>>> 2
>>> or on the command line
>>> moses -inputtype 2
>>>
>>>
>>> Hieu Hoang
>>> Research Associate (until March 2015)
>>> ** searching for interesting commercial MT position **
>>> University of Edinburgh
>>> http://www.hoang.co.uk/hieu
>>>
>>>
>>> On 7 February 2015 at 20:05, Jorg Tiedemann <tiedeman@gmail.com> wrote:
>>>
>>> I have a problem with lattice decoding and optimizing input-feature weights. I have edge weights in my lattice input (one per edge) and I defined one input feature that I'd like to optimize using MERT. However, my the input feature value is always 0 in the nbest lists even though none of the input edges has value 1 (or 0). What do I do wrong?
>>>
>>> My initial config file includes:
>>>
>>> [feature]
>>> InputFeature num-features=1 num-input-features=1 real-word-count=0
>>> ...
>>> # dense weights for feature functions
>>> [weight]
>>> InputFeature0= 1
>>>
>>> The lattice input is valid and looks like this:
>>>
>>> ((('word',0.8,1),('word',0.6,1), ....
>>>
>>> MERT tuning fails in the end especially because the input feature cannot be set.
>>> Any help is very much appreciated.
>>> Thanks,
>>> J?rg
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>
>
> --
> Hieu Hoang
> Research Associate (until March 2015)
> ** searching for interesting commercial MT position **
> University of Edinburgh
> http://www.hoang.co.uk/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150210/d8407568/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 100, Issue 36
**********************************************

Tweet