Moses-support Digest, Vol 101, Issue 39

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Unique k-best lists in moses? (Chris Dyer)
2. Re: Moses segmentation fault under multi-thread (Barry Haddow)
3. Re: Unique k-best lists in moses? (Philip Williams)


----------------------------------------------------------------------

Message: 1
Date: Wed, 11 Mar 2015 16:03:32 -0400
From: Chris Dyer <cdyer@cs.cmu.edu>
Subject: [Moses-support] Unique k-best lists in moses?
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAEHEvxOCNCt2v5LoeB83AKw3ymgwBd8NPguAuqJE8QgOrMFFWQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hi all, can anyone tell me if moses can be configured to generate
"unique" k-best lists using the Huang et al. (2006) dynamic
programming algorithm (Section 5.2 of
http://www.cis.upenn.edu/~lhuang3/amta06-sdtedl.pdf)?
Thanks--
Chris


------------------------------

Message: 2
Date: Wed, 11 Mar 2015 20:58:59 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Moses segmentation fault under
multi-thread
To: Liangyou Li <llysuda@gmail.com>, Matthias Huck
<mhuck@inf.ed.ac.uk>
Cc: moses-support@mit.edu
Message-ID: <5500AC93.7030600@staffmail.ed.ac.uk>
Content-Type: text/plain; charset="iso-8859-1"

Hi Liangyou

This is a bit strange. The stacktrace suggests corruption in the feature
vector, but it would be surprising that we haven't seen it before. Can
you reproduce the problem when running under valgrind? The memory could
be getting corrupted by a completely different source, which we'll
really only find with a memory checker.

cheers - Barry

On 11/03/15 17:50, Liangyou Li wrote:
> Thanks Matthias.
>
> This problem doesn't always happen.
> Even on the same data, sometimes it can run without crash, sometimes
> it doesn't.
> But, by now, the error only happens when the decoder outputs features.
> (-n-best-list or -output-search-graph-hypergraph)
>
> Cheers
> Liangyou
>
>
> On Wed, Mar 11, 2015 at 5:38 PM, Matthias Huck <mhuck@inf.ed.ac.uk
> <mailto:mhuck@inf.ed.ac.uk>> wrote:
>
> Hi,
>
> I've recently been using these sparse feature functions without any
> issues in multi-threaded chart-based decoding. There might be a
> problem
> with thread safety, but I currently can't tell why you got the
> segmentation fault. You should investigate this in more detail.
>
> Cheers,
> Matthias
>
>
>
> On Wed, 2015-03-11 at 12:12 +0000, Liangyou Li wrote:
> > When I run moses with sparse features in multi-threads (
> parameter -threads all), I got segmentation fault.
> >
> >
> > The three sparse features are:
> > SourceWordDeletionFeature factor=0
> > TargetWordInsertionFeature factor=0
> > WordTranslationFeature input-factor=0 output-factor=0
> >
> >
> > The exact command I used is:
> > moses_chart -threads all -f moses.ini -i input -n-best-list
> nbest 100 distinct
> >
> >
> > This error happens kind of randomly. But it only happens when
> sparse features and multi-threads are used.
> >
> >
> > I've tried several times to use gdb to trace the error.
> Fortunately, I just get the back-trace info, as listed:
> >
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > [Switching to Thread 0x7fffdae77700 (LWP 32390)]
> > 0x00007ffff711f5e3 in std::basic_ostream<char,
> std::char_traits<char> >& std::operator<< <char,
> std::char_traits<char>, std::allocator<char>
> >(std::basic_ostream<char, std::char_traits<char> >&,
> std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&) () from /usr/lib64/libstdc++.so.6
> > Missing separate debuginfos, use: zypper install
> glibc-debuginfo-2.15-22.17.1.x86_64
> libgcc47-debuginfo-4.7.1_20120723-1.1.1.x86_64
> liblzma5-debuginfo-5.0.3-12.2.2.x86_64
> libstdc++47-debuginfo-4.7.1_20120723-1.1.1.x86_64
> zlib-debuginfo-1.2.7-2.1.2.x86_64
> > (gdb) backtrace
> > #0 0x00007ffff711f5e3 in std::basic_ostream<char,
> std::char_traits<char> >& std::operator<< <char,
> std::char_traits<char>, std::allocator<char>
> >(std::basic_ostream<char, std::char_traits<char> >&,
> std::basic_string<char, std::char_traits<char>,
> std::allocator<char> > const&) () from /usr/lib64/libstdc++.so.6
> > #1 0x0000000000584711 in Moses::operator<< (out=..., name=...)
> at moses/FeatureVector.cpp:141
> > #2 0x0000000000440232 in
> Moses::ScoreComponentCollection::GetVectorForProducer
> (this=0x7fffad0e9380, sp=0xe2aab0) at
> moses/ScoreComponentCollection.cpp:293
> > #3 0x00000000004407da in
> Moses::ScoreComponentCollection::OutputFeatureScores
> (this=0x7fffad0e9380, out=..., ff=0xe2aab0, lastName="LM0") at
> moses/ScoreComponentCollection.cpp:351
> > #4 0x000000000044055a in
> Moses::ScoreComponentCollection::OutputAllFeatureScores
> (this=0x7fffad0e9380, out=...) at
> moses/ScoreComponentCollection.cpp:319
> > #5 0x0000000000561ce4 in Moses::ChartManager::OutputNBestList
> (this=0x7fffacbe0010, collector=0x9320d210, nBestList=std::vector
> of length 100, capacity 128 = {...}, translationId=569)
> > at moses/ChartManager.cpp:381
> > #6 0x00000000005619d6 in Moses::ChartManager::OutputNBest
> (this=0x7fffacbe0010, collector=0x9320d210) at
> moses/ChartManager.cpp:335
> > #7 0x0000000000467cd6 in Moses::TranslationTask::Run
> (this=0x214d90b0) at moses/TranslationTask.cpp:111
> > #8 0x0000000000530b69 in Moses::ThreadPool::Execute
> (this=0x7fffffffd710) at moses/ThreadPool.cpp:61
> > #9 0x0000000000534ecd in boost::_mfi::mf0<void,
> Moses::ThreadPool>::operator() (this=0xf263088, p=0x7fffffffd710)
> at
> /home/lly/plateform/boost_1_55_0/include/boost/bind/mem_fn_template.hpp:49
> > #10 0x0000000000534e30 in
> boost::_bi::list1<boost::_bi::value<Moses::ThreadPool*>
> >::operator()<boost::_mfi::mf0<void, Moses::ThreadPool>,
> boost::_bi::list0> (this=0xf263098, f=..., a=...)
> > at
> /home/lly/plateform/boost_1_55_0/include/boost/bind/bind.hpp:253
> > #11 0x0000000000534dd5 in boost::_bi::bind_t<void,
> boost::_mfi::mf0<void, Moses::ThreadPool>,
> boost::_bi::list1<boost::_bi::value<Moses::ThreadPool*> >
> >::operator() (this=0xf263088)
> > at
> /home/lly/plateform/boost_1_55_0/include/boost/bind/bind_template.hpp:20
> > #12 0x0000000000534d9a in
> boost::detail::thread_data<boost::_bi::bind_t<void,
> boost::_mfi::mf0<void, Moses::ThreadPool>,
> boost::_bi::list1<boost::_bi::value<Moses::ThreadPool*> > > >::run
> (this=0xf262ed0)
> > at
> /home/lly/plateform/boost_1_55_0/include/boost/thread/detail/thread.hpp:117
> > #13 0x00000000008b4752 in thread_proxy ()
> > #14 0x00007ffff6965e0e in start_thread () from
> /lib64/libpthread.so.0
> > #15 0x00007ffff669d2cd in clone () from /lib64/libc.so.6
> >
> >
> > Has anyone had the problem before ? Any ideas on solving this ?
> > Many Thanks ?
> >
> >
> >
> >
> > PS: In my experiment, I found the function " void
> CompletedRuleCollection::Add(const TargetPhraseCollection &tpc,
> const StackVec &stackVec, const std::vector<float> &stackScores,
> const ChartParserCallback &outColl) " does not consider "
> m_ruleLimit " . So after adding parameter " -rule-limit 0 ", the
> decoder can only collect one translation option.
> >
> >
> >
> >
> > Cheers
> > Liangyou
> >
> >
> >
> >
> >
> >
> > --
> >
> >
> > Liangyou Li
> > CNGL
> > School of Computing
> > Dublin City University
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> > http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
>
>
> --
>
> Liangyou Li
> CNGL
> School of Computing
> Dublin City University
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150311/fbcb3431/attachment-0001.htm

------------------------------

Message: 3
Date: Wed, 11 Mar 2015 22:56:41 +0000
From: Philip Williams <philip.williams@mac.com>
Subject: Re: [Moses-support] Unique k-best lists in moses?
To: Chris Dyer <cdyer@cs.cmu.edu>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <089E9F26-9502-4F29-B6D3-C235185EB24E@mac.com>
Content-Type: text/plain; charset=us-ascii

Hi Chris,

if you add the argument "distinct" to the -n-best-list option then that
should have the same effect, albeit far less efficiently (internally,
Moses simply generates a longer list and then filters it).

I'm guessing you already knew about that option though. If that's too
slow for what you're doing then give me a day or two and I'll find some
time to implement the Huang et al. trick. Clearly, that's what we
should be doing anyway.

Oh wait, are you talking about phrase-based or syntax-based? For
phrase-based we don't -- but obviously could -- use Huang and Chiang's
(2005) algorithm for k-best extraction. As things stand, doing the
Huang et al. unique trick in Moses would involve more effort to
implement for phrase-based than for syntax-based.

Phil

On 11 Mar 2015, at 20:03, Chris Dyer <cdyer@cs.cmu.edu> wrote:

> Hi all, can anyone tell me if moses can be configured to generate
> "unique" k-best lists using the Huang et al. (2006) dynamic
> programming algorithm (Section 5.2 of
> http://www.cis.upenn.edu/~lhuang3/amta06-sdtedl.pdf)?
> Thanks--
> Chris
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support



------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 101, Issue 39
**********************************************

Tweet