Moses-support Digest, Vol 104, Issue 67

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Exception when exiting moses (Marcin Junczys-Dowmunt)
2. Re: Exception when exiting moses (Marcin Junczys-Dowmunt)
3. Re: Exception when exiting moses (Jeroen Vermeulen)
4. Re: Major bug found in Moses (Raphael Payen)


----------------------------------------------------------------------

Message: 1
Date: Sun, 21 Jun 2015 15:47:50 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Exception when exiting moses
To: moses-support@mit.edu
Message-ID: <5586C086.5090300@amu.edu.pl>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Recompiling it just now with debug on. There is already a mistake in

https://github.com/moses-smt/mosesdecoder/blob/master/moses/TranslationModel/CompactPT/MmapAllocator.h#L156

The size is wrong, the offset is missing. But it wasn't that.

#0 0x00007ffff6e05cc9 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007ffff6e090d8 in __GI_abort () at abort.c:89
#2 0x00007ffff792e6b5 in __gnu_cxx::__verbose_terminate_handler() ()
from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007ffff792c836 in ?? () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff792b8f9 in ?? () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff792c4aa in __gxx_personality_v0 () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff73c1ff3 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
#7 0x00007ffff73c2517 in _Unwind_Resume () from
/lib/x86_64-linux-gnu/libgcc_s.so.1
#8 0x00000000004798a7 in util::UnmapOrThrow (start=<optimized out>,
length=<optimized out>)
at util/mmap.cc:52
#9 0x00000000005552c1 in _M_deallocate (__n=<optimized out>,
__p=<optimized out>, this=0x12ba3a0)
at /usr/include/c++/4.8/bits/stl_vector.h:174
#10 ~_Vector_base (this=0x12ba3a0, __in_chrg=<optimized out>)
at /usr/include/c++/4.8/bits/stl_vector.h:160
#11 ~vector (this=0x12ba3a0, __in_chrg=<optimized out>) at
/usr/include/c++/4.8/bits/stl_vector.h:416
#12 Moses::StringVector<unsigned char, unsigned long,
Moses::MmapAllocator>::~StringVector (
this=0x2746458, __in_chrg=<optimized out>) at
moses/TranslationModel/CompactPT/StringVector.h:154
#13 0x000000000055862e in
Moses::LexicalReorderingTableCompact::~LexicalReorderingTableCompact (
this=0x2746000, __in_chrg=<optimized out>)
at
moses/TranslationModel/CompactPT/LexicalReorderingTableCompact.cpp:56
#14 0x0000000000558669 in
Moses::LexicalReorderingTableCompact::~LexicalReorderingTableCompact (
this=0x2746000, __in_chrg=<optimized out>)
at
moses/TranslationModel/CompactPT/LexicalReorderingTableCompact.cpp:60
#15 0x00000000005269b3 in checked_delete<Moses::LexicalReorderingTable>
(x=<optimized out>)
at /usr/include/boost/checked_delete.hpp:34
#16 ~scoped_ptr (this=0x12c4408, __in_chrg=<optimized out>)
at /usr/include/boost/smart_ptr/scoped_ptr.hpp:82
#17 Moses::LexicalReordering::~LexicalReordering (this=0x12c4360,
__in_chrg=<optimized out>)
at moses/FF/LexicalReordering/LexicalReordering.cpp:82
#18 0x0000000000526b99 in Moses::LexicalReordering::~LexicalReordering
(this=0x12c4360,
__in_chrg=<optimized out>) at
moses/FF/LexicalReordering/LexicalReordering.cpp:83
#19 0x000000000045ea06 in
RemoveAllInColl<std::vector<Moses::FeatureFunction*> > (coll=...)
at ./moses/Util.h:436
#20 Moses::FeatureFunction::Destroy () at moses/FF/FeatureFunction.cpp:38
#21 0x0000000000417157 in batch_run () at moses/ExportInterface.cpp:274
#22 0x00000000004184d8 in decoder_main (argc=5, argv=0x7fffffffdef8) at
moses/ExportInterface.cpp:327
#23 0x00007ffff6df0ec5 in __libc_start_main (main=0x40c6b0 <main(int,
char**)>, argc=5,
argv=0x7fffffffdef8, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>,
stack_end=0x7fffffffdee8) at libc-start.c:287
#24 0x0000000000415d2a in _start ()


On 21.06.2015 15:44, Kenneth Heafield wrote:
> What exception? I can haz stack trace?
>
> On 06/21/2015 09:02 AM, Marcin Junczys-Dowmunt wrote:
>> Hi,
>> is anyone else getting exceptions when moses exits with the latest
>> master? It seems to be happening in my reordering table and breaks MERT.
>> Wasn't me though :)
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support



------------------------------

Message: 2
Date: Sun, 21 Jun 2015 16:20:23 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Exception when exiting moses
To: moses-support@mit.edu
Message-ID: <5586C827.2080608@amu.edu.pl>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Just in case, don't bother looking. Seems to be my mess after all, just
dormant for a couple of years. Slowly unraveling it.

On 21.06.2015 15:47, Marcin Junczys-Dowmunt wrote:
> Recompiling it just now with debug on. There is already a mistake in
>
> https://github.com/moses-smt/mosesdecoder/blob/master/moses/TranslationModel/CompactPT/MmapAllocator.h#L156
>
> The size is wrong, the offset is missing. But it wasn't that.
>
> #0 0x00007ffff6e05cc9 in __GI_raise (sig=sig@entry=6) at
> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
> #1 0x00007ffff6e090d8 in __GI_abort () at abort.c:89
> #2 0x00007ffff792e6b5 in __gnu_cxx::__verbose_terminate_handler() ()
> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #3 0x00007ffff792c836 in ?? () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #4 0x00007ffff792b8f9 in ?? () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #5 0x00007ffff792c4aa in __gxx_personality_v0 () from
> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
> #6 0x00007ffff73c1ff3 in ?? () from /lib/x86_64-linux-gnu/libgcc_s.so.1
> #7 0x00007ffff73c2517 in _Unwind_Resume () from
> /lib/x86_64-linux-gnu/libgcc_s.so.1
> #8 0x00000000004798a7 in util::UnmapOrThrow (start=<optimized out>,
> length=<optimized out>)
> at util/mmap.cc:52
> #9 0x00000000005552c1 in _M_deallocate (__n=<optimized out>,
> __p=<optimized out>, this=0x12ba3a0)
> at /usr/include/c++/4.8/bits/stl_vector.h:174
> #10 ~_Vector_base (this=0x12ba3a0, __in_chrg=<optimized out>)
> at /usr/include/c++/4.8/bits/stl_vector.h:160
> #11 ~vector (this=0x12ba3a0, __in_chrg=<optimized out>) at
> /usr/include/c++/4.8/bits/stl_vector.h:416
> #12 Moses::StringVector<unsigned char, unsigned long,
> Moses::MmapAllocator>::~StringVector (
> this=0x2746458, __in_chrg=<optimized out>) at
> moses/TranslationModel/CompactPT/StringVector.h:154
> #13 0x000000000055862e in
> Moses::LexicalReorderingTableCompact::~LexicalReorderingTableCompact (
> this=0x2746000, __in_chrg=<optimized out>)
> at
> moses/TranslationModel/CompactPT/LexicalReorderingTableCompact.cpp:56
> #14 0x0000000000558669 in
> Moses::LexicalReorderingTableCompact::~LexicalReorderingTableCompact (
> this=0x2746000, __in_chrg=<optimized out>)
> at
> moses/TranslationModel/CompactPT/LexicalReorderingTableCompact.cpp:60
> #15 0x00000000005269b3 in checked_delete<Moses::LexicalReorderingTable>
> (x=<optimized out>)
> at /usr/include/boost/checked_delete.hpp:34
> #16 ~scoped_ptr (this=0x12c4408, __in_chrg=<optimized out>)
> at /usr/include/boost/smart_ptr/scoped_ptr.hpp:82
> #17 Moses::LexicalReordering::~LexicalReordering (this=0x12c4360,
> __in_chrg=<optimized out>)
> at moses/FF/LexicalReordering/LexicalReordering.cpp:82
> #18 0x0000000000526b99 in Moses::LexicalReordering::~LexicalReordering
> (this=0x12c4360,
> __in_chrg=<optimized out>) at
> moses/FF/LexicalReordering/LexicalReordering.cpp:83
> #19 0x000000000045ea06 in
> RemoveAllInColl<std::vector<Moses::FeatureFunction*> > (coll=...)
> at ./moses/Util.h:436
> #20 Moses::FeatureFunction::Destroy () at moses/FF/FeatureFunction.cpp:38
> #21 0x0000000000417157 in batch_run () at moses/ExportInterface.cpp:274
> #22 0x00000000004184d8 in decoder_main (argc=5, argv=0x7fffffffdef8) at
> moses/ExportInterface.cpp:327
> #23 0x00007ffff6df0ec5 in __libc_start_main (main=0x40c6b0 <main(int,
> char**)>, argc=5,
> argv=0x7fffffffdef8, init=<optimized out>, fini=<optimized out>,
> rtld_fini=<optimized out>,
> stack_end=0x7fffffffdee8) at libc-start.c:287
> #24 0x0000000000415d2a in _start ()
>
>
> On 21.06.2015 15:44, Kenneth Heafield wrote:
>> What exception? I can haz stack trace?
>>
>> On 06/21/2015 09:02 AM, Marcin Junczys-Dowmunt wrote:
>>> Hi,
>>> is anyone else getting exceptions when moses exits with the latest
>>> master? It seems to be happening in my reordering table and breaks MERT.
>>> Wasn't me though :)
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support



------------------------------

Message: 3
Date: Sun, 21 Jun 2015 21:26:38 +0700
From: Jeroen Vermeulen <jtv@precisiontranslationtools.com>
Subject: Re: [Moses-support] Exception when exiting moses
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>, moses-support@mit.edu
Message-ID:
<B0D81211-241D-4A32-A40A-98BE03FEE36A@precisiontranslationtools.com>
Content-Type: text/plain; charset=UTF-8

Hope I didn't break the mmap allocator - though that would have been some time back anyway.

It does remind me that Boost has a portable mmap wrapper. It's in the iostreams library. Might be worth trying to replace kenlm's wrappers with that one to reduce maintenance burden.


Jeroen

On June 21, 2015 8:47:50 PM GMT+07:00, Marcin Junczys-Dowmunt <junczys@amu.edu.pl> wrote:
>Recompiling it just now with debug on. There is already a mistake in
>
>https://github.com/moses-smt/mosesdecoder/blob/master/moses/TranslationModel/CompactPT/MmapAllocator.h#L156
>
>The size is wrong, the offset is missing. But it wasn't that.
>
>#0 0x00007ffff6e05cc9 in __GI_raise (sig=sig@entry=6) at
>../nptl/sysdeps/unix/sysv/linux/raise.c:56
>#1 0x00007ffff6e090d8 in __GI_abort () at abort.c:89
>#2 0x00007ffff792e6b5 in __gnu_cxx::__verbose_terminate_handler() ()
> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>#3 0x00007ffff792c836 in ?? () from
>/usr/lib/x86_64-linux-gnu/libstdc++.so.6
>#4 0x00007ffff792b8f9 in ?? () from
>/usr/lib/x86_64-linux-gnu/libstdc++.so.6
>#5 0x00007ffff792c4aa in __gxx_personality_v0 () from
>/usr/lib/x86_64-linux-gnu/libstdc++.so.6
>#6 0x00007ffff73c1ff3 in ?? () from
>/lib/x86_64-linux-gnu/libgcc_s.so.1
>#7 0x00007ffff73c2517 in _Unwind_Resume () from
>/lib/x86_64-linux-gnu/libgcc_s.so.1
>#8 0x00000000004798a7 in util::UnmapOrThrow (start=<optimized out>,
>length=<optimized out>)
> at util/mmap.cc:52
>#9 0x00000000005552c1 in _M_deallocate (__n=<optimized out>,
>__p=<optimized out>, this=0x12ba3a0)
> at /usr/include/c++/4.8/bits/stl_vector.h:174
>#10 ~_Vector_base (this=0x12ba3a0, __in_chrg=<optimized out>)
> at /usr/include/c++/4.8/bits/stl_vector.h:160
>#11 ~vector (this=0x12ba3a0, __in_chrg=<optimized out>) at
>/usr/include/c++/4.8/bits/stl_vector.h:416
>#12 Moses::StringVector<unsigned char, unsigned long,
>Moses::MmapAllocator>::~StringVector (
> this=0x2746458, __in_chrg=<optimized out>) at
>moses/TranslationModel/CompactPT/StringVector.h:154
>#13 0x000000000055862e in
>Moses::LexicalReorderingTableCompact::~LexicalReorderingTableCompact (
> this=0x2746000, __in_chrg=<optimized out>)
> at
>moses/TranslationModel/CompactPT/LexicalReorderingTableCompact.cpp:56
>#14 0x0000000000558669 in
>Moses::LexicalReorderingTableCompact::~LexicalReorderingTableCompact (
> this=0x2746000, __in_chrg=<optimized out>)
> at
>moses/TranslationModel/CompactPT/LexicalReorderingTableCompact.cpp:60
>#15 0x00000000005269b3 in checked_delete<Moses::LexicalReorderingTable>
>
>(x=<optimized out>)
> at /usr/include/boost/checked_delete.hpp:34
>#16 ~scoped_ptr (this=0x12c4408, __in_chrg=<optimized out>)
> at /usr/include/boost/smart_ptr/scoped_ptr.hpp:82
>#17 Moses::LexicalReordering::~LexicalReordering (this=0x12c4360,
>__in_chrg=<optimized out>)
> at moses/FF/LexicalReordering/LexicalReordering.cpp:82
>#18 0x0000000000526b99 in Moses::LexicalReordering::~LexicalReordering
>(this=0x12c4360,
> __in_chrg=<optimized out>) at
>moses/FF/LexicalReordering/LexicalReordering.cpp:83
>#19 0x000000000045ea06 in
>RemoveAllInColl<std::vector<Moses::FeatureFunction*> > (coll=...)
> at ./moses/Util.h:436
>#20 Moses::FeatureFunction::Destroy () at
>moses/FF/FeatureFunction.cpp:38
>#21 0x0000000000417157 in batch_run () at moses/ExportInterface.cpp:274
>#22 0x00000000004184d8 in decoder_main (argc=5, argv=0x7fffffffdef8) at
>
>moses/ExportInterface.cpp:327
>#23 0x00007ffff6df0ec5 in __libc_start_main (main=0x40c6b0 <main(int,
>char**)>, argc=5,
> argv=0x7fffffffdef8, init=<optimized out>, fini=<optimized out>,
>rtld_fini=<optimized out>,
> stack_end=0x7fffffffdee8) at libc-start.c:287
>#24 0x0000000000415d2a in _start ()
>
>
>On 21.06.2015 15:44, Kenneth Heafield wrote:
>> What exception? I can haz stack trace?
>>
>> On 06/21/2015 09:02 AM, Marcin Junczys-Dowmunt wrote:
>>> Hi,
>>> is anyone else getting exceptions when moses exits with the latest
>>> master? It seems to be happening in my reordering table and breaks
>MERT.
>>> Wasn't me though :)
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>_______________________________________________
>Moses-support mailing list
>Moses-support@mit.edu
>http://mailman.mit.edu/mailman/listinfo/moses-support



------------------------------

Message: 4
Date: Sun, 21 Jun 2015 11:29:57 -0300
From: Raphael Payen <raphael.payen@gmail.com>
Subject: Re: [Moses-support] Major bug found in Moses
To: "Read, James C" <jcread@essex.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CA+s3C+iE2WNfyMupUdUQ0WJT5jRY8r-YZV3_uaJ4Wi3nD_P-rQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

James, did you try the modifications Philip suggested (removing the word
penalty and lowering p(f|e)?
(I doubt it will be enough to get a best paper award, but it would probably
improve your bleu, that's always a good start :) )



On Friday, June 19, 2015, Read, James C <jcread@essex.ac.uk> wrote:

> So, all I did was filter out the less likely phrase pairs and the BLEU
> score shot up. Was that such a stroke of genius? Was that not blindingly
> obvious?
>
>
> Your telling me that redesigning the search algorithm to prefer higher
> scoring phrase pairs is all we need to do to get a best paper at ACL?
>
>
> James
>
>
> ------------------------------
> *From:* Lane Schwartz <dowobeha@gmail.com
> <javascript:_e(%7B%7D,'cvml','dowobeha@gmail.com');>>
> *Sent:* Friday, June 19, 2015 7:40 PM
> *To:* Read, James C
> *Cc:* Philipp Koehn; Burger, John D.; moses-support@mit.edu
> <javascript:_e(%7B%7D,'cvml','moses-support@mit.edu');>
> *Subject:* Re: [Moses-support] Major bug found in Moses
>
> On Fri, Jun 19, 2015 at 11:28 AM, Read, James C <jcread@essex.ac.uk
> <javascript:_e(%7B%7D,'cvml','jcread@essex.ac.uk');>> wrote:
>
> What I take issue with is the en-masse denial that there is a problem
>> with the system if it behaves in such a way with no LM + no pruning and/or
>> tuning.
>>
>
> There is no mass denial taking place.
>
> Regardless of whether or not you tune, the decoder will do its best to
> find translations with the highest model score. That is the expected
> behavior.
>
> What I have tried to tell you, and what other people have tried to tell
> you, is that translations with high model scores are not necessarily good
> translations.
>
> We all want our models to be such that high model scores correspond to
> good translations, and that low model scores correspond with bad
> translations. But unfortunately, our models do not innately have this
> characteristic. We all know this. We also know a good way to deal with this
> shortcoming, namely tuning. Tuning is the process by which we attempt to
> ensure that high model scores correspond to high quality translations, and
> that low model scores correspond to low quality translations.
>
> If you can design models that naturally correspond with translation
> quality without tuning, that's great. If you can do that, you've got a
> great shot at winning a Best Paper award at ACL.
>
> In the meantime, you may want to consider an apology for your rude
> behavior and unprofessional attitude.
>
> Goodbye.
> Lane
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150621/6a774536/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 104, Issue 67
**********************************************

Tweet