Moses-support Digest, Vol 104, Issue 68

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Exception when exiting moses (Marcin Junczys-Dowmunt)
2. Re: Exception when exiting moses (Marcin Junczys-Dowmunt)
3. Re: How to force not to reuse certain files in EMS (Philipp Koehn)
4. Re: How to force not to reuse certain files in EMS (Dingyuan Wang)


----------------------------------------------------------------------

Message: 1
Date: Sun, 21 Jun 2015 16:31:26 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Exception when exiting moses
To: Jeroen Vermeulen <jtv@precisiontranslationtools.com>,
moses-support@mit.edu
Message-ID: <5586CABE.7060107@amu.edu.pl>
Content-Type: text/plain; charset=UTF-8; format=flowed

I don't think it's your fault. It seems an incorrect copy of
mmapallocator is created somewhere that tries to unmap itself at
destruction. Since the file has already been unmapped it throws an
exception. I am still not sure where that copy is coming from. Maybe
that's due to c++11? Maybe that new emplacement constructor.

On 21.06.2015 16:26, Jeroen Vermeulen wrote:
> Hope I didn't break the mmap allocator - though that would have been some time back anyway.
>
> It does remind me that Boost has a portable mmap wrapper. It's in the iostreams library. Might be worth trying to replace kenlm's wrappers with that one to reduce maintenance burden.
>
>
> Jeroen
>
> On June 21, 2015 8:47:50 PM GMT+07:00, Marcin Junczys-Dowmunt <junczys@amu.edu.pl> wrote:
>> Recompiling it just now with debug on. There is already a mistake in
>>
>> https://github.com/moses-smt/mosesdecoder/blob/master/moses/TranslationModel/CompactPT/MmapAllocator.h#L156
>>
>> The size is wrong, the offset is missing. But it wasn't that.
>>
>> #0 0x00007ffff6e05cc9 in __GI_raise (sig=sig@entry=6) at
>> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>> #1 0x00007ffff6e090d8 in __GI_abort () at abort.c:89
>> #2 0x00007ffff792e6b5 in __gnu_cxx::__verbose_terminate_handler() ()
>> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> #3 0x00007ffff792c836 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> #4 0x00007ffff792b8f9 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> #5 0x00007ffff792c4aa in __gxx_personality_v0 () from
>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>> #6 0x00007ffff73c1ff3 in ?? () from
>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>> #7 0x00007ffff73c2517 in _Unwind_Resume () from
>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>> #8 0x00000000004798a7 in util::UnmapOrThrow (start=<optimized out>,
>> length=<optimized out>)
>> at util/mmap.cc:52
>> #9 0x00000000005552c1 in _M_deallocate (__n=<optimized out>,
>> __p=<optimized out>, this=0x12ba3a0)
>> at /usr/include/c++/4.8/bits/stl_vector.h:174
>> #10 ~_Vector_base (this=0x12ba3a0, __in_chrg=<optimized out>)
>> at /usr/include/c++/4.8/bits/stl_vector.h:160
>> #11 ~vector (this=0x12ba3a0, __in_chrg=<optimized out>) at
>> /usr/include/c++/4.8/bits/stl_vector.h:416
>> #12 Moses::StringVector<unsigned char, unsigned long,
>> Moses::MmapAllocator>::~StringVector (
>> this=0x2746458, __in_chrg=<optimized out>) at
>> moses/TranslationModel/CompactPT/StringVector.h:154
>> #13 0x000000000055862e in
>> Moses::LexicalReorderingTableCompact::~LexicalReorderingTableCompact (
>> this=0x2746000, __in_chrg=<optimized out>)
>> at
>> moses/TranslationModel/CompactPT/LexicalReorderingTableCompact.cpp:56
>> #14 0x0000000000558669 in
>> Moses::LexicalReorderingTableCompact::~LexicalReorderingTableCompact (
>> this=0x2746000, __in_chrg=<optimized out>)
>> at
>> moses/TranslationModel/CompactPT/LexicalReorderingTableCompact.cpp:60
>> #15 0x00000000005269b3 in checked_delete<Moses::LexicalReorderingTable>
>>
>> (x=<optimized out>)
>> at /usr/include/boost/checked_delete.hpp:34
>> #16 ~scoped_ptr (this=0x12c4408, __in_chrg=<optimized out>)
>> at /usr/include/boost/smart_ptr/scoped_ptr.hpp:82
>> #17 Moses::LexicalReordering::~LexicalReordering (this=0x12c4360,
>> __in_chrg=<optimized out>)
>> at moses/FF/LexicalReordering/LexicalReordering.cpp:82
>> #18 0x0000000000526b99 in Moses::LexicalReordering::~LexicalReordering
>> (this=0x12c4360,
>> __in_chrg=<optimized out>) at
>> moses/FF/LexicalReordering/LexicalReordering.cpp:83
>> #19 0x000000000045ea06 in
>> RemoveAllInColl<std::vector<Moses::FeatureFunction*> > (coll=...)
>> at ./moses/Util.h:436
>> #20 Moses::FeatureFunction::Destroy () at
>> moses/FF/FeatureFunction.cpp:38
>> #21 0x0000000000417157 in batch_run () at moses/ExportInterface.cpp:274
>> #22 0x00000000004184d8 in decoder_main (argc=5, argv=0x7fffffffdef8) at
>>
>> moses/ExportInterface.cpp:327
>> #23 0x00007ffff6df0ec5 in __libc_start_main (main=0x40c6b0 <main(int,
>> char**)>, argc=5,
>> argv=0x7fffffffdef8, init=<optimized out>, fini=<optimized out>,
>> rtld_fini=<optimized out>,
>> stack_end=0x7fffffffdee8) at libc-start.c:287
>> #24 0x0000000000415d2a in _start ()
>>
>>
>> On 21.06.2015 15:44, Kenneth Heafield wrote:
>>> What exception? I can haz stack trace?
>>>
>>> On 06/21/2015 09:02 AM, Marcin Junczys-Dowmunt wrote:
>>>> Hi,
>>>> is anyone else getting exceptions when moses exits with the latest
>>>> master? It seems to be happening in my reordering table and breaks
>> MERT.
>>>> Wasn't me though :)
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support



------------------------------

Message: 2
Date: Sun, 21 Jun 2015 16:38:19 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Exception when exiting moses
To: moses-support@mit.edu
Message-ID: <5586CC5B.2070507@amu.edu.pl>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Yay, error disappears with c++11 disabled. Now I need to find out what
is going on. So I am the first C++11 victim after all although I was so
happy to welcome it :) Karma.

On 21.06.2015 16:31, Marcin Junczys-Dowmunt wrote:
> I don't think it's your fault. It seems an incorrect copy of
> mmapallocator is created somewhere that tries to unmap itself at
> destruction. Since the file has already been unmapped it throws an
> exception. I am still not sure where that copy is coming from. Maybe
> that's due to c++11? Maybe that new emplacement constructor.
>
> On 21.06.2015 16:26, Jeroen Vermeulen wrote:
>> Hope I didn't break the mmap allocator - though that would have been some time back anyway.
>>
>> It does remind me that Boost has a portable mmap wrapper. It's in the iostreams library. Might be worth trying to replace kenlm's wrappers with that one to reduce maintenance burden.
>>
>>
>> Jeroen
>>
>> On June 21, 2015 8:47:50 PM GMT+07:00, Marcin Junczys-Dowmunt <junczys@amu.edu.pl> wrote:
>>> Recompiling it just now with debug on. There is already a mistake in
>>>
>>> https://github.com/moses-smt/mosesdecoder/blob/master/moses/TranslationModel/CompactPT/MmapAllocator.h#L156
>>>
>>> The size is wrong, the offset is missing. But it wasn't that.
>>>
>>> #0 0x00007ffff6e05cc9 in __GI_raise (sig=sig@entry=6) at
>>> ../nptl/sysdeps/unix/sysv/linux/raise.c:56
>>> #1 0x00007ffff6e090d8 in __GI_abort () at abort.c:89
>>> #2 0x00007ffff792e6b5 in __gnu_cxx::__verbose_terminate_handler() ()
>>> from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> #3 0x00007ffff792c836 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> #4 0x00007ffff792b8f9 in ?? () from
>>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> #5 0x00007ffff792c4aa in __gxx_personality_v0 () from
>>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6
>>> #6 0x00007ffff73c1ff3 in ?? () from
>>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>>> #7 0x00007ffff73c2517 in _Unwind_Resume () from
>>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>>> #8 0x00000000004798a7 in util::UnmapOrThrow (start=<optimized out>,
>>> length=<optimized out>)
>>> at util/mmap.cc:52
>>> #9 0x00000000005552c1 in _M_deallocate (__n=<optimized out>,
>>> __p=<optimized out>, this=0x12ba3a0)
>>> at /usr/include/c++/4.8/bits/stl_vector.h:174
>>> #10 ~_Vector_base (this=0x12ba3a0, __in_chrg=<optimized out>)
>>> at /usr/include/c++/4.8/bits/stl_vector.h:160
>>> #11 ~vector (this=0x12ba3a0, __in_chrg=<optimized out>) at
>>> /usr/include/c++/4.8/bits/stl_vector.h:416
>>> #12 Moses::StringVector<unsigned char, unsigned long,
>>> Moses::MmapAllocator>::~StringVector (
>>> this=0x2746458, __in_chrg=<optimized out>) at
>>> moses/TranslationModel/CompactPT/StringVector.h:154
>>> #13 0x000000000055862e in
>>> Moses::LexicalReorderingTableCompact::~LexicalReorderingTableCompact (
>>> this=0x2746000, __in_chrg=<optimized out>)
>>> at
>>> moses/TranslationModel/CompactPT/LexicalReorderingTableCompact.cpp:56
>>> #14 0x0000000000558669 in
>>> Moses::LexicalReorderingTableCompact::~LexicalReorderingTableCompact (
>>> this=0x2746000, __in_chrg=<optimized out>)
>>> at
>>> moses/TranslationModel/CompactPT/LexicalReorderingTableCompact.cpp:60
>>> #15 0x00000000005269b3 in checked_delete<Moses::LexicalReorderingTable>
>>>
>>> (x=<optimized out>)
>>> at /usr/include/boost/checked_delete.hpp:34
>>> #16 ~scoped_ptr (this=0x12c4408, __in_chrg=<optimized out>)
>>> at /usr/include/boost/smart_ptr/scoped_ptr.hpp:82
>>> #17 Moses::LexicalReordering::~LexicalReordering (this=0x12c4360,
>>> __in_chrg=<optimized out>)
>>> at moses/FF/LexicalReordering/LexicalReordering.cpp:82
>>> #18 0x0000000000526b99 in Moses::LexicalReordering::~LexicalReordering
>>> (this=0x12c4360,
>>> __in_chrg=<optimized out>) at
>>> moses/FF/LexicalReordering/LexicalReordering.cpp:83
>>> #19 0x000000000045ea06 in
>>> RemoveAllInColl<std::vector<Moses::FeatureFunction*> > (coll=...)
>>> at ./moses/Util.h:436
>>> #20 Moses::FeatureFunction::Destroy () at
>>> moses/FF/FeatureFunction.cpp:38
>>> #21 0x0000000000417157 in batch_run () at moses/ExportInterface.cpp:274
>>> #22 0x00000000004184d8 in decoder_main (argc=5, argv=0x7fffffffdef8) at
>>>
>>> moses/ExportInterface.cpp:327
>>> #23 0x00007ffff6df0ec5 in __libc_start_main (main=0x40c6b0 <main(int,
>>> char**)>, argc=5,
>>> argv=0x7fffffffdef8, init=<optimized out>, fini=<optimized out>,
>>> rtld_fini=<optimized out>,
>>> stack_end=0x7fffffffdee8) at libc-start.c:287
>>> #24 0x0000000000415d2a in _start ()
>>>
>>>
>>> On 21.06.2015 15:44, Kenneth Heafield wrote:
>>>> What exception? I can haz stack trace?
>>>>
>>>> On 06/21/2015 09:02 AM, Marcin Junczys-Dowmunt wrote:
>>>>> Hi,
>>>>> is anyone else getting exceptions when moses exits with the latest
>>>>> master? It seems to be happening in my reordering table and breaks
>>> MERT.
>>>>> Wasn't me though :)
>>>>>
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support



------------------------------

Message: 3
Date: Sun, 21 Jun 2015 11:06:18 -0400
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] How to force not to reuse certain files
in EMS
To: Dingyuan Wang <abcdoyle888@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAAFADDBwTTsxW0LGn3k8KbE=OmFvpNOYrqgTd9NMQbzQZqwOYg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

EMS also looks at the time stamps of the files, so it should
re-run experiments if the time stamp of a specified file
changed.

Are you observing something different?

-phi

On Sun, Jun 21, 2015 at 9:23 AM, Dingyuan Wang <abcdoyle888@gmail.com>
wrote:

> Hi,
>
> In my experiments the main variable is the corpus text generated
> before running the EMS. Sometimes the pre-built LM changes outside the
> EMS too.
> The EMS system checkes config file changes to determine which files
> can be reused in the next experiment step. However it's a waste of
> time and resources to change the filenames of the corpus and the LM
> every time, or rerun the entire experiment, or go through the baseline
> system manually.
> I can hardly read Perl, but I think there is an easy way to make this
> possible. Maybe an option or timestamp check can help.
>
> Thanks.
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150621/531eeb76/attachment-0001.htm

------------------------------

Message: 4
Date: Sun, 21 Jun 2015 23:30:26 +0800
From: Dingyuan Wang <abcdoyle888@gmail.com>
Subject: Re: [Moses-support] How to force not to reuse certain files
in EMS
To: Philipp Koehn <phi@jhu.edu>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <5586D892.1010809@gmail.com>
Content-Type: text/plain; charset="utf-8"

Yes. I do know EMS checkes time stamps, but it doesn't check time stamps
of original corpus or LM. I just changed a word in my corpus file and
changed and `touch`ed the LM, but it refuses to run again as usual.

I've attached the config file, the files in question are:
[CORPUS:toy]/raw-stem
[LM]/lm


? 2015?06?21? 23:06, Philipp Koehn ??:
> Hi,
>
> EMS also looks at the time stamps of the files, so it should
> re-run experiments if the time stamp of a specified file
> changed.
>
> Are you observing something different?
>
> -phi
>
> On Sun, Jun 21, 2015 at 9:23 AM, Dingyuan Wang <abcdoyle888@gmail.com
> <mailto:abcdoyle888@gmail.com>> wrote:
>
> Hi,
>
> In my experiments the main variable is the corpus text generated
> before running the EMS. Sometimes the pre-built LM changes outside the
> EMS too.
> The EMS system checkes config file changes to determine which files
> can be reused in the next experiment step. However it's a waste of
> time and resources to change the filenames of the corpus and the LM
> every time, or rerun the entire experiment, or go through the baseline
> system manually.
> I can hardly read Perl, but I think there is an easy way to make this
> possible. Maybe an option or timestamp check can help.
>
> Thanks.
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zhc2zhm.ini
Type: application/x-wine-extension-ini
Size: 20205 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20150621/dc6de6d2/attachment.bin

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 104, Issue 68
**********************************************

Tweet