Moses-support Digest, Vol 87, Issue 35

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: C++11 (Marcin Junczys-Dowmunt)
2. Re: static build of mosesdecoder (Tak Kuya)
3. Re: When using moses, cpu is are not stable. (Barry Haddow)
4. Re: Get word-to-word alignments from mosesserver (Hieu Hoang)
5. Factorized corpus in EMS (burak ayd?n)


----------------------------------------------------------------------

Message: 1
Date: Wed, 15 Jan 2014 10:41:01 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] C++11
To: moses-support@mit.edu
Message-ID: <52D657AD.2030702@amu.edu.pl>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

This is called by some very shady code of mine, memory mapping is
involved and stuff. I will take a closer look today.

W dniu 15.01.2014 10:32, Rico Sennrich pisze:
> Marcin Junczys-Dowmunt <junczys@...> writes:
>
>> Revision d2d508184e35909aa5da901b81bb70f10f7794c7 breaks my compact
>> reordering model, but at runtime and only if you do a clean build
>> without any build artifacts from earlier compilations. It segfaults
>> during loading in a weird low-level place.
> Hi Marcin,
>
> strangely enough, the segfault happens in the default constructor of
> unsigned char (no idea why though), which is called during a resize. I could
> fix it by passing the default value 0 to the resize function.
>
> Here's the last bit of the backtrace for anyone interested; maybe someone
> can enlighten me why it's segfaulting:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00000000004c9df5 in _S_construct<unsigned char> (__p=0x7ffff7ff3415
> "\377\003^\255U\v(@\021\003\034\200\206\066\004`\"\221Td\224") at
> /usr/include/c++/4.7/bits/alloc_traits.h:263
> 263 { ::new((void*)__p) _Tp(std::forward<_Args>(__args)...); }
> (gdb) bt
> #0 0x00000000004c9df5 in _S_construct<unsigned char> (__p=0x7ffff7ff3415
> "\377\003^\255U\v(@\021\003\034\200\206\066\004`\"\221Td\224") at
> /usr/include/c++/4.7/bits/alloc_traits.h:263
> #1 construct<unsigned char> (__p=0x7ffff7ff3415
> "\377\003^\255U\v(@\021\003\034\200\206\066\004`\"\221Td\224", __a=...) at
> /usr/include/c++/4.7/bits/alloc_traits.h:395
> #2 __uninitialized_default_n_a<unsigned char*, unsigned long,
> Moses::MmapAllocator<unsigned char> > (__n=<optimized out>,
> __first=<optimized out>, __alloc=...) at
> /usr/include/c++/4.7/bits/stl_uninitialized.h:594
> #3 std::vector<unsigned char, Moses::MmapAllocator<unsigned char>
>> ::_M_default_append (this=this@entry=0x7fffffffc870, __n=<optimized out>)
> at /usr/include/c++/4.7/bits/vector.tcc:558
> #4 0x00000000004caba3 in resize (__new_size=<optimized out>,
> this=0x7fffffffc870) at /usr/include/c++/4.7/bits/stl_vector.h:647
> #5 Moses::StringVector<unsigned char, unsigned long,
> Moses::MmapAllocator>::loadCharArray (this=this@entry=0x9edac8, c=...,
> in=in@entry=0x9ee5d0, map=<optimized out>) at
> moses/TranslationModel/CompactPT/StringVector.h:245
>
> best,
> Rico
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support



------------------------------

Message: 2
Date: Wed, 15 Jan 2014 19:36:48 +0900
From: Tak Kuya <kuyatak@gmail.com>
Subject: Re: [Moses-support] static build of mosesdecoder
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support@mit.edu
Message-ID:
<CANaijnij8fkbwT4q0ZxhsXqhXYojSDPbs0sS584+a9mNbQKgfQ@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

Hi, Hieu. Thank you very much for your advice. First, I made sure if
my CentOS has libz.a and librt.a and it did. Kenneth (Heafiled) taught
me that Boost 1.41 might be too old for the latest Moses building.
(Thank you, Kenneth) So, I installed Boost 1.54 to my home. Then, I
read the build log of Moses once again and inspected the linker
command options and re-installed all the -lxx static libraries. Then I
made it. I also built static version of Rel.1.0 successfully and it
seems to work nicely. The command I used last was:

./bjam -d2 --with-boost=/home/kuya/boost/boost_1_54_0/stage -j5
--static -a variant=release debug-symbols=off

Thank you, again.

Best regards,
Tak Kuya

On Sat, Jan 11, 2014 at 5:23 AM, Hieu Hoang <hieuhoang@gmail.com> wrote:
> double check that libz.a and librt.a exist in
> /usr/lib
> or
> /usr/lib64
> I statically compiled on Suse Linux and it was ok. My bjam command was:
>
> ./bjam --with-boost=/home/s0565741/workspace/boost/boost_1_54_0/
> --with-irstlm=/home/s0565741/workspace/irstlm/
> --with-cmph=/home/s0565741/workspace/cmph-2.0
> --with-dalm=/home/s0565741/workspace/github/DALM -j16 -static -a
>
>
>
>
>
> On 10/01/2014 09:56, Tak Kuya wrote:
>
> Hi. How do you build mosesdecoder using static libraries like
> http://www.statmt.org/moses/RELEASE-1.0/binaries/ ? To make things
> simple, I use the master branch instead of Rel.1.0 and edit
> mosesdecoder/Jamroot a little and build the decoder only.
>
> I can build the normal "shared" version of moses without any problems
> and usually use it (on CentOS 6.5 with boost 1.41). But I failed to
> build moses with --static option. So I installed boost 1.41 to my home
> directory. But I still couldn't build moses. It said "cannot find -lz
> and -lrt" so I installed zlib-static and glibc-static with yum.
>
> And... It failed again. Here are build.log and Jamroot I used. Could
> anyone teach me how to solve the problem? Thank you.
>
> Best regards,
> Tak Kuya
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


------------------------------

Message: 3
Date: Wed, 15 Jan 2014 10:53:03 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] When using moses, cpu is are not stable.
To: Wang Pidong <wangpd@comp.nus.edu.sg>, Hansung Cho
<gyuber@gmail.com>
Cc: moses-support@mit.edu
Message-ID: <n6xlj0lk2uqj88doa6tj9651.1389783183467@email.android.com>
Content-Type: text/plain; charset=utf-8

Hi Wang

That's only true for mosserver. For regular moses and moses_chart, a fixed size thread pool is used.

Cheers - Barry

Wang Pidong <wangpd@comp.nus.edu.sg> wrote:

>The unstable state can be caused by the fact that Moses create a new thread
>to translate each new sentence, so when translating a large file, Moses
>frequently creates threads and then OS has to reschedule frequently also.
>
>Even in single-thread Moses, given each input sentence, it also needs to
>load phrase table, language model, etc. which may also be blocked due to IO
>operations, if you use binarized models.
>
>Just for your reference.
>
>Best wishes!
>Pidong
>
>
>
>On 14 January 2014 23:24, Hansung Cho <gyuber@gmail.com> wrote:
>
>> hello.
>>
>> To change the option "-threads 64" then monitoring at server CPUs status.
>> but, result is the same..
>> Try the other way?
>>
>> Attach the CPUs status picture
>>
>> thank you
>> Hansung Cho
>>
>> [image: ?? ??? 1]
>>
>>
>> 2014/1/15 Tom Hoar <tahoar@precisiontranslationtools.com>
>>
>>> To change the threads, run with '-threads x' added to your command line:
>>>
>>> moses -f /path/to/moses.ini -threads 20
>>>
>>>
>>>
>>>
>>> On 01/14/2014 11:41 PM, Philipp Koehn wrote:
>>>
>>> Hi,
>>>
>>> you can try specifying more threads than the number of CPUs.
>>> I have no idea if this speeds things up or causes blockage in
>>> your setup. Just try it out and see what happens.
>>>
>>> -phi
>>>
>>>
>>> On Tue, Jan 14, 2014 at 4:31 AM, Hansung Cho <gyuber@gmail.com> wrote:
>>>
>>>> My server configuration are multiple CPUs 16 core and running thread 16.
>>>>
>>>> How do I adjust the moses running thread?
>>>>
>>>> I need your help...
>>>>
>>>> Thank you
>>>>
>>>>
>>>> 2014/1/14 Tom Hoar <tahoar@precisiontranslationtools.com>
>>>>
>>>>> With a system as complex as Moses, why would you think the CPU usage
>>>>> would be steady? Each sentence task goes through many stages with different
>>>>> CPU demands which probably map to the variation in each cycle.
>>>>>
>>>>> You can also look first at your threading configuration. If your system
>>>>> has multiple CPUs (cores) and Moses is running multi-threaded, I suspect
>>>>> the cycles' frequencies are related to the activities on each thread.
>>>>>
>>>>>
>>>>>
>>>>> On 01/14/2014 01:09 PM, Hansung Cho wrote:
>>>>>
>>>>> hello..
>>>>>
>>>>> I'm using mosesdecoding
>>>>>
>>>>> When monitoring a physical server, Server cpu is not very stable.
>>>>> However, the state is constantly moving cpu
>>>>>
>>>>> Look at the picture below
>>>>> [image: ?? ??? 1]
>>>>>
>>>>> Is it because the moses system cache??
>>>>> Should any part option of the adjustment?
>>>>>
>>>>> My moses commit a1584c608f8bbd83921896bac9eac97d9effc0c8
>>>>> Options related to the cache are "clean-lm-cache" ( default ==1 )
>>>>>
>>>>> Will be waiting for your input.
>>>>>
>>>>> Thank you
>>>>> Hansung cho
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> --
>> Wang Pidong
>>
>> Department of Computer Science
>> School of Computing
>> National University of Singapore
>>
>>
>
>_______________________________________________
>Moses-support mailing list
>Moses-support@mit.edu
>http://mailman.mit.edu/mailman/listinfo/moses-support

--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




------------------------------

Message: 4
Date: Wed, 15 Jan 2014 11:52:28 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Get word-to-word alignments from
mosesserver
To: Jyotesh Choudhari <jyoteshrc@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbgraDww5e93LYHJA+Y7s4mSWf-m1BBWTnRRdFLRJjeK-w@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

I don't think it's been implemented. However, if you want to implement it,
I can help you. Please email me offline and I'll show you how


On 13 January 2014 13:44, Jyotesh Choudhari <jyoteshrc@gmail.com> wrote:

> Hi,
> According to Moses tutorial
> (http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc30),
> mosesserver
> returns only phrase alignments. I want word alignments too. How can I get
> these?
>
> Thanks and regards,
> Jyotesh Choudhari
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140115/a15f1f0b/attachment-0001.htm

------------------------------

Message: 5
Date: Wed, 15 Jan 2014 15:51:39 +0200
From: burak ayd?n <baydinx@gmail.com>
Subject: [Moses-support] Factorized corpus in EMS
To: moses-support@mit.edu
Message-ID:
<CAH+r-S++8BBKssp8D71=qSeavNqVDnPzk-4E2tOT7Wa5Xgirwg@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi all,

In moses web page, there are instructions for factored training. What I
want to to experiment is to give already factorized corpus. What will be
the new factor definition and new Lm definition in the EMS config?

When I try to replicate the original config.factor experiment, I observe
that there are files created under corpus/ directory :

factored.<expnum>.en
factored.<expnum>.en.pos

So in order to give factorized corpus as input, should I also prepare one
or more of the files above and write it in the EMS config as well?

Regards
Burak
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140115/417bd5eb/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 87, Issue 35
*********************************************

Tweet