Moses-support Digest, Vol 87, Issue 34

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: C++11 (Rico Sennrich)
2. Re: When using moses, cpu is are not stable.
(Marcin Junczys-Dowmunt)


----------------------------------------------------------------------

Message: 1
Date: Wed, 15 Jan 2014 09:32:51 +0000 (UTC)
From: Rico Sennrich <rico.sennrich@gmx.ch>
Subject: Re: [Moses-support] C++11
To: moses-support@MIT.EDU
Message-ID: <loom.20140115T102746-588@post.gmane.org>
Content-Type: text/plain; charset=us-ascii

Marcin Junczys-Dowmunt <junczys@...> writes:

> Revision d2d508184e35909aa5da901b81bb70f10f7794c7 breaks my compact
> reordering model, but at runtime and only if you do a clean build
> without any build artifacts from earlier compilations. It segfaults
> during loading in a weird low-level place.

Hi Marcin,

strangely enough, the segfault happens in the default constructor of
unsigned char (no idea why though), which is called during a resize. I could
fix it by passing the default value 0 to the resize function.

Here's the last bit of the backtrace for anyone interested; maybe someone
can enlighten me why it's segfaulting:

Program received signal SIGSEGV, Segmentation fault.
0x00000000004c9df5 in _S_construct<unsigned char> (__p=0x7ffff7ff3415
"\377\003^\255U\v(@\021\003\034\200\206\066\004`\"\221Td\224") at
/usr/include/c++/4.7/bits/alloc_traits.h:263
263 { ::new((void*)__p) _Tp(std::forward<_Args>(__args)...); }
(gdb) bt
#0 0x00000000004c9df5 in _S_construct<unsigned char> (__p=0x7ffff7ff3415
"\377\003^\255U\v(@\021\003\034\200\206\066\004`\"\221Td\224") at
/usr/include/c++/4.7/bits/alloc_traits.h:263
#1 construct<unsigned char> (__p=0x7ffff7ff3415
"\377\003^\255U\v(@\021\003\034\200\206\066\004`\"\221Td\224", __a=...) at
/usr/include/c++/4.7/bits/alloc_traits.h:395
#2 __uninitialized_default_n_a<unsigned char*, unsigned long,
Moses::MmapAllocator<unsigned char> > (__n=<optimized out>,
__first=<optimized out>, __alloc=...) at
/usr/include/c++/4.7/bits/stl_uninitialized.h:594
#3 std::vector<unsigned char, Moses::MmapAllocator<unsigned char>
>::_M_default_append (this=this@entry=0x7fffffffc870, __n=<optimized out>)
at /usr/include/c++/4.7/bits/vector.tcc:558
#4 0x00000000004caba3 in resize (__new_size=<optimized out>,
this=0x7fffffffc870) at /usr/include/c++/4.7/bits/stl_vector.h:647
#5 Moses::StringVector<unsigned char, unsigned long,
Moses::MmapAllocator>::loadCharArray (this=this@entry=0x9edac8, c=...,
in=in@entry=0x9ee5d0, map=<optimized out>) at
moses/TranslationModel/CompactPT/StringVector.h:245

best,
Rico




------------------------------

Message: 2
Date: Wed, 15 Jan 2014 10:38:35 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] When using moses, cpu is are not stable.
To: moses-support@mit.edu
Message-ID: <52D6571B.90108@amu.edu.pl>
Content-Type: text/plain; charset="iso-8859-1"

I forgot to add that I am using cube-pruning, with stack decoding moses
uses pretty much all the available cores in one process. Cube-pruning
seems to be so fast that lockage becomes an issue.

W dniu 15.01.2014 09:40, Marcin Junczys-Dowmunt pisze:
> From my experience more than 16 threads per moses process causes
> performance to decrease. I have 64 cores, too. I use GNU parallel to
> get around this.
>
> cat input.txt | parallel --pipe -k -j 4 ./mosesdecoder/bin/moses -f
> moses.ini -threads 16 > out.txt
>
> This launches 4 moses processes a 16 threads and translates blocks of
> roughly 1MB per processes. You only see output after a block has been
> translated. You can set block size by using for instance --block 10K ,
> then it will create and kill the moses processes much faster and
> output intermediate results.
>
> You can get GNU parallel from
> http://ftp.gnu.org/gnu/parallel/parallel-latest.tar.bz2 .
> Make sure to delete /etc/parallel/config after installation.
> Best,
> Marcin
>
> W dniu 15.01.2014 08:24, Hansung Cho pisze:
>> hello.
>>
>> To change the option "-threads 64" then monitoring at server CPUs status.
>> but, result is the same..
>> Try the other way?
>>
>> Attach the CPUs status picture
>>
>> thank you
>> Hansung Cho
>>
>> ?? ??? 1
>>
>>
>> 2014/1/15 Tom Hoar <tahoar@precisiontranslationtools.com
>> <mailto:tahoar@precisiontranslationtools.com>>
>>
>> To change the threads, run with '-threads x' added to your
>> command line:
>>
>> moses -f /path/to/moses.ini -threads 20
>>
>>
>>
>>
>> On 01/14/2014 11:41 PM, Philipp Koehn wrote:
>>> Hi,
>>>
>>> you can try specifying more threads than the number of CPUs.
>>> I have no idea if this speeds things up or causes blockage in
>>> your setup. Just try it out and see what happens.
>>>
>>> -phi
>>>
>>>
>>> On Tue, Jan 14, 2014 at 4:31 AM, Hansung Cho <gyuber@gmail.com
>>> <mailto:gyuber@gmail.com>> wrote:
>>>
>>> My server configuration are multiple CPUs 16 core and
>>> running thread 16.
>>>
>>> How do I adjust the moses running thread?
>>>
>>> I need your help...
>>>
>>> Thank you
>>>
>>>
>>> 2014/1/14 Tom Hoar <tahoar@precisiontranslationtools.com
>>> <mailto:tahoar@precisiontranslationtools.com>>
>>>
>>> With a system as complex as Moses, why would you think
>>> the CPU usage would be steady? Each sentence task goes
>>> through many stages with different CPU demands which
>>> probably map to the variation in each cycle.
>>>
>>> You can also look first at your threading configuration.
>>> If your system has multiple CPUs (cores) and Moses is
>>> running multi-threaded, I suspect the cycles'
>>> frequencies are related to the activities on each thread.
>>>
>>>
>>>
>>> On 01/14/2014 01:09 PM, Hansung Cho wrote:
>>>> hello..
>>>>
>>>> I'm using mosesdecoding
>>>>
>>>> When monitoring a physical server, Server cpu is not
>>>> very stable.
>>>> However, the state is constantly moving cpu
>>>>
>>>> Look at the picture below
>>>> ?? ??? 1
>>>>
>>>> Is it because the moses system cache??
>>>> Should any part option of the adjustment?
>>>>
>>>> My moses commit a1584c608f8bbd83921896bac9eac97d9effc0c8
>>>> Options related to the cache are "clean-lm-cache" (
>>>> default ==1 )
>>>>
>>>> Will be waiting for your input.
>>>>
>>>> Thank you
>>>> Hansung cho
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140115/12978c7f/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 4432 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140115/12978c7f/attachment.png
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 4887 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140115/12978c7f/attachment-0001.png

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 87, Issue 34
*********************************************

Tweet