Moses-support Digest, Vol 87, Issue 32

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: When using moses, cpu is are not stable.
(Marcin Junczys-Dowmunt)


----------------------------------------------------------------------

Message: 1
Date: Wed, 15 Jan 2014 09:40:48 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] When using moses, cpu is are not stable.
To: moses-support@mit.edu
Message-ID: <52D64990.4000202@amu.edu.pl>
Content-Type: text/plain; charset="iso-8859-1"

From my experience more than 16 threads per moses process causes
performance to decrease. I have 64 cores, too. I use GNU parallel to get
around this.

cat input.txt | parallel --pipe -k -j 4 ./mosesdecoder/bin/moses -f
moses.ini -threads 16 > out.txt

This launches 4 moses processes a 16 threads and translates blocks of
roughly 1MB per processes. You only see output after a block has been
translated. You can set block size by using for instance --block 10K ,
then it will create and kill the moses processes much faster and output
intermediate results.

You can get GNU parallel from
http://ftp.gnu.org/gnu/parallel/parallel-latest.tar.bz2 .
Make sure to delete /etc/parallel/config after installation.
Best,
Marcin

W dniu 15.01.2014 08:24, Hansung Cho pisze:
> hello.
>
> To change the option "-threads 64" then monitoring at server CPUs status.
> but, result is the same..
> Try the other way?
>
> Attach the CPUs status picture
>
> thank you
> Hansung Cho
>
> ?? ??? 1
>
>
> 2014/1/15 Tom Hoar <tahoar@precisiontranslationtools.com
> <mailto:tahoar@precisiontranslationtools.com>>
>
> To change the threads, run with '-threads x' added to your command
> line:
>
> moses -f /path/to/moses.ini -threads 20
>
>
>
>
> On 01/14/2014 11:41 PM, Philipp Koehn wrote:
>> Hi,
>>
>> you can try specifying more threads than the number of CPUs.
>> I have no idea if this speeds things up or causes blockage in
>> your setup. Just try it out and see what happens.
>>
>> -phi
>>
>>
>> On Tue, Jan 14, 2014 at 4:31 AM, Hansung Cho <gyuber@gmail.com
>> <mailto:gyuber@gmail.com>> wrote:
>>
>> My server configuration are multiple CPUs 16 core and running
>> thread 16.
>>
>> How do I adjust the moses running thread?
>>
>> I need your help...
>>
>> Thank you
>>
>>
>> 2014/1/14 Tom Hoar <tahoar@precisiontranslationtools.com
>> <mailto:tahoar@precisiontranslationtools.com>>
>>
>> With a system as complex as Moses, why would you think
>> the CPU usage would be steady? Each sentence task goes
>> through many stages with different CPU demands which
>> probably map to the variation in each cycle.
>>
>> You can also look first at your threading configuration.
>> If your system has multiple CPUs (cores) and Moses is
>> running multi-threaded, I suspect the cycles' frequencies
>> are related to the activities on each thread.
>>
>>
>>
>> On 01/14/2014 01:09 PM, Hansung Cho wrote:
>>> hello..
>>>
>>> I'm using mosesdecoding
>>>
>>> When monitoring a physical server, Server cpu is not
>>> very stable.
>>> However, the state is constantly moving cpu
>>>
>>> Look at the picture below
>>> ?? ??? 1
>>>
>>> Is it because the moses system cache??
>>> Should any part option of the adjustment?
>>>
>>> My moses commit a1584c608f8bbd83921896bac9eac97d9effc0c8
>>> Options related to the cache are "clean-lm-cache" (
>>> default ==1 )
>>>
>>> Will be waiting for your input.
>>>
>>> Thank you
>>> Hansung cho
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140115/a9f5e447/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 4432 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140115/a9f5e447/attachment.png
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 4887 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140115/a9f5e447/attachment-0001.png

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 87, Issue 32
*********************************************

Tweet