Moses-support Digest, Vol 106, Issue 2

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Parallelizer multi core (Vincent Nguyen)
2. Re: Parallelizer multi core (Hieu Hoang)
3. Language model creation error (kalu mera)


----------------------------------------------------------------------

Message: 1
Date: Sat, 1 Aug 2015 09:48:17 +0200
From: Vincent Nguyen <vnguyen@neuf.fr>
Subject: Re: [Moses-support] Parallelizer multi core
To: moses-support@mit.edu
Message-ID: <55BC79C1.3090100@neuf.fr>
Content-Type: text/plain; charset=windows-1252; format=flowed

fair enough.
One thing though ....
When you use irstlm for LM and training, then the EMS crashes in multi
thread at tuning (decoder)

What is the easiest way so that the tuning part uses KenLM each time
multi threads is activated ? (I mean in EMS).

Vincent




Le 01/08/2015 09:38, Hieu Hoang a ?crit :
> oh alright. I've made it 4 cores. The example config files are aimed at
> beginners with laptops.
>
> On 01/08/2015 10:35, Marcin Junczys-Dowmunt wrote:
>> Hi, I agree with Nick. I am using a 64-core machine. "-threads all" will
>> grind to a still-stand. I am however fine with a few more threads, say 16.
>> Best,
>> Marcin
>>
>> On 01.08.2015 00:35, Nikolay Bogoychev wrote:
>>> Hey,
>>>
>>> I have opposed this change in the past for two reasons:
>>>
>>> Using more than 4 threads doesn't help unless the user is using
>>> PhraseDictionaryCompact. See this issue
>>> https://github.com/moses-smt/mosesdecoder/issues/39 in fact on most
>>> machines you rarely want to run moses on all available threads.
>>>
>>> Also - threads all picks up virtual (hyper) threads which are in fact
>>> harmful to performance.
>>>
>>> If you want to change the default I think it would be better to have a
>>> sane default like 4.. It would boost performance for most people and
>>> if you run it on machines with less available cores it would be not
>>> too bad.
>>>
>>> Cheers,
>>>
>>> Nick
>>>
>>> On 31 Jul 2015 7:31 pm, "Hieu Hoang" <hieuhoang@gmail.com
>>> <mailto:hieuhoang@gmail.com>> wrote:
>>>
>>> good suggestion. Changed:
>>> https://github.com/moses-smt/mosesdecoder/commit/f894dec0fd8d5b15eb16c35d3d2599338894ee9d
>>> if you have any more suggestions, it's best if you can just me a
>>> patch and I'll check it in
>>>
>>> On 31/07/2015 15:59, Vincent Nguyen wrote:
>>>> for inexperienced people like me :)
>>>> Add |--decoder-flags="-threads 4"| is key
>>>>
>>>> if EMS config.basic had "-threads all" by default we would gain A
>>>> LOT of time.
>>>>
>>>> cheers,
>>>>
>>>> Vincent
>>>>
>>>>
>>>> Le 29/07/2015 22:05, Vincent Nguyen a ?crit :
>>>>> Hi,
>>>>>
>>>>> I am wondering what tasks of the EMS are really parallelized.
>>>>> I activated the script line + 8 cores.
>>>>>
>>>>> Training / binarizing / Tuning all make only one core to actually work.
>>>>>
>>>>> Am I correct ?
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>> --
>>> Hieu Hoang
>>> Researcher
>>> New York University, Abu Dhabi
>>> http://www.hoang.co.uk/hieu
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>


------------------------------

Message: 2
Date: Sat, 1 Aug 2015 11:53:38 +0400
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Parallelizer multi core
To: moses-support@mit.edu
Message-ID: <55BC7B02.7050908@gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed

in the [LM] section, make sure
type = 8


On 01/08/2015 11:48, Vincent Nguyen wrote:
> fair enough.
> One thing though ....
> When you use irstlm for LM and training, then the EMS crashes in multi
> thread at tuning (decoder)
>
> What is the easiest way so that the tuning part uses KenLM each time
> multi threads is activated ? (I mean in EMS).
>
> Vincent
>
>
>
>
> Le 01/08/2015 09:38, Hieu Hoang a ?crit :
>> oh alright. I've made it 4 cores. The example config files are aimed at
>> beginners with laptops.
>>
>> On 01/08/2015 10:35, Marcin Junczys-Dowmunt wrote:
>>> Hi, I agree with Nick. I am using a 64-core machine. "-threads all" will
>>> grind to a still-stand. I am however fine with a few more threads, say 16.
>>> Best,
>>> Marcin
>>>
>>> On 01.08.2015 00:35, Nikolay Bogoychev wrote:
>>>> Hey,
>>>>
>>>> I have opposed this change in the past for two reasons:
>>>>
>>>> Using more than 4 threads doesn't help unless the user is using
>>>> PhraseDictionaryCompact. See this issue
>>>> https://github.com/moses-smt/mosesdecoder/issues/39 in fact on most
>>>> machines you rarely want to run moses on all available threads.
>>>>
>>>> Also - threads all picks up virtual (hyper) threads which are in fact
>>>> harmful to performance.
>>>>
>>>> If you want to change the default I think it would be better to have a
>>>> sane default like 4.. It would boost performance for most people and
>>>> if you run it on machines with less available cores it would be not
>>>> too bad.
>>>>
>>>> Cheers,
>>>>
>>>> Nick
>>>>
>>>> On 31 Jul 2015 7:31 pm, "Hieu Hoang" <hieuhoang@gmail.com
>>>> <mailto:hieuhoang@gmail.com>> wrote:
>>>>
>>>> good suggestion. Changed:
>>>> https://github.com/moses-smt/mosesdecoder/commit/f894dec0fd8d5b15eb16c35d3d2599338894ee9d
>>>> if you have any more suggestions, it's best if you can just me a
>>>> patch and I'll check it in
>>>>
>>>> On 31/07/2015 15:59, Vincent Nguyen wrote:
>>>>> for inexperienced people like me :)
>>>>> Add |--decoder-flags="-threads 4"| is key
>>>>>
>>>>> if EMS config.basic had "-threads all" by default we would gain A
>>>>> LOT of time.
>>>>>
>>>>> cheers,
>>>>>
>>>>> Vincent
>>>>>
>>>>>
>>>>> Le 29/07/2015 22:05, Vincent Nguyen a ?crit :
>>>>>> Hi,
>>>>>>
>>>>>> I am wondering what tasks of the EMS are really parallelized.
>>>>>> I activated the script line + 8 cores.
>>>>>>
>>>>>> Training / binarizing / Tuning all make only one core to actually work.
>>>>>>
>>>>>> Am I correct ?
>>>>>> _______________________________________________
>>>>>> Moses-support mailing list
>>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>> --
>>>> Hieu Hoang
>>>> Researcher
>>>> New York University, Abu Dhabi
>>>> http://www.hoang.co.uk/hieu
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>

--
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu



------------------------------

Message: 3
Date: Sat, 1 Aug 2015 11:17:23 +0300
From: kalu mera <kalumera7@gmail.com>
Subject: [Moses-support] Language model creation error
To: moses-support@mit.edu
Message-ID:
<CAHGhZRi6EW3G52vm-t7VkhaC-va9og2zkquUZGJJYNp=PriRSA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Dear Members,
I am trying to create a language model creation, I entered this command
kalumera@kalumera-Satellite-C50-A534:~/mosesdecoder$ ./bjam
--with-boost=~/workspace/temp/boost_1_55_0 -j4

but the build failed

Please check the attachment for the command i entered and the error, and
help advise me on how to rectify the problem

Christine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150801/bfde7a1c/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: language model creation.odt
Type: application/vnd.oasis.opendocument.text
Size: 59806 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20150801/bfde7a1c/attachment.bin

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 106, Issue 2
*********************************************

Tweet