Moses-support Digest, Vol 108, Issue 80

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Moses on SGE clarification (Vincent Nguyen)
2. Re: Moses on SGE clarification (Philipp Koehn)
3. Re: Moses on SGE clarification (Vincent Nguyen)


----------------------------------------------------------------------

Message: 1
Date: Thu, 29 Oct 2015 15:12:36 +0100
From: Vincent Nguyen <vnguyen@neuf.fr>
Subject: Re: [Moses-support] Moses on SGE clarification
To: Philipp Koehn <phi@jhu.edu>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <56322954.4070909@neuf.fr>
Content-Type: text/plain; charset="utf-8"


OK guys, not an easy stuff ...
I fought to get the prerequisites working but but now at least jobs
start .....

and crash.

I'll post later the details of the preliminary steps, could be useful.

my crash is when lplmz starts.

I have a sharepoint mounted on my nodes and all bin are well seen from
the nodes, including the lplmz program.

but I was thinking, do I need to actually install some packages on the
nodes themselves ? I mean packages that do not fall under /mosesdecoder/
folder ?


thanks,

V


Le 29/10/2015 13:26, Philipp Koehn a ?crit :
> Hi,
>
> these machine names are just there for convenience.
>
> If you want experiment.perl to submit jobs per qsub,
> all you have to do is to run experiment.perl with the
> additional switch "-cluster".
>
> You can also put the head node's name into the
> experiment.machines file, then you do not need to
> use the switch anymore.
>
> -phi
>
> On Wed, Oct 28, 2015 at 10:20 AM, Vincent Nguyen <vnguyen@neuf.fr
> <mailto:vnguyen@neuf.fr>> wrote:
>
> Hi there,
>
> I need some clarification before screwing up some files.
> I just setup a SGE cluster with a Master + 2 Nodes.
>
> to make it clear let say my cluster name is "default", my master
> headnode is "master", my 2 other nodes are "node1" and "node2"
>
>
> for EMS :
>
> I opened the default experiment.machines file and I see :
>
> cluster: townhill seville hermes lion seville sannox lutzow frontend
> multicore-4: freddie
> multicore-8: tyr thor odin crom
> multicore-16: saxnot vali vili freyja bragi hoenir
> multicore-24: syn hel skaol saga buri loki sif magni
> multicore-32: gna snotra lofn thrud
>
> townhill and others are what ? name machines / nodes ? name of several
> clusters ?
> should I just put "default" or "master node1 node2" ?
>
> multicore-X: should I put machine names here
> if my 3 machines are 8 cores each
> multicore-8: master node1 node2
> right ?
>
>
> then in the config file for EMS:
>
> #generic-parallelizer =
> $moses-script-dir/ems/support/generic-parallelizer.perl
> #generic-parallelizer =
> $moses-script-dir/ems/support/generic-multicore-parallelizer.perl
>
> which one should take if my nodes are multicore ? still the first
> one ?
>
>
> ### cluster settings (if run on a cluster machine)
> # number of jobs to be submitted in parallel
> #
> #jobs = 10
> should I count approx 1 job per core on the total cores of my 3
> machines ?
>
> # arguments to qsub when scheduling a job
> #qsub-settings = ""
> can this stay empty ?
>
> # project for priviledges and usage accounting
> #qsub-project = iccs_smt
> standard value ?
>
> # memory and time
> #qsub-memory = 4
> #qsub-hours = 48
> 4 what ? GB ?
>
> ### multi-core settings
> # when the generic parallelizer is used, the number of cores
> # specified here
> cores = 4
> is this ignored if generic-parallelizer.perl is chosen ?
>
>
> is there a way to put more load on one specific node ?
>
> Many thanks,
> V.
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151029/b9ef12f1/attachment-0001.html

------------------------------

Message: 2
Date: Thu, 29 Oct 2015 10:18:16 -0400
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] Moses on SGE clarification
To: Vincent Nguyen <vnguyen@neuf.fr>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAAFADDBMcz862Jcox0nb8F-KSteUtZrSnssv6m3w0R8JXXGFRQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

make sure that all the paths are valid on all the nodes --- so definitely
no relative paths.
And of course, the binaries need to be executable on all nodes as well.

-phi

On Thu, Oct 29, 2015 at 10:12 AM, Vincent Nguyen <vnguyen@neuf.fr> wrote:

>
> OK guys, not an easy stuff ...
> I fought to get the prerequisites working but but now at least jobs start
> .....
>
> and crash.
>
> I'll post later the details of the preliminary steps, could be useful.
>
> my crash is when lplmz starts.
>
> I have a sharepoint mounted on my nodes and all bin are well seen from the
> nodes, including the lplmz program.
>
> but I was thinking, do I need to actually install some packages on the
> nodes themselves ? I mean packages that do not fall under /mosesdecoder/
> folder ?
>
>
> thanks,
>
> V
>
>
>
> Le 29/10/2015 13:26, Philipp Koehn a ?crit :
>
> Hi,
>
> these machine names are just there for convenience.
>
> If you want experiment.perl to submit jobs per qsub,
> all you have to do is to run experiment.perl with the
> additional switch "-cluster".
>
> You can also put the head node's name into the
> experiment.machines file, then you do not need to
> use the switch anymore.
>
> -phi
>
> On Wed, Oct 28, 2015 at 10:20 AM, Vincent Nguyen <vnguyen@neuf.fr> wrote:
>
>> Hi there,
>>
>> I need some clarification before screwing up some files.
>> I just setup a SGE cluster with a Master + 2 Nodes.
>>
>> to make it clear let say my cluster name is "default", my master
>> headnode is "master", my 2 other nodes are "node1" and "node2"
>>
>>
>> for EMS :
>>
>> I opened the default experiment.machines file and I see :
>>
>> cluster: townhill seville hermes lion seville sannox lutzow frontend
>> multicore-4: freddie
>> multicore-8: tyr thor odin crom
>> multicore-16: saxnot vali vili freyja bragi hoenir
>> multicore-24: syn hel skaol saga buri loki sif magni
>> multicore-32: gna snotra lofn thrud
>>
>> townhill and others are what ? name machines / nodes ? name of several
>> clusters ?
>> should I just put "default" or "master node1 node2" ?
>>
>> multicore-X: should I put machine names here
>> if my 3 machines are 8 cores each
>> multicore-8: master node1 node2
>> right ?
>>
>>
>> then in the config file for EMS:
>>
>> #generic-parallelizer =
>> $moses-script-dir/ems/support/generic-parallelizer.perl
>> #generic-parallelizer =
>> $moses-script-dir/ems/support/generic-multicore-parallelizer.perl
>>
>> which one should take if my nodes are multicore ? still the first one ?
>>
>>
>> ### cluster settings (if run on a cluster machine)
>> # number of jobs to be submitted in parallel
>> #
>> #jobs = 10
>> should I count approx 1 job per core on the total cores of my 3 machines ?
>>
>> # arguments to qsub when scheduling a job
>> #qsub-settings = ""
>> can this stay empty ?
>>
>> # project for priviledges and usage accounting
>> #qsub-project = iccs_smt
>> standard value ?
>>
>> # memory and time
>> #qsub-memory = 4
>> #qsub-hours = 48
>> 4 what ? GB ?
>>
>> ### multi-core settings
>> # when the generic parallelizer is used, the number of cores
>> # specified here
>> cores = 4
>> is this ignored if generic-parallelizer.perl is chosen ?
>>
>>
>> is there a way to put more load on one specific node ?
>>
>> Many thanks,
>> V.
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151029/c1801c94/attachment-0001.html

------------------------------

Message: 3
Date: Thu, 29 Oct 2015 16:09:06 +0100
From: Vincent Nguyen <vnguyen@neuf.fr>
Subject: Re: [Moses-support] Moses on SGE clarification
To: Philipp Koehn <phi@jhu.edu>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <56323692.3070107@neuf.fr>
Content-Type: text/plain; charset="utf-8"


I get this error :

moses@sgenode1:/netshr/working-en-fr$ /netshr/mosesdecoder/bin/lmplz
--text /netshr/working-en-fr/lm/europarl.truecased.7 --order 5 --arpa
/netshr/working-en-fr/lm/europarl.lm.7 --prune 0 0 1 -T
/netshr/working-en-fr/lm -S 20%
=== 1/5 Counting and sorting n-grams ===
Reading /netshr/working-en-fr/lm/europarl.truecased.7
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
tcmalloc: large alloc 2755821568 bytes == 0x25d28000 @
****************************************************************************************************
Segmentation fault (core dumped)
moses@sgenode1:/netshr/working-en-fr$

I installed the libgoogle-pertools-dev but same error.
Just to be clear, all these packages below are just necessary to build
Moses, do I need specific packages
to run one or other binary ?
confused....


Ubuntu

Install the following packages using the command

sudo apt-get install [package name]

Packages:

g++
git
subversion
automake
libtool
zlib1g-dev
libboost-all-dev
libbz2-dev
liblzma-dev
python-dev
graphviz
imagemagick
libgoogle-perftools-dev (for tcmalloc)






Le 29/10/2015 15:18, Philipp Koehn a ?crit :
> Hi,
>
> make sure that all the paths are valid on all the nodes --- so
> definitely no relative paths.
> And of course, the binaries need to be executable on all nodes as well.
>
> -phi
>
> On Thu, Oct 29, 2015 at 10:12 AM, Vincent Nguyen <vnguyen@neuf.fr
> <mailto:vnguyen@neuf.fr>> wrote:
>
>
> OK guys, not an easy stuff ...
> I fought to get the prerequisites working but but now at least
> jobs start .....
>
> and crash.
>
> I'll post later the details of the preliminary steps, could be useful.
>
> my crash is when lplmz starts.
>
> I have a sharepoint mounted on my nodes and all bin are well seen
> from the nodes, including the lplmz program.
>
> but I was thinking, do I need to actually install some packages on
> the nodes themselves ? I mean packages that do not fall under
> /mosesdecoder/ folder ?
>
>
> thanks,
>
> V
>
>
>
> Le 29/10/2015 13:26, Philipp Koehn a ?crit :
>> Hi,
>>
>> these machine names are just there for convenience.
>>
>> If you want experiment.perl to submit jobs per qsub,
>> all you have to do is to run experiment.perl with the
>> additional switch "-cluster".
>>
>> You can also put the head node's name into the
>> experiment.machines file, then you do not need to
>> use the switch anymore.
>>
>> -phi
>>
>> On Wed, Oct 28, 2015 at 10:20 AM, Vincent Nguyen <vnguyen@neuf.fr
>> <mailto:vnguyen@neuf.fr>> wrote:
>>
>> Hi there,
>>
>> I need some clarification before screwing up some files.
>> I just setup a SGE cluster with a Master + 2 Nodes.
>>
>> to make it clear let say my cluster name is "default", my master
>> headnode is "master", my 2 other nodes are "node1" and "node2"
>>
>>
>> for EMS :
>>
>> I opened the default experiment.machines file and I see :
>>
>> cluster: townhill seville hermes lion seville sannox lutzow
>> frontend
>> multicore-4: freddie
>> multicore-8: tyr thor odin crom
>> multicore-16: saxnot vali vili freyja bragi hoenir
>> multicore-24: syn hel skaol saga buri loki sif magni
>> multicore-32: gna snotra lofn thrud
>>
>> townhill and others are what ? name machines / nodes ? name
>> of several
>> clusters ?
>> should I just put "default" or "master node1 node2" ?
>>
>> multicore-X: should I put machine names here
>> if my 3 machines are 8 cores each
>> multicore-8: master node1 node2
>> right ?
>>
>>
>> then in the config file for EMS:
>>
>> #generic-parallelizer =
>> $moses-script-dir/ems/support/generic-parallelizer.perl
>> #generic-parallelizer =
>> $moses-script-dir/ems/support/generic-multicore-parallelizer.perl
>>
>> which one should take if my nodes are multicore ? still the
>> first one ?
>>
>>
>> ### cluster settings (if run on a cluster machine)
>> # number of jobs to be submitted in parallel
>> #
>> #jobs = 10
>> should I count approx 1 job per core on the total cores of my
>> 3 machines ?
>>
>> # arguments to qsub when scheduling a job
>> #qsub-settings = ""
>> can this stay empty ?
>>
>> # project for priviledges and usage accounting
>> #qsub-project = iccs_smt
>> standard value ?
>>
>> # memory and time
>> #qsub-memory = 4
>> #qsub-hours = 48
>> 4 what ? GB ?
>>
>> ### multi-core settings
>> # when the generic parallelizer is used, the number of cores
>> # specified here
>> cores = 4
>> is this ignored if generic-parallelizer.perl is chosen ?
>>
>>
>> is there a way to put more load on one specific node ?
>>
>> Many thanks,
>> V.
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151029/74726360/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 108, Issue 80
**********************************************

Tweet