Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Moses vs Moses2 in its multi-threading (Ivan Zapreev)
2. Re: Moses vs Moses2 in its multi-threading (Hieu Hoang)
3. Re: Moses vs Moses2 in its multi-threading (Ivan Zapreev)
----------------------------------------------------------------------
Message: 1
Date: Thu, 16 Mar 2017 13:25:39 +0100
From: Ivan Zapreev <ivan.zapreev@gmail.com>
Subject: [Moses-support] Moses vs Moses2 in its multi-threading
To: moses-support@mit.edu
Message-ID:
<CAOwV4it8W-TWGtrzK2PHdeF+pn=aGiREKVpUnMoBLfPmmZ1m6Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Dear All,
I have a question and perhaps want to draw your attention to the following
fact. The Moses2 webpage http://www.statmt.org/moses/?n=Site.Moses2 claims
that Moses2 has better multi-threading support than Moses and scales better
on multi-core machines. As can be extrapolated from the performance picture
on that webpage the performance of Moses and Moses2 on a single core are
therefore almost the same.
We develop our own free and open source distributed SMT infrastructure and
our empirical comparison of the translation tools shows that things are not
quite like that. Actually Moses2 is about 2 times faster than Moses on a
single thread and with adding more threads on a multi-core machine this
speed difference is only reducing ... We actually observe that Moses2
scales worse in the number of threads than Moses and that the performance
benefits of Moses2 seem to be solely due to faster single sentence
decoding. So I am curious why do we have so much different results from the
official ones? Could some one please give me a hint on that? Is the
information on the Moses2 webpage outdated?
The results of our empirical evaluation can be found by the following link:
https://github.com/ivan-zapreev/Basic-Translation-
Infrastructure#translation-server-evaluation
The experimental setup is thoroughly described in:
1.
https://github.com/ivan-zapreev/Basic-Translation-Infrastructure#test-set-up-1
2.
https://github.com/ivan-zapreev/Basic-Translation-Infrastructure#test-server
3.
https://github.com/ivan-zapreev/Basic-Translation-Infrastructure#mosesmoses2
Clearly, the models were located on a local disk and there were no
processes affecting the timing results. The experiments were repeated
multiple times and average values with standard deviations were obtained.
We can not make the models available just like that, as for one they are
very big, see the experimental setup description:
- *Language Model* - 48.9 Gb (5-gram model);
- *Translation Model* - 1.3 Gb (5 features model);
- *Reordering Model* - 9.7 Gb (8 features model);
Moreover, they are obtained on a Chinese to English OpenMT MT-04 data set
so they can be reproduced from there.
Thank you in advance!
Kind regards,
Dr. Ivan S. Zapreev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170316/569fbadc/attachment-0001.html
------------------------------
Message: 2
Date: Thu, 16 Mar 2017 12:53:20 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Moses vs Moses2 in its multi-threading
To: Ivan Zapreev <ivan.zapreev@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbjEd-n6hjnMZ_qYQDx5YAn1FPdkyd2QDOQ5grjMD7a5YQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
* Looking for MT/NLP opportunities *
Hieu Hoang
http://moses-smt.org/
On 16 March 2017 at 12:25, Ivan Zapreev <ivan.zapreev@gmail.com> wrote:
> Dear All,
>
> I have a question and perhaps want to draw your attention to the following
> fact. The Moses2 webpage http://www.statmt.org/moses/?n=Site.Moses2
> claims that Moses2 has better multi-threading support than Moses and scales
> better on multi-core machines. As can be extrapolated from the performance
> picture on that webpage the performance of Moses and Moses2 on a single
> core are therefore almost the same.
>
> We develop our own free and open source distributed SMT infrastructure and
> our empirical comparison of the translation tools shows that things are not
> quite like that. Actually Moses2 is about 2 times faster than Moses on a
> single thread and with adding more threads on a multi-core machine this
> speed difference is only reducing ... We actually observe that Moses2
> scales worse in the number of threads than Moses and that the performance
> benefits of Moses2 seem to be solely due to faster single sentence
> decoding. So I am curious why do we have so much different results from the
> official ones? Could some one please give me a hint on that? Is the
> information on the Moses2 webpage outdated?
>
I'm curious too. The webpage is up to date, the info should be all correct
>
> The results of our empirical evaluation can be found by the following link:
>
> https://github.com/ivan-zapreev/Basic-Translation-Infrastruc
> ture#translation-server-evaluation
>
> The experimental setup is thoroughly described in:
>
> 1. https://github.com/ivan-zapreev/Basic-Translation-
> Infrastructure#test-set-up-1
> 2. https://github.com/ivan-zapreev/Basic-Translation-
> Infrastructure#test-server
> 3. https://github.com/ivan-zapreev/Basic-Translation-
> Infrastructure#mosesmoses2
>
> Some clarifications:
So your server has 20 cores (40 hyperthreads) and 16GB RAM? If that's
correct, then the RAM size would be a problem - you need as much RAM as the
total size of your models, plus more for working memory and the OS.
Do you run Moses command line, or the server? My timings are based on the
command line, the server is a little slower.
Do you run Moses directly, or is another evaluation process running it? Are
you sure that evaluation process is working as it should?
Do you minimise the effect of disk read by pre-loading the models into
filesystem cache? This is usually done by running this before running the
decoder
cat [binary model files] > /dev/null
> Clearly, the models were located on a local disk and there were no
> processes affecting the timing results. The experiments were repeated
> multiple times and average values with standard deviations were obtained.
>
> We can not make the models available just like that, as for one they are
> very big, see the experimental setup description:
>
it may take a while, but I can't replicate your results without it.
Alternatively, I can provide you with my models so you can try & replicate
my results.
>
> - *Language Model* - 48.9 Gb (5-gram model);
> - *Translation Model* - 1.3 Gb (5 features model);
> - *Reordering Model* - 9.7 Gb (8 features model);
>
> Moreover, they are obtained on a Chinese to English OpenMT MT-04 data set
> so they can be reproduced from there.
>
> Thank you in advance!
>
> Kind regards,
>
> Dr. Ivan S. Zapreev
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170316/b2215650/attachment-0001.html
------------------------------
Message: 3
Date: Thu, 16 Mar 2017 14:16:58 +0100
From: Ivan Zapreev <ivan.zapreev@gmail.com>
Subject: Re: [Moses-support] Moses vs Moses2 in its multi-threading
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAOwV4it_xyfq8LxU-uoCDDpNLk8ApnWMkp8pW1MLZu=UM1sJjQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Dear Hieu,
Thank you for a prompt and detailed reply!
>>> So your server has 20 cores (40 hyperthreads) and 16GB RAM? If that's
correct, then the RAM size would be a problem - you need as much RAM as the
total size of your models, plus more for working memory and the OS.
The amount of memory is 256 Gb and not 16. There are a number of 16 Gb
plates installed.
To my knowledge the machine is not hyperthreaded but just has 40 cores,
although I am now getting a bit doubtful about that.
>> Do you run Moses command line, or the server? My timings are based on
the command line, the server is a little slower.
Both Moses and Moses2 are run in the console mode (not server). The model
loading time is excluded from the measurements. I could not manage to get
the asynchronous XML-RPC to work, so for my experiments that would be as if
I used Moses/Moses2 in a single-thread mode. Therefore I used the
command-line version.
>>> Do you run Moses directly, or is another evaluation process running it?
Are you sure that evaluation process is working as it should?
Moses is run from command time under the "time" command of Linux, and so
are other systems we used in comarison. We look at the runtime and not the
CPU times, but we perform a number of experiments to measure the average
times and control the standard deviations.
>>> Do you minimise the effect of disk read by pre-loading the models into
filesystem cache? This is usually done by running this before running the
decoder cat [binary model files] > /dev/null
Nop, we did not do pre-loading, for none of the tools but perhaps this is
not an issue as we just measure the average model loading times and
subtract them from the average run-time with decoding. So the model loading
times are excluded from the results. Our goal was to measure and compare
the decoding times and how they scale in the number of threads.
>>> it may take a while, but I can't replicate your results without it.
Alternatively, I can provide you with my models so you can try & replicate
my results.
The experiments are run on an internal server which is not visible from
outside. I shall explore the possibilities of sharing the models, but I am
doubtful it is possible. The university network is very restricted. Yet, I
am definitely open to re-running your experiments. If possible.
Kind regards,
Ivan
<http://www.tainichok.ru/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170316/2e0c258f/attachment.html
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 125, Issue 33
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 125, Issue 33"
Post a Comment