Moses-support Digest, Vol 89, Issue 51

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: KenLM memory usage (Alexandra Birch)
2. Re: Problem in Web Based Version of MT System using MOSES -
Please help (Barry Haddow)
3. Re: KenLM memory usage (Marcin Junczys-Dowmunt)


----------------------------------------------------------------------

Message: 1
Date: Thu, 20 Mar 2014 14:35:17 +0000
From: Alexandra Birch <lexi.birch@gmail.com>
Subject: Re: [Moses-support] KenLM memory usage
To: moses-support@mit.edu
Message-ID:
<CA+h82t6gQ-upTgbjTBRzSmYCWQxApVmV50yykotkgc3t_G7KMg@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

I have found the answer on the kenlm web page and it seems to be working:

Full or lazy loading

KenLM supports lazy loading via mmap. This allows you to further reduce
memory usage, especially with trie which has good memory locality. In
Moses, this is controlled by the language model number in moses.ini. Using
language model number 8 will load the full model into memory (MAP_POPULATE
on Linux and read() on other OSes). Language model number 9 will lazily
load the model using mmap. I recommend fully loading if you have the RAM
for it; it actually takes less time to load the full model and use it
because the disk does not have to seek during decoding. Lazy loading works
best with local disk and is not recommended for networked filesystems.



On Thu, Mar 20, 2014 at 2:32 PM, Alexandra Birch <lexi.birch@gmail.com>wrote:

> Hi there,
>
> I want to run 4 MT servers at the same time on a machine with limited
> memory. Kenlm seems to reserve the amount of memory which the language
> model would have taken if it had been loaded into memory. So I don't have
> enough memory to run all these servers and the machine grinds to a halt if
> I try. Is there any flag I could use which would limit the amount of memory
> reserved?
>
> Lexi
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140320/84b1ecfa/attachment-0001.htm

------------------------------

Message: 2
Date: Thu, 20 Mar 2014 15:06:31 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Problem in Web Based Version of MT System
using MOSES - Please help
To: "Vishal Goyal(????? ????)" <vishal.pup@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <532B03F7.5010107@staffmail.ed.ac.uk>
Content-Type: text/plain; charset=UTF-8; format=flowed

Hi Vishal

There's no other code in Moses for web page translation. It just takes a
bit more work to integrate it with the recent versions of Moses,

cheers - Barry

On 19/03/14 17:04, Vishal Goyal(????? ????) wrote:
> Thanks for kind reply.
> Then kindly suggest how can we make the web translation reality? Which
> new we need to use...
>
>
>
> On Wed, Mar 19, 2014 at 2:07 PM, Barry Haddow
> <bhaddow@staffmail.ed.ac.uk <mailto:bhaddow@staffmail.ed.ac.uk>> wrote:
>
> Hi Vishal
>
> The web translation components that you mention are quite old and
> haven't been used much in a while.
>
> As far as I can see, the main problem is that the translate.cgi
> expects to have many copies of the daemon.pl <http://daemon.pl>
> running, all listening on different ports. Each one should wrap a
> different instance of Moses. So this web translation system is not
> multi-threaded (it was written before Moses had threads) it's
> multi-process,
>
> cheers - Barry
>
>
> On 18/03/14 15:02, Vishal Goyal(????? ????) wrote:
>
> Respected All,
> Greetings
>
> We have installed the moses server and web server on a single
> machine on linux platform. when tested on the local host the
> system work fine, when we installed the system on web server
> for public use. The system some work fine but most of the we
> are not getting the translation of the input text instead the
> input text is transliterated in the post processing script
> written in transliterate.pl <http://transliterate.pl>
> <http://transliterate.pl>.
>
> Moses server is running as daemon.pl <http://daemon.pl>
> <http://daemon.pl>. The index.cgi is being used to take user
> input and translate.cgi start the moses thread for tranlation.
>
>
> When error log is inspected it show that only one thread of
> moses is running and all other threads are terminated with the
> error message (error log attached.)
>
> Please also find the attached source code of daemon.pl
> <http://daemon.pl> <http://daemon.pl>, index.cgi and translate.cgi
>
>
>
> --
> *Regards,*
>
> Vishal Goyal,
> Ph.D., M.Tech., MCA, M.C.S.D.
> Assistant Professor(Stage III) and Placement Coordinator,
> Department of Computer Science,
> Punjabi University Patiala-147002
> [*Online Hindi to Punjabi Machine Translation Tool -*
> http://h2p.learnpunjabi.org ]
> *[Research Cell: An International Journal of Engineering
> Sciences, http://ijoes.vidyapublications.com]*
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
>
>
> --
> *Regards,*
> Vishal Goyal,
> Ph.D., M.Tech., MCA, M.C.S.D.
> Assistant Professor(Stage III) and Placement Coordinator,
> Department of Computer Science,
> Punjabi University Patiala-147002
> [*Online Hindi to Punjabi Machine Translation Tool -*
> http://h2p.learnpunjabi.org ]
> *[Research Cell: An International Journal of Engineering Sciences,
> http://ijoes.vidyapublications.com]*
>


--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



------------------------------

Message: 3
Date: Thu, 20 Mar 2014 15:13:57 +0000
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] KenLM memory usage
To: moses-support@mit.edu
Message-ID: <532B05B5.9010602@amu.edu.pl>
Content-Type: text/plain; charset="iso-8859-1"

Hi,
since KenLM uses shared memory, four instances should take up the same
amount of memory as only one instance (ran yesterday 8 instances with 8
threads each with a 99GB LM on a 128 GB machine). If the model fits into
memory for a single instance it should work if you have enough memory
left for all the phrase tables and the translation process itself (I
guess this is actually the problem). Lazy loading was unbearably slow
for me with the above mentioned configuration, but I was using 64
threads in total, so a lot of concurrent disk access happing, no wonder
there.
Best,
Marcin

W dniu 20.03.2014 14:35, Alexandra Birch pisze:
> I have found the answer on the kenlm web page and it seems to be working:
>
> Full or lazy loading
>
> KenLM supports lazy loading via mmap. This allows you to further
> reduce memory usage, especially with trie which has good memory
> locality. In Moses, this is controlled by the language model number in
> moses.ini. Using language model number 8 will load the full model into
> memory (MAP_POPULATE on Linux and read() on other OSes). Language
> model number 9 will lazily load the model using mmap. I recommend
> fully loading if you have the RAM for it; it actually takes less time
> to load the full model and use it because the disk does not have to
> seek during decoding. Lazy loading works best with local disk and is
> not recommended for networked filesystems.
>
>
>
> On Thu, Mar 20, 2014 at 2:32 PM, Alexandra Birch <lexi.birch@gmail.com
> <mailto:lexi.birch@gmail.com>> wrote:
>
> Hi there,
>
> I want to run 4 MT servers at the same time on a machine with
> limited memory. Kenlm seems to reserve the amount of memory which
> the language model would have taken if it had been loaded into
> memory. So I don't have enough memory to run all these servers and
> the machine grinds to a halt if I try. Is there any flag I could
> use which would limit the amount of memory reserved?
>
> Lexi
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140320/537b6218/attachment-0001.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 89, Issue 51
*********************************************

Tweet