Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. tools for factored corpora (Viktor Pless)
2. Re: KenLM memory usage (Kenneth Heafield)
3. Re: KenLM memory usage (Alexandra Birch)
4. Installation issues (Linux mint) (Martin McCaffery)
----------------------------------------------------------------------
Message: 1
Date: Thu, 20 Mar 2014 17:37:42 +0100
From: Viktor Pless <viktor.pless@gmail.com>
Subject: [Moses-support] tools for factored corpora
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAD3iPeFib_qJ8c1dkK2WS3O0T0Yq46y8vwdi=BLst=HDHSs+bg@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi, what tools can be used to lemmatize/POS-tag/etc. a corpus in moses
format (with the pipes)? I need them regarding Spanish, English, Hungarian.
Thanks in advance.
Viktor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140320/f3477770/attachment-0001.htm
------------------------------
Message: 2
Date: Thu, 20 Mar 2014 09:53:21 -0700
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] KenLM memory usage
To: moses-support@mit.edu
Message-ID: <532B1D01.8010605@kheafield.com>
Content-Type: text/plain; charset=ISO-8859-1
Hi Lexi,
I take it that these models are different, not the same model loaded
into each process (in which case they would have shared). I'd really
recommend trying to compress things more (e.g. trie -a 64 -q 8) before
going to lazy loading.
Kenneth
On 03/20/14 08:13, Marcin Junczys-Dowmunt wrote:
> Hi,
> since KenLM uses shared memory, four instances should take up the same
> amount of memory as only one instance (ran yesterday 8 instances with 8
> threads each with a 99GB LM on a 128 GB machine). If the model fits into
> memory for a single instance it should work if you have enough memory
> left for all the phrase tables and the translation process itself (I
> guess this is actually the problem). Lazy loading was unbearably slow
> for me with the above mentioned configuration, but I was using 64
> threads in total, so a lot of concurrent disk access happing, no wonder
> there.
> Best,
> Marcin
>
> W dniu 20.03.2014 14:35, Alexandra Birch pisze:
>> I have found the answer on the kenlm web page and it seems to be working:
>>
>> Full or lazy loading
>>
>> KenLM supports lazy loading via mmap. This allows you to further
>> reduce memory usage, especially with trie which has good memory
>> locality. In Moses, this is controlled by the language model number in
>> moses.ini. Using language model number 8 will load the full model into
>> memory (MAP_POPULATE on Linux and read() on other OSes). Language
>> model number 9 will lazily load the model using mmap. I recommend
>> fully loading if you have the RAM for it; it actually takes less time
>> to load the full model and use it because the disk does not have to
>> seek during decoding. Lazy loading works best with local disk and is
>> not recommended for networked filesystems.
>>
>>
>>
>> On Thu, Mar 20, 2014 at 2:32 PM, Alexandra Birch <lexi.birch@gmail.com
>> <mailto:lexi.birch@gmail.com>> wrote:
>>
>> Hi there,
>>
>> I want to run 4 MT servers at the same time on a machine with
>> limited memory. Kenlm seems to reserve the amount of memory which
>> the language model would have taken if it had been loaded into
>> memory. So I don't have enough memory to run all these servers and
>> the machine grinds to a halt if I try. Is there any flag I could
>> use which would limit the amount of memory reserved?
>>
>> Lexi
>>
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
------------------------------
Message: 3
Date: Thu, 20 Mar 2014 16:58:08 +0000
From: Alexandra Birch <lexi.birch@gmail.com>
Subject: Re: [Moses-support] KenLM memory usage
To: Kenneth Heafield <moses@kheafield.com>
Cc: moses-support@mit.edu
Message-ID:
<CA+h82t5AYxc6U1n8WvYDdum8MHNv-69ztYUdxAHvPZ80TmHOEg@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi Ken,
Yes, different models, different languages. Thanks! Yes lazy loading is
absolutely dead slow.
Lexi
On Thu, Mar 20, 2014 at 4:53 PM, Kenneth Heafield <moses@kheafield.com>wrote:
> Hi Lexi,
>
> I take it that these models are different, not the same model
> loaded
> into each process (in which case they would have shared). I'd really
> recommend trying to compress things more (e.g. trie -a 64 -q 8) before
> going to lazy loading.
>
> Kenneth
>
> On 03/20/14 08:13, Marcin Junczys-Dowmunt wrote:
> > Hi,
> > since KenLM uses shared memory, four instances should take up the same
> > amount of memory as only one instance (ran yesterday 8 instances with 8
> > threads each with a 99GB LM on a 128 GB machine). If the model fits into
> > memory for a single instance it should work if you have enough memory
> > left for all the phrase tables and the translation process itself (I
> > guess this is actually the problem). Lazy loading was unbearably slow
> > for me with the above mentioned configuration, but I was using 64
> > threads in total, so a lot of concurrent disk access happing, no wonder
> > there.
> > Best,
> > Marcin
> >
> > W dniu 20.03.2014 14:35, Alexandra Birch pisze:
> >> I have found the answer on the kenlm web page and it seems to be
> working:
> >>
> >> Full or lazy loading
> >>
> >> KenLM supports lazy loading via mmap. This allows you to further
> >> reduce memory usage, especially with trie which has good memory
> >> locality. In Moses, this is controlled by the language model number in
> >> moses.ini. Using language model number 8 will load the full model into
> >> memory (MAP_POPULATE on Linux and read() on other OSes). Language
> >> model number 9 will lazily load the model using mmap. I recommend
> >> fully loading if you have the RAM for it; it actually takes less time
> >> to load the full model and use it because the disk does not have to
> >> seek during decoding. Lazy loading works best with local disk and is
> >> not recommended for networked filesystems.
> >>
> >>
> >>
> >> On Thu, Mar 20, 2014 at 2:32 PM, Alexandra Birch <lexi.birch@gmail.com
> >> <mailto:lexi.birch@gmail.com>> wrote:
> >>
> >> Hi there,
> >>
> >> I want to run 4 MT servers at the same time on a machine with
> >> limited memory. Kenlm seems to reserve the amount of memory which
> >> the language model would have taken if it had been loaded into
> >> memory. So I don't have enough memory to run all these servers and
> >> the machine grinds to a halt if I try. Is there any flag I could
> >> use which would limit the amount of memory reserved?
> >>
> >> Lexi
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Moses-support mailing list
> >> Moses-support@mit.edu
> >> http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140320/b1fa9a41/attachment-0001.htm
------------------------------
Message: 4
Date: Thu, 20 Mar 2014 17:40:35 +0000
From: Martin McCaffery <mm689@st-andrews.ac.uk>
Subject: [Moses-support] Installation issues (Linux mint)
To: moses-support@mit.edu
Message-ID:
<CABbQ7_NmsGxtPOXCzxoobNxesEZmygpjErG3bMn=wCfbeeCXKQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi all,
I've been trying to get Moses up and running locally and been running into
problems. At first I had issues with trying to get Boost working on my
local machine, but after tweaking Boost the errors have got smaller.
Now when running ./bjam I seem to simply be getting g++ compile and testing
errors, but I can't tell much more than that. There are some Boost-related
undefined references, though: are there any specific configuration details
I need for Boost that I'm missing?
Any thoughts would be welcome.
Thanks!
Martin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140320/23adb7c5/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log.gz
Type: application/x-gzip
Size: 4137 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140320/23adb7c5/attachment.bin
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 89, Issue 52
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 89, Issue 52"
Post a Comment