Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: unsolved) Moses eating up too much memory (Barry Haddow)
2. Re: unsolved) Moses eating up too much memory (Hieu Hoang)
3. Re: How to use the NeuralLM (Hieu Hoang)
4. truecase.perl error (Read, James C)
5. Re: truecase.perl error (Dimitris Mavroeidis)
----------------------------------------------------------------------
Message: 1
Date: Tue, 29 Oct 2013 08:44:46 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] unsolved) Moses eating up too much memory
To: Andrew Shin <ravenyj@hotmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <526F757E.40003@staffmail.ed.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hi Andrew
In your moses.ini file I see the following line:
> 0.00820395 on dev /Users/ipadmin/Desktop/blcorpus/tuning.true.src
This is extremely low, and suggests something went wrong in tuning ...
although this should not cause a memory leak.
I also noticed that you are decoding files of 700,000 sentences. The
problem with this is that Moses will load the whole file into memory
before it starts decoding. Again, this should not take up 21G of RAM,
but it would be worth trying decoding a small set to see if you still
have the memory problems,
cheers - Barry
On 29/10/13 06:10, Andrew Shin wrote:
> I apologize since my previous question on the same topic didn't
> specify enough.
>
> I'm following the steps in
> http://www.statmt.org/moses/?n=Moses.Baseline exactly the same way,
> and when I try to run Moses with the following command:
> ~/mosesdecoder/bin/moses -f ~/working/mert-work/moses.ini (often with
> -output-hypo-score option)
> it starts eating up RAM until there's no space and starts taking hard
> disk space as swap memory,
> while it says its reading from phrase table.
> It went up to 17GB of swap memory and pretty much stopped.
>
> I binarised the phrase table and reordering table, and the following
> are the sizes of the binarised files
> -rw-r--r-- 1 ipadmin staff 856K Oct 28 10:45 phrase-table.binphr.idx
> -rw-r--r-- 1 ipadmin staff 140M Oct 28 10:45
> phrase-table.binphr.srctree.wa
> -rw-r--r-- 1 ipadmin staff 1.6M Oct 28 10:45
> phrase-table.binphr.srcvoc
> -rw-r--r-- 1 ipadmin staff 1.2G Oct 28 10:45
> phrase-table.binphr.tgtdata.wa
> -rw-r--r-- 1 ipadmin staff 1.5M Oct 28 10:45
> phrase-table.binphr.tgtvoc
> -rw-r--r-- 1 ipadmin staff 856K Oct 28 11:00
> reordering-table.binlexr.idx
> -rw-r--r-- 1 ipadmin staff 1.3G Oct 28 11:00
> reordering-table.binlexr.srctree
> -rw-r--r-- 1 ipadmin staff 916M Oct 28 11:00
> reordering-table.binlexr.tgtdata
> -rw-r--r-- 1 ipadmin staff 1.6M Oct 28 11:00
> reordering-table.binlexr.voc0
> -rw-r--r-- 1 ipadmin staff 1.5M Oct 28 11:00
> reordering-table.binlexr.voc1
>
> I also attached moses.ini which I tried to run.
> I'm very sure that this is not the way it should be, but can't figure
> out what is wrong since I followed the tutorial above..(and it's
> worked before) Please let me know if there is any additional
> information I should provide.
>
> I sincerely thank you very much for your time.
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
------------------------------
Message: 2
Date: Tue, 29 Oct 2013 11:30:49 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] unsolved) Moses eating up too much memory
To: Andrew Shin <ravenyj@hotmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAEKMkbg7TnxO11+kvOOZADhvzHOUkGOJ8+YsDxfjt6FXf1qidg@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
In the moses.ini file, change
PhraseDictionaryMemory
to
PhraseDictionaryBinary
This will let the decoder know that you want to use the binary phrase-table
On 29 October 2013 06:10, Andrew Shin <ravenyj@hotmail.com> wrote:
> I apologize since my previous question on the same topic didn't specify
> enough.
>
> I'm following the steps in http://www.statmt.org/moses/?n=Moses.Baseline exactly
> the same way,
> and when I try to run Moses with the following command:
> ~/mosesdecoder/bin/moses -f ~/working/mert-work/moses.ini (often with
> -output-hypo-score option)
> it starts eating up RAM until there's no space and starts taking hard disk
> space as swap memory,
> while it says its reading from phrase table.
> It went up to 17GB of swap memory and pretty much stopped.
>
> I binarised the phrase table and reordering table, and the following are
> the sizes of the binarised files
> -rw-r--r-- 1 ipadmin staff 856K Oct 28 10:45 phrase-table.binphr.idx
> -rw-r--r-- 1 ipadmin staff 140M Oct 28 10:45
> phrase-table.binphr.srctree.wa
> -rw-r--r-- 1 ipadmin staff 1.6M Oct 28 10:45
> phrase-table.binphr.srcvoc
> -rw-r--r-- 1 ipadmin staff 1.2G Oct 28 10:45
> phrase-table.binphr.tgtdata.wa
> -rw-r--r-- 1 ipadmin staff 1.5M Oct 28 10:45
> phrase-table.binphr.tgtvoc
> -rw-r--r-- 1 ipadmin staff 856K Oct 28 11:00
> reordering-table.binlexr.idx
> -rw-r--r-- 1 ipadmin staff 1.3G Oct 28 11:00
> reordering-table.binlexr.srctree
> -rw-r--r-- 1 ipadmin staff 916M Oct 28 11:00
> reordering-table.binlexr.tgtdata
> -rw-r--r-- 1 ipadmin staff 1.6M Oct 28 11:00
> reordering-table.binlexr.voc0
> -rw-r--r-- 1 ipadmin staff 1.5M Oct 28 11:00
> reordering-table.binlexr.voc1
>
> I also attached moses.ini which I tried to run.
> I'm very sure that this is not the way it should be, but can't figure out
> what is wrong since I followed the tutorial above..(and it's worked before)
> Please let me know if there is any additional information I should provide.
>
> I sincerely thank you very much for your time.
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131029/3fd9693e/attachment-0001.htm
------------------------------
Message: 3
Date: Tue, 29 Oct 2013 11:36:45 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] How to use the NeuralLM
To: Li Xiang <lixiang.ict@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbjnou5cNZSx1+Fmhho-YRyiENhRsBWSrUytWyDD4_x-qA@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
I just integrated Ashish Vaswani's Neural LM into the decoder, based on
code by Lane Schwartz.
To compile Moses with the new LM, do
./bjam .... --with-nplm=[path/to/nplm]
I don't know if it works yet, or know how to use it. You should read
Ashish's paper and download his code
http://nlg.isi.edu/software/nplm/
If you find out how to use it, please let us know!
On 29 October 2013 02:28, Li Xiang <lixiang.ict@gmail.com> wrote:
> Hi,
>
> I just find the NeuralLM is added into the Moses. Could you give me an
> example about the usage? Thanks.
>
> --
> Xiang Li
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131029/a29952e3/attachment-0001.htm
------------------------------
Message: 4
Date: Tue, 29 Oct 2013 14:24:53 +0000
From: "Read, James C" <jcread@essex.ac.uk>
Subject: [Moses-support] truecase.perl error
To: "moses-support@mit.edu" <moses-support@MIT.EDU>
Message-ID:
<F00840E41983C645928E21E3C35F4EB1012CF33CA6@mbx1-node2.essex.ac.uk>
Content-Type: text/plain; charset="iso-8859-1"
Hi,
I'm following the instructions to build a baseline system on http://www.statmt.org/moses/?n=Moses.Baseline
I get up to
~/mosesdecoder/scripts/recaser/truecase.perl \
--model ~/corpus/truecase-model.en \
< ~/corpus/news-commentary-v8.fr-en.tok.en \
> ~/corpus/news-commentary-v8.fr-en.true.en
and then I get the following error:
ERROR: could not open '/home/myname/corpus/truecase-model.en' at ./truecase.perl line 18.
which is strange because then when I look in corpus the file is there whereas it previously wasn't. I've looked at the script and can't see any reason why the script should die. It checks if it can open the file and dies if not. Any idea what is happening?
James
------------------------------
Message: 5
Date: Tue, 29 Oct 2013 16:42:18 +0200
From: Dimitris Mavroeidis <dmavroeidis@csri.gr>
Subject: Re: [Moses-support] truecase.perl error
To: "Read, James C" <jcread@essex.ac.uk>, "moses-support@mit.edu"
<moses-support@mit.edu>
Message-ID: <526FC94A.9030008@csri.gr>
Content-Type: text/plain; charset=UTF-8; format=flowed
Hi James,
Try using absolute paths everywhere, instead of relative. For instance,
try "/home/james/nlp_tools/mosesdecoder/scripts/recaser/truecase.perl"
(hypothetical path) instead of
"~/mosesdecoder/scripts/recaser/truecase.perl".
Best,
Dimitris
On 29/10/2013 04:24 ??, Read, James C wrote:
> Hi,
>
> I'm following the instructions to build a baseline system on http://www.statmt.org/moses/?n=Moses.Baseline
>
> I get up to
>
>
> ~/mosesdecoder/scripts/recaser/truecase.perl \
> --model ~/corpus/truecase-model.en \
> < ~/corpus/news-commentary-v8.fr-en.tok.en \
> > ~/corpus/news-commentary-v8.fr-en.true.en
>
> and then I get the following error:
>
> ERROR: could not open '/home/myname/corpus/truecase-model.en' at ./truecase.perl line 18.
>
> which is strange because then when I look in corpus the file is there whereas it previously wasn't. I've looked at the script and can't see any reason why the script should die. It checks if it can open the file and dies if not. Any idea what is happening?
>
> James
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 84, Issue 42
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 84, Issue 42"
Post a Comment