Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Running moses fills up hard disk space.. (Tom Hoar)
2. multi-threaded tuning not working (Andrew Shin)
3. Re: multi-threaded tuning not working (Hieu Hoang)
4. How to use the NeuralLM (Li Xiang)
5. unsolved) Moses eating up too much memory (Andrew Shin)
----------------------------------------------------------------------
Message: 1
Date: Mon, 28 Oct 2013 23:26:37 +0700
From: Tom Hoar <tahoar@precisiontranslationtools.com>
Subject: Re: [Moses-support] Running moses fills up hard disk space..
To: Moses-Support <moses-support@mit.edu>
Message-ID: <526E903D.9070309@precisiontranslationtools.com>
Content-Type: text/plain; charset="iso-2022-jp"
Running the Moses binary doesn't use either of the perl scripts. I've
had experiences where moses buffered the translated output and
accumulated significant RAM, but never hard drive space.
What's the size of your nbest.txt output file? Is it possible that 25
the 25 n-best output sentences for each of 700,000 source sentences are
consuming all your hard drive?
I'm at a loss. Maybe someone else has experience with this.
On 10/28/2013 07:38 PM, Andrew Shin wrote:
> thank you for your answer.
>
> I've trained, tuned, and binarised the data and ran the moses with the
> usual command
> ~/mosesdecoder/bin/moses -f ~/working/mert-work/moses.ini
> with the option -output-hypo-score -n-best-list nbest.txt 25.
> Eating up the disk drive happens after that when it says reading from
> phrase table.
> My input files were about 700,000 lines each.
> I followed the steps exactly from
> http://www.statmt.org/moses/?n=Moses.Baseline
> and it worked without a problem last time..
>
> Could it be train-model.perl and mert-moses.pl are being used at that
> stage?
> If it's still unclear, is there an estimate of how much disk would be
> needed for 700,000 lines of input?
>
> Thank you very much for your time in advance.
>
>
> ------------------------------------------------------------------------
> Date: Mon, 28 Oct 2013 18:43:40 +0700
> From: tahoar@precisiontranslationtools.com
> To: moses-support@mit.edu
> Subject: Re: [Moses-support] Running moses fills up hard disk space..
>
> Your description is a bit unclear. Are you talking about running the
> Moses binary? Running Moses with binarized phrase table, reordering
> table and language model (or hierarchical equivalent) -- and
> configuring the moses.ini file to use them -- should not consume any
> additional hard disk space. So, what exactly did you binarize and what
> are you running?
>
> If you're talking about running train-model.perl or mert-moses.pl,
> these two scripts can consume considerable hard drive space while they
> do their work. It's not uncommon to consume 100-200 GB of hard drive
> space if you have a large, complex corpus.
>
>
>
> On 10/28/2013 10:09 AM, Andrew Shin wrote:
>
> Hello,
>
>
> I'm running Moses after binarising it,
> and while it's reading from phrase table,
> it rapidly fills up hard disk space until there is no more space..
> I had 21GB and it wasn't enough..
> Once I quit terminal, the space is freed again..
> I successfully ran Moses before, so I don't understand why this
> happens..
> Is this natural? If so, how much reserve space do I need?
> and If not, what could have gone wrong?
>
> Thank you very much for your help in advance.
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________ Moses-support mailing
> list Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131028/623d711f/attachment-0001.htm
------------------------------
Message: 2
Date: Tue, 29 Oct 2013 07:49:38 +0900
From: Andrew Shin <ravenyj@hotmail.com>
Subject: [Moses-support] multi-threaded tuning not working
To: "moses-support@mit.edu" <moses-support@MIT.EDU>
Message-ID: <BLU171-W42713EC38CDF285CA5C473B2080@phx.gbl>
Content-Type: text/plain; charset="iso-2022-jp"
Hi,
I've done tuning without the multi-core option,and it took about 20 hours for 650,000 lines of input files, and 5000 lines of tuning reference.I tried tuning with the multi-core option as--decoder-flags="-threads 4"but it doesn't initiate anything.I have macbook pro with 2.4 GHz i5 dual core.Is there any known issue with this option?
Thank you for your help in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131029/2d237386/attachment-0001.htm
------------------------------
Message: 3
Date: Tue, 29 Oct 2013 00:51:37 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] multi-threaded tuning not working
To: Andrew Shin <ravenyj@hotmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <F6D8B2CB-7890-43F9-97CC-0DB915BC4061@gmail.com>
Content-Type: text/plain; charset="us-ascii"
20 hours for tuning isn't too bad. What is the EXACT command you ran? Can you also post your configuration file and Moses.ini for when you ask questions to help people debug your problem
Sent while bumping into things
> On 28 Oct 2013, at 22:49, Andrew Shin <ravenyj@hotmail.com> wrote:
>
> Hi,
>
> I've done tuning without the multi-core option,
> and it took about 20 hours for 650,000 lines of input files, and 5000 lines of tuning reference.
> I tried tuning with the multi-core option as
> --decoder-flags="-threads 4"
> but it doesn't initiate anything.
> I have macbook pro with 2.4 GHz i5 dual core.
> Is there any known issue with this option?
>
> Thank you for your help in advance.
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131029/c81e632a/attachment-0001.htm
------------------------------
Message: 4
Date: Tue, 29 Oct 2013 10:28:43 +0800
From: Li Xiang <lixiang.ict@gmail.com>
Subject: [Moses-support] How to use the NeuralLM
To: moses-support <moses-support@mit.edu>
Message-ID:
<CA+fVw+5aBgzwErUxnvDJqwd7t6xLxof7GEhr1pw7xEZxipR-uQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
I just find the NeuralLM is added into the Moses. Could you give me an
example about the usage? Thanks.
--
Xiang Li
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131029/0fbf4dd1/attachment-0001.htm
------------------------------
Message: 5
Date: Tue, 29 Oct 2013 15:10:35 +0900
From: Andrew Shin <ravenyj@hotmail.com>
Subject: [Moses-support] unsolved) Moses eating up too much memory
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <BLU171-W1141196EB16185C0F196A78B2090@phx.gbl>
Content-Type: text/plain; charset="iso-2022-jp"
I apologize since my previous question on the same topic didn't specify enough.I'm following the steps in http://www.statmt.org/moses/?n=Moses.Baseline exactly the same way,and when I try to run Moses with the following command:~/mosesdecoder/bin/moses -f ~/working/mert-work/moses.ini (often with -output-hypo-score option)it starts eating up RAM until there's no space and starts taking hard disk space as swap memory,while it says its reading from phrase table.It went up to 17GB of swap memory and pretty much stopped.I binarised the phrase table and reordering table, and the following are the sizes of the binarised files-rw-r--r-- 1 ipadmin staff 856K Oct 28 10:45 phrase-table.binphr.idx-rw-r--r-- 1 ipadmin staff 140M Oct 28 10:45 phrase-table.binphr.srctree.wa-rw-r--r-- 1 ipadmin staff 1.6M Oct 28 10:45 phrase-table.binphr.srcvoc-rw-r--r-- 1 ipadmin staff 1.2G Oct 28 10:45 phrase-table.binphr.tgtdata.wa-rw-r--r-- 1 ipadmin staff 1.5M Oct 28 10:45 phr!
ase-table.binphr.tgtvoc-rw-r--r-- 1 ipadmin staff 856K Oct 28 11:00 reordering-table.binlexr.idx-rw-r--r-- 1 ipadmin staff 1.3G Oct 28 11:00 reordering-table.binlexr.srctree-rw-r--r-- 1 ipadmin staff 916M Oct 28 11:00 reordering-table.binlexr.tgtdata-rw-r--r-- 1 ipadmin staff 1.6M Oct 28 11:00 reordering-table.binlexr.voc0-rw-r--r-- 1 ipadmin staff 1.5M Oct 28 11:00 reordering-table.binlexr.voc1I also attached moses.ini which I tried to run.I'm very sure that this is not the way it should be, but can't figure out what is wrong since I followed the tutorial above..(and it's worked before) Please let me know if there is any additional information I should provide.I sincerely thank you very much for your time.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131029/ea69d541/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: moses.ini
Type: application/octet-stream
Size: 1233 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20131029/ea69d541/attachment.obj
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 84, Issue 41
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 84, Issue 41"
Post a Comment