Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. (no subject) (Winston Yuen)
2. Re: (no subject) (Tom Hoar)
3. Re: My phrase-table.tgz is 20-bytes long (Tom Hoar)
4. Re: (no subject) (Lane Schwartz)
5. Re: (no subject) (Lane Schwartz)
----------------------------------------------------------------------
Message: 1
Date: Fri, 27 Feb 2015 00:27:16 +0000
From: Winston Yuen <luuyuen@gmail.com>
Subject: [Moses-support] (no subject)
To: moses-support@mit.edu
Message-ID:
<CAAX6XPuyJKiBJimZyjV8RxngKTiO4hhPznVix5GgHRNwDHeuVw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Is there Xcode or Eclipse project for Moses? I try to find it online but
cannot find it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150227/17b6e7da/attachment-0001.htm
------------------------------
Message: 2
Date: Fri, 27 Feb 2015 08:00:54 +0700
From: Tom Hoar <tahoar@precisiontranslationtools.com>
Subject: Re: [Moses-support] (no subject)
To: moses-support@mit.edu
Message-ID: <54EFC1C6.8040409@precisiontranslationtools.com>
Content-Type: text/plain; charset="windows-1252"
Not to my knowledge. Your contribution of one would be welcome.
On 02/27/2015 07:27 AM, Winston Yuen wrote:
> Is there Xcode or Eclipse project for Moses? I try to find it online
> but cannot find it.
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150227/680f3541/attachment-0001.htm
------------------------------
Message: 3
Date: Fri, 27 Feb 2015 08:25:01 +0700
From: Tom Hoar <tahoar@precisiontranslationtools.com>
Subject: Re: [Moses-support] My phrase-table.tgz is 20-bytes long
To: moses-support@mit.edu
Message-ID: <54EFC76D.8020708@precisiontranslationtools.com>
Content-Type: text/plain; charset="utf-8"
The temp space challenge with train-model.perl is not only how much temp
space is needed, but also where are the temp files created?
train-model.perl seems unpredictable about where temp files are placed.
In general, the /tmp folder goes unused unless you specify it with the
"--temp-dir" option. Even when specified, only the sort functions use
the configuration it. Without setting the option,e sort temp files go to
the "--model-dir" folder, which by default is "--root-dir / model".
Finally, if not specified, --root-dir defaults to the current working
directory (i.e. cwd or ".").
That said, the temp folder for extraction step 3 is different. Its temp
files default to a newly-created "tmp" folder under the parent directory
of the path value set with "--extract-file". If "--extract-file" is not
set, the default is "--model-dir / extract", which also follows the
"--model-dir" path defined above.
Because of this complexity, we advise our customers to use hardware
configuration that relies on one large root folder partition... i.e. 1
or 2 TB mounted as "/" root without any other mounts. This gives Moses
one contiguous pool of temp storage independent of the configuration
complexities. I acknowledge this recommendation violates normal system
administrators' configurations, but without significant changes to
train-model.perl, this is our best recommendation.
On 02/27/2015 04:28 AM, Barry Haddow wrote:
> Hi Alexander
>
> From the error logs, it looks as though alignment went fine, the
> training pipeline reports 24860460 lines of aligned bitext. Since the
> extract files were empty, I'd suggest that extraction crashed, and the
> most likely is that it ran out of disk. I'm not sure what happened to
> the error messages.
>
> For 25M sentence pairs, the final phrase table could easily be 30G and
> the intermediate files are larger. You probably need more like 500G to
> be safe.
>
> I would follow Tom's advice and start with a much smaller corpus to
> see how the process works. Also, for the full corpus, you could look
> in to fast_align (https://github.com/clab/fast_align) for alignment as
> it is much faster than mgiza (e.g. 2 days versus 2 weeks), and use EMS
> for large jobs since it's much easier to restart a failed step.
>
> cheers - Barry
>
> On 26/02/15 15:06, ????????? ??????? wrote:
>>
>> Hi Barry!
>>
>> Here you can download training.out
>> https://www.dropbox.com/s/d0f0n99x4wbw3mo/training.out.gz?dl=1
>>
>> I have about 50 Gb of free space in working dir.
>>
>>
>> 2015-02-25 17:19 GMT+07:00 Barry Haddow <bhaddow@staffmail.ed.ac.uk
>> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>>
>> Hi Alexander,
>>
>> It looks like something went wrong at the extract stage. If you
>> could make your training.out available then we can look for clues.
>>
>> Could the system have run out of disk space, either in the
>> working directory or in /tmp? A lot of space is required to build
>> the extract files and phrase tables.
>>
>> cheers - Barry
>>
>>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150227/2f4f190f/attachment-0001.htm
------------------------------
Message: 4
Date: Fri, 27 Feb 2015 05:49:00 -0600
From: Lane Schwartz <dowobeha@gmail.com>
Subject: Re: [Moses-support] (no subject)
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZki12jPV_Kvssm+b1aa0s871J_GwXYDVomf=RBtSyB8aA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
I think that Hieu has an Eclipse project.
On Thu, Feb 26, 2015 at 7:00 PM, Tom Hoar
<tahoar@precisiontranslationtools.com> wrote:
> Not to my knowledge. Your contribution of one would be welcome.
>
> On 02/27/2015 07:27 AM, Winston Yuen wrote:
>
> Is there Xcode or Eclipse project for Moses? I try to find it online but
> cannot find it.
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
--
When a place gets crowded enough to require ID's, social collapse is not
far away. It is time to go elsewhere. The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
------------------------------
Message: 5
Date: Fri, 27 Feb 2015 05:49:23 -0600
From: Lane Schwartz <dowobeha@gmail.com>
Subject: Re: [Moses-support] (no subject)
To: undisclosed-recipients:;
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZmB3mkxrBgnP3ikNxS9w12Xhe5naWTyKbEaLeB9RUJB-Q@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Check in the contrib directory. It may be there.
On Fri, Feb 27, 2015 at 5:49 AM, Lane Schwartz <dowobeha@gmail.com> wrote:
> I think that Hieu has an Eclipse project.
>
> On Thu, Feb 26, 2015 at 7:00 PM, Tom Hoar
> <tahoar@precisiontranslationtools.com> wrote:
>> Not to my knowledge. Your contribution of one would be welcome.
>>
>> On 02/27/2015 07:27 AM, Winston Yuen wrote:
>>
>> Is there Xcode or Eclipse project for Moses? I try to find it online but
>> cannot find it.
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
>
> --
> When a place gets crowded enough to require ID's, social collapse is not
> far away. It is time to go elsewhere. The best thing about space travel
> is that it made it possible to go elsewhere.
> -- R.A. Heinlein, "Time Enough For Love"
--
When a place gets crowded enough to require ID's, social collapse is not
far away. It is time to go elsewhere. The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 100, Issue 94
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 100, Issue 94"
Post a Comment