Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. RAM and SWAP reached its limit (Asad)
2. Re: RAM and SWAP reached its limit (rohit dholakia)
3. What is the meaning of in_state and out_state for a State
class (Li Xiang)
4. Re: What is the meaning of in_state and out_state for a State
class (Kenneth Heafield)
5. Preparing corpus (Arththika Paramanathan)
6. Re: Preparing corpus (rohit dholakia)
----------------------------------------------------------------------
Message: 1
Date: Mon, 23 Dec 2013 01:48:30 +0500
From: Asad <asad_12204@yahoo.com>
Subject: [Moses-support] RAM and SWAP reached its limit
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <4C6D1AEA-27E6-496C-A638-3DBCFE356AFC@yahoo.com>
Content-Type: text/plain; charset=us-ascii
Hi,
I am using the sample model mentioned in the moses manual.
In testing phases when entering command "~/mosesdecoder/bin/moses -f ~/working/mert-work/moses.ini" it uses RAM and SWAP memory in a way that they reached to their limit(3gb RAM and 3gb SWAP) and after that system stops working. Also the CPUs were acting normal (60-70%).
I would have captured the screen but system wasn't responding.
PS. It was reading "phrase-table....gz" when the RAM and SWAP reached to its limit
Regards
Asad A.Malik
------------------------------
Message: 2
Date: Sun, 22 Dec 2013 12:51:57 -0800
From: rohit dholakia <rdholaki@sfu.ca>
Subject: Re: [Moses-support] RAM and SWAP reached its limit
To: Asad <asad_12204@yahoo.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAA==Lgt4OesEm0yGsQrCBdq4_LOH1pKL0q0aokrBkyfAz=1-dw@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi Asad,
Did you binarize your phrase table ? If you go down some more in the
manual you are referring to, they mention that you should binarize the
phrase table and reordering table and then run the same command again.
Full phrase table is quite large and can take much more than just 3 Gigs.
On Sun, Dec 22, 2013 at 12:48 PM, Asad <asad_12204@yahoo.com> wrote:
> Hi,
>
> I am using the sample model mentioned in the moses manual.
> In testing phases when entering command "~/mosesdecoder/bin/moses -f
> ~/working/mert-work/moses.ini" it uses RAM and SWAP memory in a way that
> they reached to their limit(3gb RAM and 3gb SWAP) and after that system
> stops working. Also the CPUs were acting normal (60-70%).
> I would have captured the screen but system wasn't responding.
>
> PS. It was reading "phrase-table....gz" when the RAM and SWAP reached to
> its limit
>
> Regards
> Asad A.Malik
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131222/ae2c7d04/attachment-0001.htm
------------------------------
Message: 3
Date: Mon, 23 Dec 2013 11:13:25 +0800
From: Li Xiang <lixiang.ict@gmail.com>
Subject: [Moses-support] What is the meaning of in_state and out_state
for a State class
To: moses-support <moses-support@mit.edu>
Message-ID:
<CA+fVw+60neEbySn+fxPPLg2NRFRLeT5AyGLDK8gfaVX-t8=W8w@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
I want to add a new member in state to keep some information for my work.
But I am not very clear about what out_state stands for given in_state and
a new word.
In the function of ScoreExceptBackoff, I find that out_state.words[0] =
new_word. Why? For the member words in a state, is the words a sequence of
n-gram words before the new word or a inverted sequence?
Thanks.
--
Xiang Li
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131223/3d2dd870/attachment-0001.htm
------------------------------
Message: 4
Date: Sun, 22 Dec 2013 19:32:39 -0800
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] What is the meaning of in_state and
out_state for a State class
To: moses-support@mit.edu
Message-ID: <52B7AED7.9000304@kheafield.com>
Content-Type: text/plain; charset=ISO-8859-1
Hi,
State encodes the context of a language model query. So you can always
query p(foo | bar baz) but it's more efficient to query p(foo |
state(bar baz)) assuming that you already knew state(bar baz). In the
function call, state(bar, baz) is called in_state.
The function call also tells you state(bar baz foo) so that you can use
it later to add another word. That's out_state.
For entirely too much detail on state, see
http://www.cs.jhu.edu/~zfli/pubs/scalable_decoder_zhifei_acl_ssst_08.pdf
http://kheafield.com/professional/avenue/kenlm_talk.pdf sides 11--13
http://kheafield.com/professional/edinburgh/left_paper.pdf
state.words is indeed encoded in reverse order.
Kenneth
On 12/22/13 19:13, Li Xiang wrote:
> Hi,
>
> I want to add a new member in state to keep some information for my
> work. But I am not very clear about what out_state stands for given
> in_state and a new word.
>
> In the function of ScoreExceptBackoff, I find that out_state.words[0] =
> new_word. Why? For the member words in a state, is the words a sequence
> of n-gram words before the new word or a inverted sequence?
>
> Thanks.
>
> --
> Xiang Li
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
------------------------------
Message: 5
Date: Mon, 23 Dec 2013 12:55:27 +0530
From: Arththika Paramanathan <arthiparamanathan@gmail.com>
Subject: [Moses-support] Preparing corpus
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAJSfqEygFQiFUzd+0GUSrT5i_Erj1phD4wLKXYw=8UuywbWZYQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi,
I tried to prepare corpus for English - Tamil. But, it shows this warning
WARNING: No known abbreviations for language 'ta', attempting fall-back to
English version...
Where I want to specify this?
--
regards,
P.Arththika
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131223/cf880720/attachment-0001.htm
------------------------------
Message: 6
Date: Sun, 22 Dec 2013 23:32:41 -0800
From: rohit dholakia <rdholaki@sfu.ca>
Subject: Re: [Moses-support] Preparing corpus
To: Arththika Paramanathan <arthiparamanathan@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAA==LgvcmfvzzHaoo+d2EJ0NGBQsGLxasCi+wYNhCauZ3krq+A@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi Arththika,
I believe you are doing
$SCRIPTS_ROOTDIR/tokenizer/tokenizer.perl -l ta < input > output
Now, Moses does not have a Tamil tokenizer. Hence, the onus is on you to
decide how to tokenize and lowercase the text.
The absence of a tamil tokenizer is what is referred to above in the
warning.
On Sun, Dec 22, 2013 at 11:25 PM, Arththika Paramanathan <
arthiparamanathan@gmail.com> wrote:
> Hi,
> I tried to prepare corpus for English - Tamil. But, it shows this warning
> WARNING: No known abbreviations for language 'ta', attempting fall-back to
> English version...
> Where I want to specify this?
>
> --
> regards,
> P.Arththika
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131222/d4e8d286/attachment.htm
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 86, Issue 68
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 86, Issue 68"
Post a Comment