Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: 12-gram language model ARPA file for 16GB (liling tan)
2. Re: 12-gram language model ARPA file for 16GB
(Marcin Junczys-Dowmunt)
3. CeateOnDiskPt throws 'Already Saved' Exception on phrase
table in sample_models/phrase-model (kwame porter robinson)
4. Re: CeateOnDiskPt throws 'Already Saved' Exception on phrase
table in sample_models/phrase-model (Hieu Hoang)
5. lmplz error (Hieu Hoang)
----------------------------------------------------------------------
Message: 1
Date: Mon, 4 May 2015 21:06:48 +0200
From: liling tan <alvations@gmail.com>
Subject: Re: [Moses-support] 12-gram language model ARPA file for 16GB
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAKzPaJLAYmH56-ehpLABcE5JhMJTk9RX32in8bX7q-R6-vO-eg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
@Marcin, thank you for the helpful insight. I guess i'll need to ask for
more HDD space from my supervisor =)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150504/a270f217/attachment-0001.htm
------------------------------
Message: 2
Date: Mon, 04 May 2015 21:12:05 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] 12-gram language model ARPA file for 16GB
To: moses-support@mit.edu
Message-ID: <5547C485.2060001@amu.edu.pl>
Content-Type: text/plain; charset="windows-1252"
Yeah, I guess going up by one order of magnitude might be necessary.
This might be a good opportunity to try pruning? It will same some space
from the third step on.
W dniu 04.05.2015 o 21:06, liling tan pisze:
> @Marcin, thank you for the helpful insight. I guess i'll need to ask
> for more HDD space from my supervisor =)
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150504/2dd5b2b4/attachment-0001.htm
------------------------------
Message: 3
Date: Mon, 4 May 2015 18:46:48 -0400
From: kwame porter robinson <k.porter.robinson@gmail.com>
Subject: [Moses-support] CeateOnDiskPt throws 'Already Saved'
Exception on phrase table in sample_models/phrase-model
To: moses-support@mit.edu
Message-ID:
<CA+RrxgVFcDNTgbMnBVCdyLc3s6j=_ba9X3yt=DOEqN9wRQMPJQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
I am attempting to binarize a large ascii phrase table using
CreateOnDiskPt. Each phrase table row contains the source phrase, target
phrase and a single score. I am getting a 'Already Saved' exception for
phrase tables larger than 9 lines. I am using Moses release 2.1.
The exception is reproduced below:
1) Using the 10 line phrase table below*,
fromhttp://www.statmt.org/moses/download/sample-models.tgz.
2) Truncate the phrase-table with 'head phrase-table -n9 > small-pt'
3) Truncate another version of the phrase-table with 'head phrase-table
-n10 > big-pt'
4) CreateOnDisk works with 'CreateOnDiskPt 0 0 1 10 2 small-pt
myphrasetable'
5) CreateOnDisk throws the following exception with 'CreateOnDiskPt 0 0 1
10 2 big-pt myphrasetable'
---
Starting : [0] seconds
terminate called after throwing an instance of 'util::Exception'
what(): OnDiskPt/PhraseNode.cpp:97 in void
OnDiskPt::PhraseNode::Save(OnDiskPt::OnDiskWrapper&, size_t, size_t) threw
util::Exception because `m_saved'.
Already saved
Aborted (core dumped)
---
Any thoughts on how to fix this? For hints I've looked
at:https://www.mail-archive.com/moses-support%40mit.edu/msg11999.html
andhttps://www.mail-archive.com/moses-support%40mit.edu/msg10602.html
but was unable to resolve this.
Thank you for your time.
* The phrase table
----
der ||| the ||| 0.3 ||| |||
das ||| the ||| 0.4 ||| |||
das ||| it ||| 0.1 ||| |||
das ||| this ||| 0.1 ||| |||
die ||| the ||| 0.3 ||| |||
ist ||| is ||| 1.0 ||| |||
ist ||| 's ||| 1.0 ||| |||
das ist ||| it is ||| 0.2 ||| |||
das ist ||| this is ||| 0.8 ||| |||
es ist ||| it is ||| 0.8 ||| |||
---
- Kwame
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150504/8271cc8e/attachment-0001.htm
------------------------------
Message: 4
Date: Tue, 05 May 2015 08:48:52 +0400
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] CeateOnDiskPt throws 'Already Saved'
Exception on phrase table in sample_models/phrase-model
To: kwame porter robinson <k.porter.robinson@gmail.com>,
moses-support@mit.edu
Message-ID: <55484BB4.1030509@gmail.com>
Content-Type: text/plain; charset="windows-1252"
On 05/05/2015 02:46, kwame porter robinson wrote:
> Hi,
>
> I am attempting to binarize a large ascii phrase table using
> CreateOnDiskPt. Each phrase table row contains the source phrase, target
> phrase and a single score. I am getting a 'Already Saved' exception for
> phrase tables larger than 9 lines. I am using Moses release 2.1.
>
> The exception is reproduced below:
>
> 1) Using the 10 line phrase table below*, from
> http://www.statmt.org/moses/download/sample-models.tgz.
>
> 2) Truncate the phrase-table with 'head phrase-table -n9 > small-pt'
>
> 3) Truncate another version of the phrase-table with 'head phrase-table
> -n10 > big-pt'
>
> 4) CreateOnDisk works with 'CreateOnDiskPt 0 0 1 10 2 small-pt
> myphrasetable'
it should be
CreateOnDiskPt 1 1 ....
The 1st two arguments are the NUMBER of source and target factors.
>
> 5) CreateOnDisk throws the following exception with 'CreateOnDiskPt 0 0 1
> 10 2 big-pt myphrasetable'
>
> ---
> Starting : [0] seconds
> terminate called after throwing an instance of 'util::Exception'
> what(): OnDiskPt/PhraseNode.cpp:97 in void
> OnDiskPt::PhraseNode::Save(OnDiskPt::OnDiskWrapper&, size_t, size_t) threw
> util::Exception because `m_saved'.
> Already saved
> Aborted (core dumped)
> ---
>
> Any thoughts on how to fix this? For hints I've looked at:
> https://www.mail-archive.com/moses-support%40mit.edu/msg11999.html and
> https://www.mail-archive.com/moses-support%40mit.edu/msg10602.html
> but was unable to resolve this.
>
> Thank you for your time.
>
> * The phrase table
> ----
> der ||| the ||| 0.3 ||| |||
> das ||| the ||| 0.4 ||| |||
> das ||| it ||| 0.1 ||| |||
> das ||| this ||| 0.1 ||| |||
> die ||| the ||| 0.3 ||| |||
> ist ||| is ||| 1.0 ||| |||
> ist ||| 's ||| 1.0 ||| |||
> das ist ||| it is ||| 0.2 ||| |||
> das ist ||| this is ||| 0.8 ||| |||
> es ist ||| it is ||| 0.8 ||| |||
> ---
>
> - Kwame
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
--
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150505/4b39a80c/attachment-0001.htm
------------------------------
Message: 5
Date: Tue, 5 May 2015 12:00:42 +0400
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: [Moses-support] lmplz error
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbgVbB4fLKFCV_p3WqHNy8YF1i-7aL1eaqOCC+p_3jwJNQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
I'm getting this error when running lmplz (thru the EMS) which has never
happened before
.../lmplz --text .../MultiUN.ar-en.tok.2 --order 5 --arpa
.../MultiUN.ar-en.lm.3 --prune 0 0 1 -T .../lm -S 20% --prune 0 0 1
/scratch/hh65/workspace/github/mosesdecoder/lm/builder/lmplz_main.cc:54 in
std::vector<long unsigned int> {anonymous}::ParsePruning(const
std::vector<std::basic_string<char> >&, std::size_t) threw util::Exception
because `prune_thresholds.size() > order'.
You specified pruning thresholds for orders 1 through 6 but the model only
has order 5
Changing --order 6 gives another error:
/scratch/hh65/workspace/github/mosesdecoder/lm/builder/lmplz_main.cc:60 in
std::vector<long unsigned int> {anonymous}::ParsePruning(const
std::vector<std::basic_string<char> >&, std::size_t) threw util::Exception
because `lower_threshold > *it'.
Pruning thresholds should be in non-decreasing order. Otherwise substrings
would be removed, which is bad for query-time data structures.
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150505/edd28568/attachment.htm
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 103, Issue 8
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 103, Issue 8"
Post a Comment