Moses-support Digest, Vol 103, Issue 9

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: lmplz error (Marcin Junczys-Dowmunt)
2. Re: lmplz error (Hieu Hoang)
3. Extract phrases with no internal alignment points (Miguel Gra?a)
4. Re: CeateOnDiskPt throws 'Already Saved' Exception on phrase
table in sample_models/phrase-model (kwame porter robinson)


----------------------------------------------------------------------

Message: 1
Date: Tue, 05 May 2015 10:03:34 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] lmplz error
To: moses-support@mit.edu
Message-ID: <55487956.3080907@amu.edu.pl>
Content-Type: text/plain; charset="windows-1252"

Hi,
lose the second "--prune 0 0 1" :)

W dniu 05.05.2015 o 10:00, Hieu Hoang pisze:
> I'm getting this error when running lmplz (thru the EMS) which has
> never happened before
>
> .../lmplz --text .../MultiUN.ar-en.tok.2 --order 5 --arpa
> .../MultiUN.ar-en.lm.3 --prune 0 0 1 -T .../lm -S 20% --prune 0 0 1
>
> /scratch/hh65/workspace/github/mosesdecoder/lm/builder/lmplz_main.cc:54 in
> std::vector<long unsigned int> {anonymous}::ParsePruning(const
> std::vector<std::basic_string<char> >&, std::size_t) threw
> util::Exception because `prune_thresholds.size() > order'.
> You specified pruning thresholds for orders 1 through 6 but the model
> only has order 5
>
> Changing --order 6 gives another error:
>
> /scratch/hh65/workspace/github/mosesdecoder/lm/builder/lmplz_main.cc:60 in
> std::vector<long unsigned int> {anonymous}::ParsePruning(const
> std::vector<std::basic_string<char> >&, std::size_t) threw
> util::Exception because `lower_threshold > *it'.
> Pruning thresholds should be in non-decreasing order. Otherwise
> substrings would be removed, which is bad for query-time data structures.
>
>
> Hieu Hoang
> Researcher
> New York University, Abu Dhabi
> http://www.hoang.co.uk/hieu
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150505/3532d70d/attachment-0001.htm

------------------------------

Message: 2
Date: Tue, 5 May 2015 12:16:36 +0400
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] lmplz error
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbih1Y-khy-dFe2fK9jW0uST+xSJQ_h-KsB6gJb2K_mA5Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

ah thx, fixed 1 bug, made another


Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

On 5 May 2015 at 12:03, Marcin Junczys-Dowmunt <junczys@amu.edu.pl> wrote:

> Hi,
> lose the second "--prune 0 0 1" :)
>
> W dniu 05.05.2015 o 10:00, Hieu Hoang pisze:
>
> I'm getting this error when running lmplz (thru the EMS) which has never
> happened before
>
> .../lmplz --text .../MultiUN.ar-en.tok.2 --order 5 --arpa
> .../MultiUN.ar-en.lm.3 --prune 0 0 1 -T .../lm -S 20% --prune 0 0 1
>
> /scratch/hh65/workspace/github/mosesdecoder/lm/builder/lmplz_main.cc:54 in
> std::vector<long unsigned int> {anonymous}::ParsePruning(const
> std::vector<std::basic_string<char> >&, std::size_t) threw util::Exception
> because `prune_thresholds.size() > order'.
> You specified pruning thresholds for orders 1 through 6 but the model only
> has order 5
>
> Changing --order 6 gives another error:
>
> /scratch/hh65/workspace/github/mosesdecoder/lm/builder/lmplz_main.cc:60 in
> std::vector<long unsigned int> {anonymous}::ParsePruning(const
> std::vector<std::basic_string<char> >&, std::size_t) threw util::Exception
> because `lower_threshold > *it'.
> Pruning thresholds should be in non-decreasing order. Otherwise
> substrings would be removed, which is bad for query-time data structures.
>
>
> Hieu Hoang
> Researcher
> New York University, Abu Dhabi
> http://www.hoang.co.uk/hieu
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150505/27a94193/attachment-0001.htm

------------------------------

Message: 3
Date: Tue, 5 May 2015 11:18:15 +0200
From: Miguel Gra?a <miguelrgraca@gmail.com>
Subject: [Moses-support] Extract phrases with no internal alignment
points
To: moses-support@mit.edu
Message-ID:
<CANBbWMMconW3nEXR1m=fW1J=M1sfoZ-2vNhpi=8vx_q6jJ4S3Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hello,

I couldn't find any information about this. Is it possible to not only
extract phrases which have at least one alignment point, but also extract
phrases where there are no alignment points?

Thanks in advance,
Miguel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150505/e72a4cbf/attachment-0001.htm

------------------------------

Message: 4
Date: Tue, 5 May 2015 09:22:57 -0400
From: kwame porter robinson <k.porter.robinson@gmail.com>
Subject: Re: [Moses-support] CeateOnDiskPt throws 'Already Saved'
Exception on phrase table in sample_models/phrase-model
To: moses-support@mit.edu
Message-ID:
<CA+RrxgU8En1J6Bi=3C0XKoOgBL0HnOwdnEN0SdAG0mnjTK_C9w@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hmm, so I tried that and received the same exception:

$ CreateOnDiskPt 1 1 1 10 2 big-pt ondiskpt

Starting : [0] seconds
terminate called after throwing an instance of 'util::Exception'
what(): OnDiskPt/PhraseNode.cpp:97 in void
OnDiskPt::PhraseNode::Save(OnDiskPt::OnDiskWrapper&, size_t, size_t) threw
util::Exception because `m_saved'.
Already saved
Aborted (core dumped)

- Kwame

On Tue, May 5, 2015 at 12:48 AM, Hieu Hoang <hieuhoang@gmail.com> wrote:

>
>
> On 05/05/2015 02:46, kwame porter robinson wrote:
>
> Hi,
>
> I am attempting to binarize a large ascii phrase table using
> CreateOnDiskPt. Each phrase table row contains the source phrase, target
> phrase and a single score. I am getting a 'Already Saved' exception for
> phrase tables larger than 9 lines. I am using Moses release 2.1.
>
> The exception is reproduced below:
>
> 1) Using the 10 line phrase table below*, fromhttp://www.statmt.org/moses/download/sample-models.tgz.
>
> 2) Truncate the phrase-table with 'head phrase-table -n9 > small-pt'
>
> 3) Truncate another version of the phrase-table with 'head phrase-table
> -n10 > big-pt'
>
> 4) CreateOnDisk works with 'CreateOnDiskPt 0 0 1 10 2 small-pt
> myphrasetable'
>
> it should be
> CreateOnDiskPt 1 1 ....
> The 1st two arguments are the NUMBER of source and target factors.
>
> 5) CreateOnDisk throws the following exception with 'CreateOnDiskPt 0 0 1
> 10 2 big-pt myphrasetable'
>
> ---
> Starting : [0] seconds
> terminate called after throwing an instance of 'util::Exception'
> what(): OnDiskPt/PhraseNode.cpp:97 in void
> OnDiskPt::PhraseNode::Save(OnDiskPt::OnDiskWrapper&, size_t, size_t) threw
> util::Exception because `m_saved'.
> Already saved
> Aborted (core dumped)
> ---
>
> Any thoughts on how to fix this? For hints I've looked at:https://www.mail-archive.com/moses-support%40mit.edu/msg11999.html andhttps://www.mail-archive.com/moses-support%40mit.edu/msg10602.html
> but was unable to resolve this.
>
> Thank you for your time.
>
> * The phrase table
> ----
> der ||| the ||| 0.3 ||| |||
> das ||| the ||| 0.4 ||| |||
> das ||| it ||| 0.1 ||| |||
> das ||| this ||| 0.1 ||| |||
> die ||| the ||| 0.3 ||| |||
> ist ||| is ||| 1.0 ||| |||
> ist ||| 's ||| 1.0 ||| |||
> das ist ||| it is ||| 0.2 ||| |||
> das ist ||| this is ||| 0.8 ||| |||
> es ist ||| it is ||| 0.8 ||| |||
> ---
>
> - Kwame
>
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> --
> Hieu Hoang
> Researcher
> New York University, Abu Dhabihttp://www.hoang.co.uk/hieu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150505/1052da31/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 103, Issue 9
*********************************************

Tweet