Moses-support Digest, Vol 88, Issue 21

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. binarized rule-table (Mohammad Salameh)
2. Re: binarized rule-table (Mohammad Salameh)
3. Re: binarized rule-table (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Wed, 12 Feb 2014 02:14:25 -0700
From: Mohammad Salameh <msalameh83@gmail.com>
Subject: [Moses-support] binarized rule-table
To: moses-support@MIT.EDU
Message-ID:
<CAJjti4Z9-bhCqJJw2mVBwKnxbr_SZKT5V+iyZO1+ZDUyOy7dVQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi ,
I am trying to run a heirarichal phrase based model using a binarized rule
table.
I am using the string-to-tree example in the sample-models.tgz provided in
the tutorial. I am expecting the *rules* directory with
Misc.dat Source.dat TargetColl.dat TargetInd.dat Vocab.dat
to binarized directory of rule-table generated by running CreateOnDiskPt.

I copied moses,ini and changed these 2 lines to

*PhraseDictionaryMemory input-factor=0 output-factor=0
path=string-to-tree/rule-table num-features=1 table-limit=20*
*PhraseDictionaryMemory0= 0.5*

to


*PhraseDictionaryBinary input-factor=0 output-factor=0
path=string-to-tree/rule-table num-features=1 table-limit=20*

*PhraseDictionaryBinary0= 0.5*

When I run
echo 'das ist ein haus' | moses_chart -f string-to-tree/moses-bin.ini > out
I am getting the following error:



Defined parameters (per moses.ini or switch):
config: string-to-tree/moses-bin.ini
cube-pruning-pop-limit: 1000
feature: KENLM name=LM factor=0 order=3 num-features=1
path=lm/europarl.srilm.gz WordPenalty UnknownWordPenalty
PhraseDictionaryBinary input-factor=0 output-factor=0
path=string-to-tree/rules/ num-features=1 table-limit=20
input-factors: 0
inputtype: 3
mapping: 0 T 0
max-chart-span: 20 1000
non-terminals: X S
search-algorithm: 3
translation-details: translation-details.log
weight: WordPenalty0= 0 LM= 0.5 PhraseDictionaryBinary0= 0.5
/local/scratch/msalameh/SMT/mosesdecoder2.1/bin
line=KENLM name=LM factor=0 order=3 num-features=1 path=lm/europarl.srilm.gz
FeatureFunction: LM start: 0 end: 0
Loading the LM will be faster if you build a binary file.
Reading lm/europarl.srilm.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
**The ARPA file is missing <unk>. Substituting log10 probability -100.000.
**************************************************************************************************
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 2 end: 2
line=PhraseDictionaryBinary input-factor=0 output-factor=0
path=string-to-tree/rules/ num-features=1 table-limit=20
FeatureFunction: PhraseDictionaryBinary0 start: 3 end: 3
Loading LM
Loading WordPenalty0
Loading UnknownWordPenalty0
Loading PhraseDictionaryBinary0
max-chart-span: 20
IO from STDOUT/STDIN
Created input-output object : [1.041] seconds
terminate called after throwing an instance of 'util::Exception'
what(): ./moses/PDTAimp.h:231 in void Moses::PDTAimp::Create(const
std::vector<long unsigned int>&, const std::vector<long unsigned int>&,
const string&, const std::vector<float>&) threw util::Exception'.
bin ttable does not exist
Aborted (core dumped)

Is there anything I am missing?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140212/8c790087/attachment-0001.htm

------------------------------

Message: 2
Date: Wed, 12 Feb 2014 02:23:58 -0700
From: Mohammad Salameh <msalameh83@gmail.com>
Subject: Re: [Moses-support] binarized rule-table
To: moses-support <moses-support@mit.edu>
Message-ID:
<CADbe+DFsOfR5KYkD8n3ztC2ZhZLpgen1GbDkw-UwvyxOefrpiQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

sorry I copied moses,ini and changed these 2 lines to

*PhraseDictionaryMemory input-factor=0 output-factor=0
path=string-to-tree/rule-table num-features=1 table-limit=20*
*PhraseDictionaryMemory0= 0.5*

to


*PhraseDictionaryBinary input-factor=0 output-factor=0
path=string-to-tree/rules num-features=1 table-limit=20*
*PhraseDictionaryBinary0= 0.5*

which is giving me the above error , is there anything I am missing?


On Wed, Feb 12, 2014 at 2:14 AM, Mohammad Salameh <msalameh83@gmail.com>wrote:

> Hi ,
> I am trying to run a heirarichal phrase based model using a binarized
> rule table.
> I am using the string-to-tree example in the sample-models.tgz provided
> in the tutorial. I am expecting the *rules* directory with
> Misc.dat Source.dat TargetColl.dat TargetInd.dat Vocab.dat
> to binarized directory of rule-table generated by running CreateOnDiskPt.
>
> I copied moses,ini and changed these 2 lines to
>
> *PhraseDictionaryMemory input-factor=0 output-factor=0
> path=string-to-tree/rule-table num-features=1 table-limit=20 *
> *PhraseDictionaryMemory0= 0.5*
>
> to
>
>
> *PhraseDictionaryBinary input-factor=0 output-factor=0
> path=string-to-tree/rule-table num-features=1 table-limit=20 *
>
> *PhraseDictionaryBinary0= 0.5*
>
> When I run
> echo 'das ist ein haus' | moses_chart -f string-to-tree/moses-bin.ini > out
> I am getting the following error:
>
>
>
> Defined parameters (per moses.ini or switch):
> config: string-to-tree/moses-bin.ini
> cube-pruning-pop-limit: 1000
> feature: KENLM name=LM factor=0 order=3 num-features=1
> path=lm/europarl.srilm.gz WordPenalty UnknownWordPenalty
> PhraseDictionaryBinary input-factor=0 output-factor=0
> path=string-to-tree/rules/ num-features=1 table-limit=20
> input-factors: 0
> inputtype: 3
> mapping: 0 T 0
> max-chart-span: 20 1000
> non-terminals: X S
> search-algorithm: 3
> translation-details: translation-details.log
> weight: WordPenalty0= 0 LM= 0.5 PhraseDictionaryBinary0= 0.5
> /local/scratch/msalameh/SMT/mosesdecoder2.1/bin
> line=KENLM name=LM factor=0 order=3 num-features=1
> path=lm/europarl.srilm.gz
> FeatureFunction: LM start: 0 end: 0
> Loading the LM will be faster if you build a binary file.
> Reading lm/europarl.srilm.gz
>
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> **The ARPA file is missing <unk>. Substituting log10 probability -100.000.
>
> **************************************************************************************************
> line=WordPenalty
> FeatureFunction: WordPenalty0 start: 1 end: 1
> line=UnknownWordPenalty
> FeatureFunction: UnknownWordPenalty0 start: 2 end: 2
> line=PhraseDictionaryBinary input-factor=0 output-factor=0
> path=string-to-tree/rules/ num-features=1 table-limit=20
> FeatureFunction: PhraseDictionaryBinary0 start: 3 end: 3
> Loading LM
> Loading WordPenalty0
> Loading UnknownWordPenalty0
> Loading PhraseDictionaryBinary0
> max-chart-span: 20
> IO from STDOUT/STDIN
> Created input-output object : [1.041] seconds
> terminate called after throwing an instance of 'util::Exception'
> what(): ./moses/PDTAimp.h:231 in void Moses::PDTAimp::Create(const
> std::vector<long unsigned int>&, const std::vector<long unsigned int>&,
> const string&, const std::vector<float>&) threw util::Exception'.
> bin ttable does not exist
> Aborted (core dumped)
>
> Is there anything I am missing?
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140212/0ea29b7c/attachment-0001.htm

------------------------------

Message: 3
Date: Wed, 12 Feb 2014 10:02:29 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] binarized rule-table
To: Mohammad Salameh <msalameh83@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbjdBUsd-QAg4Pee5iYuednt5ia9V0nCuwE7-pSZyOtGnA@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

I think everything is correct, except it should be *PhraseDictionaryOnDisk
rather than **PhraseDictionaryBinary. *

*However, for string-to-tree, the binary/ondisk phrase-table is very slow.
You're better off sticking with memory phrase-table, but that requires a
lot of memory*




On 12 February 2014 09:14, Mohammad Salameh <msalameh83@gmail.com> wrote:

> Hi ,
> I am trying to run a heirarichal phrase based model using a binarized
> rule table.
> I am using the string-to-tree example in the sample-models.tgz provided
> in the tutorial. I am expecting the *rules* directory with
> Misc.dat Source.dat TargetColl.dat TargetInd.dat Vocab.dat
> to binarized directory of rule-table generated by running CreateOnDiskPt.
>
> I copied moses,ini and changed these 2 lines to
>
> *PhraseDictionaryMemory input-factor=0 output-factor=0
> path=string-to-tree/rule-table num-features=1 table-limit=20 *
> *PhraseDictionaryMemory0= 0.5*
>
> to
>
>
> *PhraseDictionaryBinary input-factor=0 output-factor=0
> path=string-to-tree/rule-table num-features=1 table-limit=20 *
>
> *PhraseDictionaryBinary0= 0.5*
>
> When I run
> echo 'das ist ein haus' | moses_chart -f string-to-tree/moses-bin.ini > out
> I am getting the following error:
>
>
>
> Defined parameters (per moses.ini or switch):
> config: string-to-tree/moses-bin.ini
> cube-pruning-pop-limit: 1000
> feature: KENLM name=LM factor=0 order=3 num-features=1
> path=lm/europarl.srilm.gz WordPenalty UnknownWordPenalty
> PhraseDictionaryBinary input-factor=0 output-factor=0
> path=string-to-tree/rules/ num-features=1 table-limit=20
> input-factors: 0
> inputtype: 3
> mapping: 0 T 0
> max-chart-span: 20 1000
> non-terminals: X S
> search-algorithm: 3
> translation-details: translation-details.log
> weight: WordPenalty0= 0 LM= 0.5 PhraseDictionaryBinary0= 0.5
> /local/scratch/msalameh/SMT/mosesdecoder2.1/bin
> line=KENLM name=LM factor=0 order=3 num-features=1
> path=lm/europarl.srilm.gz
> FeatureFunction: LM start: 0 end: 0
> Loading the LM will be faster if you build a binary file.
> Reading lm/europarl.srilm.gz
>
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> **The ARPA file is missing <unk>. Substituting log10 probability -100.000.
>
> **************************************************************************************************
> line=WordPenalty
> FeatureFunction: WordPenalty0 start: 1 end: 1
> line=UnknownWordPenalty
> FeatureFunction: UnknownWordPenalty0 start: 2 end: 2
> line=PhraseDictionaryBinary input-factor=0 output-factor=0
> path=string-to-tree/rules/ num-features=1 table-limit=20
> FeatureFunction: PhraseDictionaryBinary0 start: 3 end: 3
> Loading LM
> Loading WordPenalty0
> Loading UnknownWordPenalty0
> Loading PhraseDictionaryBinary0
> max-chart-span: 20
> IO from STDOUT/STDIN
> Created input-output object : [1.041] seconds
> terminate called after throwing an instance of 'util::Exception'
> what(): ./moses/PDTAimp.h:231 in void Moses::PDTAimp::Create(const
> std::vector<long unsigned int>&, const std::vector<long unsigned int>&,
> const string&, const std::vector<float>&) threw util::Exception'.
> bin ttable does not exist
> Aborted (core dumped)
>
> Is there anything I am missing?
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140212/23c87f51/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 88, Issue 21
*********************************************

Tweet