Moses-support Digest, Vol 127, Issue 3

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: CreateProbingPT2 exception (Nikolay Bogoychev)
2. Re: CreateProbingPT2 exception (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Tue, 2 May 2017 10:56:49 +0100
From: Nikolay Bogoychev <nheart@gmail.com>
Subject: Re: [Moses-support] CreateProbingPT2 exception
To: Mike Ladwig <mdladwig@gmail.com>
Cc: Moses Support <moses-support@mit.edu>
Message-ID:
<CAJzPUEz79sXH2eOKpNq0zbE1Se-5GD4a0EnenSZVG=sLvxd7ng@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hey Mike,

Is it possible for you to make the phrase table available
/home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.txt.gz
publically so we can try to reproduce the problem?

Cheers,

Nick

On Tue, May 2, 2017 at 6:06 AM, Mike Ladwig <mdladwig@gmail.com> wrote:
> Got an exception creating a PT2 with yesterday's master:
>
> Binarize phrase and reordering model in probing table format:
> /home/mike/stelae5/mosesdecoder/scripts/generic/binarize4moses2.perl
> --phrase-table=/home/mike/stelae-projects/de-en/phrasemodel/model/phrase-table.gz
> --lex-ro=/home/mike/stelae-projects/de-en/phrasemodel/model/reordering-table.wbe-msd-bidirectional-fe.gz
> --output-dir=/home/mike/stelae-projects/de-en/phrasemodel/PT2
> --num-lex-scores=6
> Executing: gzip -dc
> /home/mike/stelae-projects/de-en/phrasemodel/model/phrase-table.gz |
> /home/mike/stelae5/mosesdecoder/scripts/generic/../../contrib/sigtest-filter/filter-pt
> -n 0 | gzip -c >
> /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.gz
> sh:
> /home/mike/stelae5/mosesdecoder/scripts/generic/../../contrib/sigtest-filter/filter-pt:
> No such file or directory
> Executing:
> /home/mike/stelae5/mosesdecoder/scripts/generic/../../bin/processLexicalTableMin
> -in
> /home/mike/stelae-projects/de-en/phrasemodel/model/reordering-table.wbe-msd-bidirectional-fe.gz
> -out /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/lex-ro -T .
> -threads all
> Used options:
> Text reordering table will be read from:
> /home/mike/stelae-projects/de-en/phrasemodel/model/reordering-table.wbe-msd-bidirectional-fe.gz
> Output reordering table will be written to:
> /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/lex-ro.minlexr
> Step size for source landmark phrases: 2^10=1024
> Phrase fingerprint size: 16 bits / P(fp)=1.52588e-05
> Single Huffman code set for score components: no
> Using score quantization: no
> Running with 24 threads
>
> Pass 1/2: Creating phrase index + Counting scores
> ..................................................[5000000]
> ..................................................[10000000]
> ..................................................[15000000]
> ..................................................[20000000]
> ..................................................[25000000]
> ..................................................[30000000]
> ..................................................[35000000]
> ..................................................[40000000]
> ..................................................[45000000]
> ..................................................[50000000]
> ..................................................[55000000]
> ..................................................[60000000]
> ..................................................[65000000]
> ..................................................[70000000]
> ..................................................[75000000]
> ....................
>
> Intermezzo: Calculating Huffman code sets
> Creating Huffman codes for 32003 scores
> Creating Huffman codes for 16732 scores
> Creating Huffman codes for 31335 scores
> Creating Huffman codes for 32076 scores
> Creating Huffman codes for 15096 scores
> Creating Huffman codes for 31659 scores
>
> Pass 2/2: Compressing scores
> ..................................................[5000000]
> ..................................................[10000000]
> ..................................................[15000000]
> ..................................................[20000000]
> ..................................................[25000000]
> ..................................................[30000000]
> ..................................................[35000000]
> ..................................................[40000000]
> ..................................................[45000000]
> ..................................................[50000000]
> ..................................................[55000000]
> ..................................................[60000000]
> ..................................................[65000000]
> ..................................................[70000000]
> ..................................................[75000000]
> ....................
>
> Saving to
> /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/lex-ro.minlexr
> Done
> Executing:
> /home/mike/stelae5/mosesdecoder/scripts/generic/../../bin/addLexROtoPT
> /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.gz
> /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/lex-ro.minlexr |
> gzip -c >
> /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.withLexRO.gz
> Executing: ln -s pt.withLexRO.gz
> /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.txt.gz
> Executing:
> /home/mike/stelae5/mosesdecoder/scripts/generic/../../bin/CreateProbingPT2
> --num-scores 4 --log-prob --input-pt
> /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.txt.gz
> --output-dir /home/mike/stelae-projects/de-en/phrasemodel/PT2
> --num-lex-scores 6
> Starting...
> Reading phrase table finished, writing remaining files to disk.
> terminate called after throwing an instance of 'util::ProbingSizeException'
> what(): ./util/probing_hash_table.hh:150 in
> util::ProbingHashTable<EntryT, HashT, EqualT, ModT>::Entry*
> util::ProbingHashTable<EntryT, HashT, EqualT, ModT>::Insert(const T&) [with
> T = probingpt::Entry; EntryT = probingpt::Entry; HashT = boost::hash<long
> unsigned int>; EqualT = std::equal_to<long unsigned int>; ModT =
> util::DivMod; util::ProbingHashTable<EntryT, HashT, EqualT,
> ModT>::MutableIterator = probingpt::Entry*; util::ProbingHashTable<EntryT,
> HashT, EqualT, ModT>::Entry = probingpt::Entry] threw ProbingSizeException
> because `++entries_ >= buckets_'.
> Hash table with 1 buckets is full.
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

Message: 2
Date: Tue, 2 May 2017 10:59:30 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] CreateProbingPT2 exception
To: Nikolay Bogoychev <nheart@gmail.com>
Cc: Moses Support <moses-support@mit.edu>, Mike Ladwig
<mdladwig@gmail.com>
Message-ID:
<CAEKMkbjt-9oEx-soji68MF26D-yQMcFtfMQN=MUmGXtpZGxrmg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

also, make sure you compile
contrib/sigtest-filter
I've added some checks

https://github.com/moses-smt/mosesdecoder/commit/101e52da60a653b3d62dfff75187a657fcf60a94

* Looking for MT/NLP opportunities *
Hieu Hoang
http://moses-smt.org/


On 2 May 2017 at 10:56, Nikolay Bogoychev <nheart@gmail.com> wrote:

> Hey Mike,
>
> Is it possible for you to make the phrase table available
> /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.txt.gz
> publically so we can try to reproduce the problem?
>
> Cheers,
>
> Nick
>
> On Tue, May 2, 2017 at 6:06 AM, Mike Ladwig <mdladwig@gmail.com> wrote:
> > Got an exception creating a PT2 with yesterday's master:
> >
> > Binarize phrase and reordering model in probing table format:
> > /home/mike/stelae5/mosesdecoder/scripts/generic/binarize4moses2.perl
> > --phrase-table=/home/mike/stelae-projects/de-en/
> phrasemodel/model/phrase-table.gz
> > --lex-ro=/home/mike/stelae-projects/de-en/phrasemodel/
> model/reordering-table.wbe-msd-bidirectional-fe.gz
> > --output-dir=/home/mike/stelae-projects/de-en/phrasemodel/PT2
> > --num-lex-scores=6
> > Executing: gzip -dc
> > /home/mike/stelae-projects/de-en/phrasemodel/model/phrase-table.gz |
> > /home/mike/stelae5/mosesdecoder/scripts/generic/.
> ./../contrib/sigtest-filter/filter-pt
> > -n 0 | gzip -c >
> > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.gz
> > sh:
> > /home/mike/stelae5/mosesdecoder/scripts/generic/.
> ./../contrib/sigtest-filter/filter-pt:
> > No such file or directory
> > Executing:
> > /home/mike/stelae5/mosesdecoder/scripts/generic/../../bin/
> processLexicalTableMin
> > -in
> > /home/mike/stelae-projects/de-en/phrasemodel/model/
> reordering-table.wbe-msd-bidirectional-fe.gz
> > -out /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/lex-ro -T .
> > -threads all
> > Used options:
> > Text reordering table will be read from:
> > /home/mike/stelae-projects/de-en/phrasemodel/model/
> reordering-table.wbe-msd-bidirectional-fe.gz
> > Output reordering table will be written to:
> > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/lex-ro.minlexr
> > Step size for source landmark phrases: 2^10=1024
> > Phrase fingerprint size: 16 bits / P(fp)=1.52588e-05
> > Single Huffman code set for score components: no
> > Using score quantization: no
> > Running with 24 threads
> >
> > Pass 1/2: Creating phrase index + Counting scores
> > ..................................................[5000000]
> > ..................................................[10000000]
> > ..................................................[15000000]
> > ..................................................[20000000]
> > ..................................................[25000000]
> > ..................................................[30000000]
> > ..................................................[35000000]
> > ..................................................[40000000]
> > ..................................................[45000000]
> > ..................................................[50000000]
> > ..................................................[55000000]
> > ..................................................[60000000]
> > ..................................................[65000000]
> > ..................................................[70000000]
> > ..................................................[75000000]
> > ....................
> >
> > Intermezzo: Calculating Huffman code sets
> > Creating Huffman codes for 32003 scores
> > Creating Huffman codes for 16732 scores
> > Creating Huffman codes for 31335 scores
> > Creating Huffman codes for 32076 scores
> > Creating Huffman codes for 15096 scores
> > Creating Huffman codes for 31659 scores
> >
> > Pass 2/2: Compressing scores
> > ..................................................[5000000]
> > ..................................................[10000000]
> > ..................................................[15000000]
> > ..................................................[20000000]
> > ..................................................[25000000]
> > ..................................................[30000000]
> > ..................................................[35000000]
> > ..................................................[40000000]
> > ..................................................[45000000]
> > ..................................................[50000000]
> > ..................................................[55000000]
> > ..................................................[60000000]
> > ..................................................[65000000]
> > ..................................................[70000000]
> > ..................................................[75000000]
> > ....................
> >
> > Saving to
> > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/lex-ro.minlexr
> > Done
> > Executing:
> > /home/mike/stelae5/mosesdecoder/scripts/generic/../../bin/addLexROtoPT
> > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.gz
> > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/lex-ro.minlexr |
> > gzip -c >
> > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.withLexRO.gz
> > Executing: ln -s pt.withLexRO.gz
> > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.txt.gz
> > Executing:
> > /home/mike/stelae5/mosesdecoder/scripts/generic/.
> ./../bin/CreateProbingPT2
> > --num-scores 4 --log-prob --input-pt
> > /home/mike/stelae-projects/de-en/phrasemodel/tmp.15419/pt.txt.gz
> > --output-dir /home/mike/stelae-projects/de-en/phrasemodel/PT2
> > --num-lex-scores 6
> > Starting...
> > Reading phrase table finished, writing remaining files to disk.
> > terminate called after throwing an instance of
> 'util::ProbingSizeException'
> > what(): ./util/probing_hash_table.hh:150 in
> > util::ProbingHashTable<EntryT, HashT, EqualT, ModT>::Entry*
> > util::ProbingHashTable<EntryT, HashT, EqualT, ModT>::Insert(const T&)
> [with
> > T = probingpt::Entry; EntryT = probingpt::Entry; HashT = boost::hash<long
> > unsigned int>; EqualT = std::equal_to<long unsigned int>; ModT =
> > util::DivMod; util::ProbingHashTable<EntryT, HashT, EqualT,
> > ModT>::MutableIterator = probingpt::Entry*;
> util::ProbingHashTable<EntryT,
> > HashT, EqualT, ModT>::Entry = probingpt::Entry] threw
> ProbingSizeException
> > because `++entries_ >= buckets_'.
> > Hash table with 1 buckets is full.
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170502/21ff6683/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 127, Issue 3
*********************************************

Tweet