Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. probingPT with lexicalized reordering and moses2 (Jorg Tiedemann)
2. Re: probingPT with lexicalized reordering and moses2 (Hieu Hoang)
3. Discount fallback in KenLM on super small file (liling tan)
4. Re: Discount fallback in KenLM on super small file
(Kenneth Heafield)
----------------------------------------------------------------------
Message: 1
Date: Sun, 23 Apr 2017 19:30:41 +0300
From: Jorg Tiedemann <tiedeman@gmail.com>
Subject: [Moses-support] probingPT with lexicalized reordering and
moses2
To: moses-support <moses-support@mit.edu>
Message-ID: <97985914-199F-4AB0-8CC6-B9C472A018CA@gmail.com>
Content-Type: text/plain; charset="utf-8"
a quick question about probingPT with integrated lexicalized reordering:
How do I specify the path to the binarized model in the moses2 config file?
Thanks!
J?rg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170423/7f7bc040/attachment-0001.html
------------------------------
Message: 2
Date: Sun, 23 Apr 2017 20:58:28 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] probingPT with lexicalized reordering and
moses2
To: Jorg Tiedemann <tiedeman@gmail.com>, moses-support
<moses-support@mit.edu>
Message-ID: <81f63de4-edc9-8f23-3b2e-014ac4bf0580@gmail.com>
Content-Type: text/plain; charset="windows-1252"
[feature]
ProbingPT path={directory of probing pt} ...
this is an example of the moses.ini file with it:
http://statmt.org/~s0565741/download/for-ivan/moses.tuned.ini.3.probing.intergratedRO
On 23/04/2017 17:30, Jorg Tiedemann wrote:
>
> a quick question about probingPT with integrated lexicalized reordering:
> How do I specify the path to the binarized model in the moses2 config
> file?
>
> Thanks!
> J?rg
>
>
>
>
>
>
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
--
Hieu Hoang
http://moses-smt.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170423/6fa1cfb2/attachment-0001.html
------------------------------
Message: 3
Date: Mon, 24 Apr 2017 11:18:41 +0800
From: liling tan <alvations@gmail.com>
Subject: [Moses-support] Discount fallback in KenLM on super small
file
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAKzPaJL9m2X3GEKX4bivtJru686EzumWpzbSNE-WwJKdC8RU+A@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Dear Moses community,
Is it correct that when using --discount_fallback, if discount is
computable from Kneyser-Ney, the fallback will not be used?
Regards,
Liling
For context:
$ cat test.zh
? ? ? ?
~$ time ~/mosesdecoder/bin/lmplz -o 3 --discount_fallback=0 < test.zh >
test.zh.arpa
=== 1/5 Counting and sorting n-grams === Reading /home/ltan/test.zh
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Unigram tokens 4 types 7
=== 2/5 Calculating and sorting adjusted counts
=== Chain sizes: 1:84 2:75238187008 3:141071597568
Substituting fallback discounts for order 0: D1=0 D2=0 D3+=0
Substituting fallback discounts for order 1: D1=0 D2=0 D3+=0
Substituting fallback discounts for order 2: D1=0 D2=0 D3+=0
Statistics:
1 7 D1=0 D2=0 D3+=0
2 5 D1=0 D2=0 D3+=0
3 4 D1=0 D2=0 D3+=0
Memory estimate for binary LM:
type B probing 376 assuming -p 1.5 probing 436 assuming -r models
-p 1.5 trie 298 without quantization trie 3328 assuming -q 8 -b 8
quantization trie 321 assuming -a 22 array pointer compression trie
3351 assuming -a 22 -q 8 -b 8 array pointer compression and quantization
=== 3/5 Calculating and sorting initial probabilities ===
Chain sizes: 1:84 2:80 3:80
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
####################################################################################################
=== 4/5 Calculating and writing order-interpolated probabilities ===
Chain sizes: 1:84 2:80 3:80
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
####################################################################################################
=== 5/5 Writing ARPA model ===
Name:lmplz VmPeak:211512300 kB VmRSS:4740 kB RSSMax:48754048
kB user:2.64 sys:159.492
CPU:162.132 real:162.148
real 2m42.152s
user 0m2.644s
sys 2m39.492s
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170423/93374b85/attachment-0001.html
------------------------------
Message: 4
Date: Mon, 24 Apr 2017 06:53:26 +0000
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] Discount fallback in KenLM on super small
file
To: moses-support@mit.edu
Message-ID: <D2B65503-ACF5-43DE-BCBA-C3B9132111FC@kheafield.com>
Content-Type: text/plain; charset="utf-8"
Yes. Formally, the condition is in range, not just computable.
On April 24, 2017 4:18:41 AM GMT+01:00, liling tan <alvations@gmail.com> wrote:
>Dear Moses community,
>
>Is it correct that when using --discount_fallback, if discount is
>computable from Kneyser-Ney, the fallback will not be used?
>
>
>Regards,
>Liling
>
>For context:
>
>
>
>$ cat test.zh
>
>? ? ? ?
>
>
>~$ time ~/mosesdecoder/bin/lmplz -o 3 --discount_fallback=0 < test.zh >
>test.zh.arpa
>
>
>
>=== 1/5 Counting and sorting n-grams === Reading /home/ltan/test.zh
>
>----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>
>
>****************************************************************************************************
>
>
>Unigram tokens 4 types 7
>
>
>
>=== 2/5 Calculating and sorting adjusted counts
>
>
>
>=== Chain sizes: 1:84 2:75238187008 3:141071597568
>
>
>
>Substituting fallback discounts for order 0: D1=0 D2=0 D3+=0
>
>Substituting fallback discounts for order 1: D1=0 D2=0 D3+=0
>
>Substituting fallback discounts for order 2: D1=0 D2=0 D3+=0
>
>Statistics:
>
>1 7 D1=0 D2=0 D3+=0
>
>2 5 D1=0 D2=0 D3+=0
>
>3 4 D1=0 D2=0 D3+=0
>
>Memory estimate for binary LM:
>
>type B probing 376 assuming -p 1.5 probing 436 assuming -r
>models
>-p 1.5 trie 298 without quantization trie 3328 assuming -q 8 -b
>8
>quantization trie 321 assuming -a 22 array pointer compression
>trie
>3351 assuming -a 22 -q 8 -b 8 array pointer compression and
>quantization
>
>
>
>=== 3/5 Calculating and sorting initial probabilities ===
>
>Chain sizes: 1:84 2:80 3:80
>
>----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>
>
>####################################################################################################
>
>
>
>
>=== 4/5 Calculating and writing order-interpolated probabilities ===
>
>Chain sizes: 1:84 2:80 3:80
>----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>####################################################################################################
>
>
>
>
>=== 5/5 Writing ARPA model ===
>
>Name:lmplz VmPeak:211512300 kB VmRSS:4740 kB
>RSSMax:48754048
>kB user:2.64 sys:159.492
>CPU:162.132 real:162.148
>
>
>
>real 2m42.152s
>
>user 0m2.644s
>
>sys 2m39.492s
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170424/23e7490d/attachment.html
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 126, Issue 32
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 126, Issue 32"
Post a Comment