Moses-support Digest, Vol 125, Issue 53

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. lattice mbr output empty translation result (Angli Liu)
2. lmplz crashed on joint_order (Dingyuan Wang)
3. Re: lmplz crashed on joint_order (Kenneth Heafield)


----------------------------------------------------------------------

Message: 1
Date: Wed, 29 Mar 2017 01:53:20 -0700
From: Angli Liu <anglil@cs.washington.edu>
Subject: [Moses-support] lattice mbr output empty translation result
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAJXA_ePYQwGYKtXTnijJ6FVRANhXDZ+7rpVfXbtKysyv2syQfg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi, I was using lattice mbr to decode the source sentences; the model was
tuned using mert. However, despite the fact that other decoding methods
such as maximum probability decoding and consensus decoding can output
results without a problem, mbr decoding using the -lmbr flag let the
decoder output an empty file, whatever size, scale and pruning factor I
set.

In its simplest form, the code that caused this problem is essentially
equivalent to the following:

moses \
-f moses.ini \
-output-unknowns file1 \
-n-best-list file2 50 \
-output-search-graph file3 \
-lmbr \
(-lmbr-p 0.8 -lmbr-r 0.8 -mbr-scale 5 -lmbr-pruning-factor 50) \
< in_file \
> out_file

1. parameters in parentheses are optional, though either way nothing was
output by the decoder.
2. the problem essentially is that it is out_file that tuned out to be
empty.

What was the problem? Thanks for your input in advance!

-Angli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170329/0c01239e/attachment-0001.html

------------------------------

Message: 2
Date: Wed, 29 Mar 2017 22:39:20 +0800
From: Dingyuan Wang <abcdoyle888@gmail.com>
Subject: [Moses-support] lmplz crashed on joint_order
To: moses-support <moses-support@mit.edu>
Message-ID: <722cae85-2948-a4a4-18ab-4fbb2f655c6b@gmail.com>
Content-Type: text/plain; charset=utf-8

Dear list,

lmplz crashed on my machine recently. Command is

lmplz -o 4 -S 70% --text zhc-simp.txt --arpa zhc.lm --prune 0 1 1 2

=== 1/5 Counting and sorting n-grams ===
Reading /home/gumble/docs/E/corpus/zhs/zhc-simp.txt
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
tcmalloc: large alloc 2340552704 bytes == 0x55e7ed4f4000 @
tcmalloc: large alloc 9362194432 bytes == 0x55e878d14000 @
****************************************************************************************************
Unigram tokens 886453003 types 66249
=== 2/5 Calculating and sorting adjusted counts ===
Chain sizes: 1:794988 2:1961835648 3:3678441728 4:5885507072
tcmalloc: large alloc 5885509632 bytes == 0x55e7ed4f4000 @
tcmalloc: large alloc 1961836544 bytes == 0x55e94c29c000 @
tcmalloc: large alloc 3678445568 bytes == 0x55e9c1190000 @
Statistics:
1 66249 D1=0.549028 D2=1.18255 D3+=0.99644
2 14266408/22790840 D1=0.615082 D2=1.06095 D3+=1.47555
3 87810872/205978808 D1=0.742285 D2=1.17282 D3+=1.49899
4 62909089/415283792 D1=0.698985 D2=1.20588 D3+=1.54463
Memory estimate for binary LM:
type MB
probing 3417 assuming -p 1.5
probing 4002 assuming -r models -p 1.5
trie 1653 without quantization
trie 908 assuming -q 8 -b 8 quantization
trie 1418 assuming -a 22 array pointer compression
trie 674 assuming -a 22 -q 8 -b 8 array pointer compression and
quantization
=== 3/5 Calculating and sorting initial probabilities ===
tcmalloc: large alloc 4119576576 bytes == 0x55e94c1d8000 @
tcmalloc: large alloc 9966813184 bytes == 0x55eaaf630000 @
Chain sizes: 1:794988 2:228262528 3:1756217440 4:1509818136
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
##**********###############################################################-----##**********++#############################################################-----##************#############################################################-----##************####################################################################************####################################################################************+###################################################################*************###################################################################*************#####################################################################################
=== 4/5 Calculating and writing order-interpolated probabilities ===
Chain sizes: 1:794988 2:228262528 3:1756217440 4:1509818136
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
---------------------------------------------------------------------------------------------------terminate
called after throwing an instance of 'lm::FormatLoadException'
what(): ./lm/common/joint_order.hh:61 in void lm::JointOrder(const
util::stream::ChainPositions&, Callback&) [with Callback =
lm::builder::{anonymous}::Callback<lm::builder::{anonymous}::OutputProbBackoff>;
Compare = lm::SuffixOrder] threw FormatLoadException because `order !=
current + 1'.
Detected n-gram without matching suffix


--
Dingyuan Wang


------------------------------

Message: 3
Date: Wed, 29 Mar 2017 15:40:41 +0000
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] lmplz crashed on joint_order
To: moses-support@mit.edu, Dingyuan Wang <abcdoyle888@gmail.com>
Message-ID: <07AD9698-3016-44BB-9D5B-07AA42F91396@kheafield.com>
Content-Type: text/plain; charset="utf-8"

How embarrassing. Can you try on head from github.com/kpu/kenlm ? If that fails, I can take this off list.

Kenneth

On March 29, 2017 3:39:20 PM GMT+01:00, Dingyuan Wang <abcdoyle888@gmail.com> wrote:
>Dear list,
>
>lmplz crashed on my machine recently. Command is
>
>lmplz -o 4 -S 70% --text zhc-simp.txt --arpa zhc.lm --prune 0 1 1 2
>
>=== 1/5 Counting and sorting n-grams ===
>Reading /home/gumble/docs/E/corpus/zhs/zhc-simp.txt
>----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>tcmalloc: large alloc 2340552704 bytes == 0x55e7ed4f4000 @
>tcmalloc: large alloc 9362194432 bytes == 0x55e878d14000 @
>****************************************************************************************************
>Unigram tokens 886453003 types 66249
>=== 2/5 Calculating and sorting adjusted counts ===
>Chain sizes: 1:794988 2:1961835648 3:3678441728 4:5885507072
>tcmalloc: large alloc 5885509632 bytes == 0x55e7ed4f4000 @
>tcmalloc: large alloc 1961836544 bytes == 0x55e94c29c000 @
>tcmalloc: large alloc 3678445568 bytes == 0x55e9c1190000 @
>Statistics:
>1 66249 D1=0.549028 D2=1.18255 D3+=0.99644
>2 14266408/22790840 D1=0.615082 D2=1.06095 D3+=1.47555
>3 87810872/205978808 D1=0.742285 D2=1.17282 D3+=1.49899
>4 62909089/415283792 D1=0.698985 D2=1.20588 D3+=1.54463
>Memory estimate for binary LM:
>type MB
>probing 3417 assuming -p 1.5
>probing 4002 assuming -r models -p 1.5
>trie 1653 without quantization
>trie 908 assuming -q 8 -b 8 quantization
>trie 1418 assuming -a 22 array pointer compression
>trie 674 assuming -a 22 -q 8 -b 8 array pointer compression and
>quantization
>=== 3/5 Calculating and sorting initial probabilities ===
>tcmalloc: large alloc 4119576576 bytes == 0x55e94c1d8000 @
>tcmalloc: large alloc 9966813184 bytes == 0x55eaaf630000 @
>Chain sizes: 1:794988 2:228262528 3:1756217440 4:1509818136
>----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>##**********###############################################################-----##**********++#############################################################-----##************#############################################################-----##************####################################################################************####################################################################************+###################################################################*************###################################################################*************#####################################################################################
>=== 4/5 Calculating and writing order-interpolated probabilities ===
>Chain sizes: 1:794988 2:228262528 3:1756217440 4:1509818136
>----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>---------------------------------------------------------------------------------------------------terminate
>called after throwing an instance of 'lm::FormatLoadException'
> what(): ./lm/common/joint_order.hh:61 in void lm::JointOrder(const
>util::stream::ChainPositions&, Callback&) [with Callback =
>lm::builder::{anonymous}::Callback<lm::builder::{anonymous}::OutputProbBackoff>;
>Compare = lm::SuffixOrder] threw FormatLoadException because `order !=
>current + 1'.
>Detected n-gram without matching suffix
>
>
>--
>Dingyuan Wang
>_______________________________________________
>Moses-support mailing list
>Moses-support@mit.edu
>http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170329/420481bd/attachment-0001.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 125, Issue 53
**********************************************

Tweet