Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Error in using the new moses.ini file (Jelita Asian)
2. Re: Error in using the new moses.ini file (Hieu Hoang)
3. Kenlm, lmplz, pruning singleton n-grams, mmapping error with
build_binary (Marcin Junczys-Dowmunt)
----------------------------------------------------------------------
Message: 1
Date: Wed, 21 Aug 2013 17:11:43 +0700
From: Jelita Asian <jelitayang@gmail.com>
Subject: [Moses-support] Error in using the new moses.ini file
To: moses-support@mit.edu
Message-ID:
<CAOmUaargPH94u_TjQ+cU4fWK8pVFHnfA5CgwWv9MGtoE42ySZQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Dear all,
I have recently tried to use the new moses file to run with the
newmoses.ini file, I got this error:
My run command is:
./moses.exe -f ./moses.ini -input-file input > output
The moses.ini file is something like:
[input-factors]
0
# mapping steps
[mapping]
0 T 0
#1 T 1
[distortion-limit]
6
# feature functions
[feature]
Distortion
UnknownWordPenalty
WordPenalty
PhraseDictionaryBinary name=TranslationModel0 num-features=5 \
path=bin_en/phrase-table.AllCorpus30Apr2013.for_train.en-id
output-factor=0
LexicalReordering num-features=6 name=LexicalReordering0 \
#type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 \
type=wbe-monotonicity-bidirectional-fe input-factor=0 output-factor=0 \
path=bin_en/reordering-table.AllCorpus30Apr2013.for_train.en-id.wbe-monotonicity-bidirectional-fe
#path=bin_en/wbe-monotonicity-bidirectional-fe
KENLM name=LM0 order=5 factor=0 path=bin_en/
id.LM-AllCorpus30Apr2013.for_train-IRSTLM-4-1-improved-kneser-ney-0-1.blm.mmlazyken=0
#PhraseDictionaryBinary name=TranslationModel1 num-features=5 \
# path=/path/to/german-english/phrase-table output-factor=0
#LexicalReordering num-features=6 name=LexicalReordering1 \
# type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 \
# path=/path/to/german-english/reordering-table
#KENLM name=LM1 order=5 factor=0
path=/path/to/german-english/language-model lazyken=0
# core weights - not used
[weight]
Distortion0= 0
WordPenalty0= 0
TranslationModel0= 0 0 0 0 0
LexicalReordering0= 0 0 0 0 0 0
LM0= 0
TranslationModel1= 0 0 0 0 0
LexicalReordering1= 0 0 0 0 0 0
LM1= 0
[alternate-weight-setting]
id=en ignore-ff=LM1,LexicalReordering1 ignore-decoding-path=1
Distortion0= 0.3
LexicalReordering0= 0.074 -0.008 0.002 0.050 0.033 0.042
LM0= 0.5
WordPenalty0= -1
TranslationModel0= 0.2 0.2 0.2 0.2 0.2
#id=de ignore-ff=LM0,LexicalReordering0 ignore-decoding-path=0
#LexicalReordering1= 0.013 -0.012 0.053 0.116 0.006 0.080
#Distortion0= 0.171
#LM0= 0.136
#WordPenalty0= 0.060
#TranslationModel1= 0.112 0.160 -0.001 0.067 0.006
When I run it, I got this message:
Defined parameters (per moses.ini or switch):
alternate-weight-setting: id=en ignore-ff=LM1,LexicalReordering1
ignore-
decoding-path=1 Distortion0= 0.3 LexicalReordering0= 0.074 -0.008 0.002
0.050 0.
033 0.042 LM0= 0.5 WordPenalty0= -1 TranslationModel0= 0.2 0.2 0.2 0.2 0.2
config: ./newmoses.ini
distortion-limit: 6
feature: Distortion UnknownWordPenalty WordPenalty
PhraseDictionaryBinar
y name=TranslationModel0 num-features=5 \
path=bin_en/phrase-table.AllCorpus30Ap
r2013.for_train.en-id output-factor=0 LexicalReordering num-features=6
name=Lexi
calReordering0 \ type=wbe-monotonicity-bidirectional-fe input-factor=0
output-fa
ctor=0 \
path=bin_en/reordering-table.AllCorpus30Apr2013.for_train.en-id.wbe-mon
otonicity-bidirectional-fe KENLM name=LM0 order=5 factor=0
path=bin_en/id.LM-All
Corpus30Apr2013.for_train-IRSTLM-4-1-improved-kneser-ney-0-1.blm.mmlazyken=0
input-factors: 0
input-file: test.tmp
mapping: 0 T 0
weight: Distortion0= 0 WordPenalty0= 0 TranslationModel0= 0 0 0 0 0
Lexi
calReordering0= 0 0 0 0 0 0 LM0= 0 TranslationModel1= 0 0 0 0 0
LexicalReorderin
g1= 0 0 0 0 0 0 LM1= 0
ERROR:Unknown parameter alternate-weight-setting
ERROR:Unknown parameter feature
ERROR:Unknown parameter weight
ERROR:No phrase translation table (ttable-file)
Moses took 0 seconds.
What is the cause of the problems?
I commented the rest because I want to test with one language first.
The input file sample is:
<seg weight-setting=en>harry noticed that ron kept moving his prefects
badge around first placing it on his bedside table then putting it into his
jeans pocket </seg>
Another separate question after the first question is solved is:
Can we overwrite the default configuration in moses.ini using something like
./moses.exe -f ./moses.ini -weight-t 1 -weight-l 0.7 -weight-d 0.6
-weight-w 0.6 -mbr 1 -mbr-size 520 -input-file input > output?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20130821/bc13b1c6/attachment-0001.htm
------------------------------
Message: 2
Date: Wed, 21 Aug 2013 11:16:03 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Error in using the new moses.ini file
To: Jelita Asian <jelitayang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbgHWBS7F5CvfRr_ivOrZPANXVTtNOSRgb8txb+CpU2KdA@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
you're using the old decoder. Please 'git pull' the new code from github,
compile and try again
On 21 August 2013 11:11, Jelita Asian <jelitayang@gmail.com> wrote:
> Dear all,
>
> I have recently tried to use the new moses file to run with the
> newmoses.ini file, I got this error:
> My run command is:
> ./moses.exe -f ./moses.ini -input-file input > output
>
> The moses.ini file is something like:
> [input-factors]
> 0
>
> # mapping steps
> [mapping]
> 0 T 0
> #1 T 1
>
> [distortion-limit]
> 6
>
> # feature functions
> [feature]
> Distortion
> UnknownWordPenalty
> WordPenalty
> PhraseDictionaryBinary name=TranslationModel0 num-features=5 \
> path=bin_en/phrase-table.AllCorpus30Apr2013.for_train.en-id
> output-factor=0
> LexicalReordering num-features=6 name=LexicalReordering0 \
> #type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 \
> type=wbe-monotonicity-bidirectional-fe input-factor=0 output-factor=0
> \
>
> path=bin_en/reordering-table.AllCorpus30Apr2013.for_train.en-id.wbe-monotonicity-bidirectional-fe
> #path=bin_en/wbe-monotonicity-bidirectional-fe
> KENLM name=LM0 order=5 factor=0 path=bin_en/
> id.LM-AllCorpus30Apr2013.for_train-IRSTLM-4-1-improved-kneser-ney-0-1.blm.mmlazyken=0
> #PhraseDictionaryBinary name=TranslationModel1 num-features=5 \
> # path=/path/to/german-english/phrase-table output-factor=0
> #LexicalReordering num-features=6 name=LexicalReordering1 \
> # type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 \
> # path=/path/to/german-english/reordering-table
> #KENLM name=LM1 order=5 factor=0
> path=/path/to/german-english/language-model lazyken=0
>
> # core weights - not used
> [weight]
> Distortion0= 0
> WordPenalty0= 0
> TranslationModel0= 0 0 0 0 0
> LexicalReordering0= 0 0 0 0 0 0
> LM0= 0
> TranslationModel1= 0 0 0 0 0
> LexicalReordering1= 0 0 0 0 0 0
> LM1= 0
>
> [alternate-weight-setting]
> id=en ignore-ff=LM1,LexicalReordering1 ignore-decoding-path=1
> Distortion0= 0.3
> LexicalReordering0= 0.074 -0.008 0.002 0.050 0.033 0.042
> LM0= 0.5
> WordPenalty0= -1
> TranslationModel0= 0.2 0.2 0.2 0.2 0.2
> #id=de ignore-ff=LM0,LexicalReordering0 ignore-decoding-path=0
> #LexicalReordering1= 0.013 -0.012 0.053 0.116 0.006 0.080
> #Distortion0= 0.171
> #LM0= 0.136
> #WordPenalty0= 0.060
> #TranslationModel1= 0.112 0.160 -0.001 0.067 0.006
>
> When I run it, I got this message:
>
> Defined parameters (per moses.ini or switch):
> alternate-weight-setting: id=en ignore-ff=LM1,LexicalReordering1
> ignore-
> decoding-path=1 Distortion0= 0.3 LexicalReordering0= 0.074 -0.008 0.002
> 0.050 0.
> 033 0.042 LM0= 0.5 WordPenalty0= -1 TranslationModel0= 0.2 0.2 0.2 0.2 0.2
> config: ./newmoses.ini
> distortion-limit: 6
> feature: Distortion UnknownWordPenalty WordPenalty
> PhraseDictionaryBinar
> y name=TranslationModel0 num-features=5 \
> path=bin_en/phrase-table.AllCorpus30Ap
> r2013.for_train.en-id output-factor=0 LexicalReordering num-features=6
> name=Lexi
> calReordering0 \ type=wbe-monotonicity-bidirectional-fe input-factor=0
> output-fa
> ctor=0 \
> path=bin_en/reordering-table.AllCorpus30Apr2013.for_train.en-id.wbe-mon
> otonicity-bidirectional-fe KENLM name=LM0 order=5 factor=0
> path=bin_en/id.LM-All
> Corpus30Apr2013.for_train-IRSTLM-4-1-improved-kneser-ney-0-1.blm.mmlazyken=0
> input-factors: 0
> input-file: test.tmp
> mapping: 0 T 0
> weight: Distortion0= 0 WordPenalty0= 0 TranslationModel0= 0 0 0 0
> 0 Lexi
> calReordering0= 0 0 0 0 0 0 LM0= 0 TranslationModel1= 0 0 0 0 0
> LexicalReorderin
> g1= 0 0 0 0 0 0 LM1= 0
> ERROR:Unknown parameter alternate-weight-setting
> ERROR:Unknown parameter feature
> ERROR:Unknown parameter weight
> ERROR:No phrase translation table (ttable-file)
> Moses took 0 seconds.
>
> What is the cause of the problems?
> I commented the rest because I want to test with one language first.
>
> The input file sample is:
> <seg weight-setting=en>harry noticed that ron kept moving his prefects
> badge around first placing it on his bedside table then putting it into his
> jeans pocket </seg>
>
>
> Another separate question after the first question is solved is:
> Can we overwrite the default configuration in moses.ini using something
> like
> ./moses.exe -f ./moses.ini -weight-t 1 -weight-l 0.7 -weight-d 0.6
> -weight-w 0.6 -mbr 1 -mbr-size 520 -input-file input > output?
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20130821/06dff27b/attachment-0001.htm
------------------------------
Message: 3
Date: Wed, 21 Aug 2013 12:20:38 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: [Moses-support] Kenlm, lmplz, pruning singleton n-grams,
mmapping error with build_binary
To: moses-support@mit.edu
Message-ID: <e493af0c96189de2669ee94ca8c9f4dd@amu.edu.pl>
Content-Type: text/plain; charset="utf-8"
Hi,
This should probably go directly to Kenneth, but I guess answers might
interesting for others, too.
1) Is there a way to simulate the pruning function for singleton
n-grams (as in IRSTLM) when using lmplz from kenlm? I guess this is not
quite straight-forward with Improved Kneser-Ney smoothing used by lmplz.
If do it manually I probably need a ngram frequency list, or can I
somehow infer directly from the generated ARPA file what to remove?
2) Another problem: I have generated a 73GB plain-text arpa file with
lmplz, when I run build-binary, I get the following error message right
away:
./kenlm/bin/build_binary train.lm.no-tag.de.arpa
train.lm.no-tag.de.kenlm
util/mmap.cc:115 in void* util::MapOrThrow(std::size_t, bool, int,
bool, int, uint64_t) threw ErrnoException because `(ret = mmap(__null,
size, protect, flags, fd, offset)) == ((void *) -1)'.
Cannot allocate memory mmap failed for size 36347339336 at offset 0
Byte: 97 File: train.lm.no-tag.de.arpa
ERROR
I have mmapped 36GB files successfully in the past (though not on this
machine), so this is strange. There is also a lot of free disk space
available, memory is rather limited, 8GB only.
Thanks,
Marcin
P.S.: Love the speed of lmplz!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20130821/9efc39a9/attachment.htm
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 82, Issue 28
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 82, Issue 28"
Post a Comment