Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Tuning with no language model (Hieu Hoang)
2. Re: Tuning with no language model (Read, James C)
3. Re: Tuning with no language model (Read, James C)
----------------------------------------------------------------------
Message: 1
Date: Wed, 13 Jan 2016 13:39:44 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Tuning with no language model
To: "Read, James C" <jcread@essex.ac.uk>, Moses Support
<moses-support@mit.edu>
Message-ID: <569653A0.9010303@gmail.com>
Content-Type: text/plain; charset="windows-1252"
Can you please send me the moses.ini file you're using and the exact
command you're executing.
You should be able to tune without a LM. I've done it myself a couple of
years ago. There's no distinction between a LM and any other feature
function
On 13/01/16 13:22, Read, James C wrote:
>
> Returning to an old discussion we once had on this list.
>
>
> I've tried to tune some systems with no reference in the configuration
> file to a language model so that I can do Moses justice when comparing
> it to a number of lines of work I have done that use only translation
> models and various schemes of phrase table filtering.
>
>
> It seems that the tuning script will not run when supplied with a
> config file minus a reference to a language model. I could run a
> baseline with the weights learned from tuning with a config file but
> again this would be almost as meaningless as comparing my work against
> a Moses system with default weights as the weights would not be
> optimised to a system with no language model. I'm trying to come up
> with some kind of work around so that I can make a TM only Moses based
> baseline with sensible weights in the config file that are optimised
> to a TM only situation (no LM). I'm running short of ideas about how
> best to approach this.
>
>
> I was wondering if the following approach might work. If I was to make
> some kind of minimal dummy LM file with only one entry for some work
> which doesn't exist would that be enough to fool the MERT script to
> run so I could get some sensible weights optimised to a no LM scenario?
>
>
> I'm open to any other suggestions of how I can obtain such weights if
> anybody knows a clever way of doing this.
>
>
> Just to be clear this support request is not an invitation to discuss
> whether or not one would want to tune a system with no language model
> or not.
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
--
Hieu Hoang
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160113/f4f4a0cb/attachment-0001.html
------------------------------
Message: 2
Date: Wed, 13 Jan 2016 13:53:03 +0000
From: "Read, James C" <jcread@essex.ac.uk>
Subject: Re: [Moses-support] Tuning with no language model
To: Hieu Hoang <hieuhoang@gmail.com>, Moses Support
<moses-support@mit.edu>
Message-ID:
<HE1PR06MB1481B7CEAC75087014B59F3E85CB0@HE1PR06MB1481.eurprd06.prod.outlook.com>
Content-Type: text/plain; charset="iso-8859-1"
The following command works fine for when a language model is specified. Deleting the line KENLM lazyken=0 name=LM0 factor=0 path=$PWD/blm.en order=3
causes the script to fail with error:
ERROR: Failed to run '/pathtomoses/mosesdecoder/bin/moses -config /pathtodata/binarised/moses.ini -show-weights'. at /pathtomoses/mosesdecoder/scripts/training/mert-moses.pl line 1744.
#########################
### MOSES CONFIG FILE ###
#########################
# input factors
[input-factors]
0
# mapping steps
[mapping]
0 T 0
[distortion-limit]
6
# feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryCompact name=TranslationModel0 num-features=4 path=/pathtodata/binarised/phrase-table input-factor=0 output-factor=0
LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/pathtodata/binarised/reordering-table
Distortion
KENLM lazyken=0 name=LM0 factor=0 path=/pathtodata/blm.en order=3
# dense weights for feature functions
[weight]
# The default weights are NOT optimized for translation quality. You MUST tune the weights.
# Documentation for tuning is here: http://www.statmt.org/moses/?n=FactoredTraining.Tuning
UnknownWordPenalty0= 1
WordPenalty0= -1
PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2
LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
Distortion0= 0.3
LM0= 0.5
/pathtomoses/mosesdecoder/scripts/training/mert-moses.pl -no-filter-phrase-table true.fr true.en /pathtomoses/mosesdecoder/bin/moses binarised/moses.ini --mertdir /pathtomoses/mosesdecoder/bin/
<http://www.hoang.co.uk/hieu>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160113/953fbe21/attachment-0001.html
------------------------------
Message: 3
Date: Wed, 13 Jan 2016 13:54:02 +0000
From: "Read, James C" <jcread@essex.ac.uk>
Subject: Re: [Moses-support] Tuning with no language model
To: Hieu Hoang <hieuhoang@gmail.com>, Moses Support
<moses-support@mit.edu>
Message-ID:
<HE1PR06MB1481332238BEE4B63E9BC93585CB0@HE1PR06MB1481.eurprd06.prod.outlook.com>
Content-Type: text/plain; charset="iso-8859-1"
The following command works fine for when a language model is specified. Deleting the line KENLM lazyken=0 name=LM0 factor=0 path=/pathtodata/blm.en order=3
causes the script to fail with error:
ERROR: Failed to run '/pathtomoses/mosesdecoder/bin/moses -config /pathtodata/binarised/moses.ini -show-weights'. at /pathtomoses/mosesdecoder/scripts/training/mert-moses.pl line 1744.
#########################
### MOSES CONFIG FILE ###
#########################
# input factors
[input-factors]
0
# mapping steps
[mapping]
0 T 0
[distortion-limit]
6
# feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryCompact name=TranslationModel0 num-features=4 path=/pathtodata/binarised/phrase-table input-factor=0 output-factor=0
LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/pathtodata/binarised/reordering-table
Distortion
KENLM lazyken=0 name=LM0 factor=0 path=/pathtodata/blm.en order=3
# dense weights for feature functions
[weight]
# The default weights are NOT optimized for translation quality. You MUST tune the weights.
# Documentation for tuning is here: http://www.statmt.org/moses/?n=FactoredTraining.Tuning
UnknownWordPenalty0= 1
WordPenalty0= -1
PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2
LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
Distortion0= 0.3
LM0= 0.5
/pathtomoses/mosesdecoder/scripts/training/mert-moses.pl -no-filter-phrase-table true.fr true.en /pathtomoses/mosesdecoder/bin/moses binarised/moses.ini --mertdir /pathtomoses/mosesdecoder/bin/
<http://www.hoang.co.uk/hieu>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160113/cb90c2a8/attachment.html
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 111, Issue 28
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 111, Issue 28"
Post a Comment