Moses-support Digest, Vol 87, Issue 43

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Tuning hyperparamters with MERT (Hieu Hoang)
2. Re: Tuning hyperparamters with MERT (Barry Haddow)
3. Re: Tuning hyperparamters with MERT (Kevin Gimpel)


----------------------------------------------------------------------

Message: 1
Date: Fri, 17 Jan 2014 18:08:26 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Tuning hyperparamters with MERT
To: Prashant Mathur <prashant@fbk.eu>
Cc: Moses <moses-support@mit.edu>
Message-ID:
<CAEKMkbh2G57GghdciSdYr-k96D8ME2-t0J03iwchFPcAXqRRSQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

interesting. I don't think you can use MERT because it assumes that there
is a linear relationship between weights and model scores.

However, I think you can use PRO.

I added a feature function
HyperParameterAsWeight
that uses it's weights to change some hyper-parameters such as stack size
and beam width.

It's a hack, but I hope it's a start of what you want to do. Change it
however you want. I haven't tested it so it's up to you to test it

Fyi, the commit is here

https://github.com/moses-smt/mosesdecoder/commit/20e7d078a6ed23e6a293801955ba09e622f993df





On 17 January 2014 16:10, Prashant Mathur <prashant@fbk.eu> wrote:

> Hi All,
>
> I am tuning the standard models with an extended feature.
> As far as I understood, we can tune only the feature weights in mert.
> In my case I have some more hyper parameters that I want to tune
> together with the feature weights .. for example. feature learning rate.
> Once the nbest-list is created these parameters cannot have any effect
> on the list, but they are useful for generating a better nbest-list.
> Is there a way to tune all the parameters and hyper parameters together?
>
> Thanks,
> Prashant
>
> PS: Till now I was tuning these hyper parameters using Simplex.
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140117/104385e7/attachment-0001.htm

------------------------------

Message: 2
Date: Fri, 17 Jan 2014 21:31:29 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Tuning hyperparamters with MERT
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>, Prashant Mathur
<prashant@fbk.eu>
Cc: Moses <moses-support@mit.edu>
Message-ID: <52D9A131.9020509@staffmail.ed.ac.uk>
Content-Type: text/plain; charset="iso-8859-1"

Hi Prashant

PRO also assumes that the model score depends linearly on the weights,
as do the mira variants. There have been some attempts at "tuning" of
hyperparameters, but it's a tricky problem. For example, see:

http://books.nips.cc/papers/files/nips25/NIPS2012_0650.pdf (for parsing)
http://www.aclweb.org/anthology/W13-2229.pdf

cheers - Barry


On 17/01/14 18:08, Hieu Hoang wrote:
> interesting. I don't think you can use MERT because it assumes that
> there is a linear relationship between weights and model scores.
>
> However, I think you can use PRO.
>
> I added a feature function
> HyperParameterAsWeight
> that uses it's weights to change some hyper-parameters such as stack
> size and beam width.
>
> It's a hack, but I hope it's a start of what you want to do. Change it
> however you want. I haven't tested it so it's up to you to test it
>
> Fyi, the commit is here
> https://github.com/moses-smt/mosesdecoder/commit/20e7d078a6ed23e6a293801955ba09e622f993df
>
>
>
>
>
> On 17 January 2014 16:10, Prashant Mathur <prashant@fbk.eu
> <mailto:prashant@fbk.eu>> wrote:
>
> Hi All,
>
> I am tuning the standard models with an extended feature.
> As far as I understood, we can tune only the feature weights in mert.
> In my case I have some more hyper parameters that I want to tune
> together with the feature weights .. for example. feature learning
> rate.
> Once the nbest-list is created these parameters cannot have any effect
> on the list, but they are useful for generating a better nbest-list.
> Is there a way to tune all the parameters and hyper parameters
> together?
>
> Thanks,
> Prashant
>
> PS: Till now I was tuning these hyper parameters using Simplex.
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140117/a44a0319/attachment-0001.htm

------------------------------

Message: 3
Date: Fri, 17 Jan 2014 15:36:01 -0600
From: Kevin Gimpel <kgimpel@cs.cmu.edu>
Subject: Re: [Moses-support] Tuning hyperparamters with MERT
To: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Cc: Hieu Hoang <hieu.hoang@ed.ac.uk>, Moses <moses-support@mit.edu>
Message-ID:
<CACAqqMv6mL4iVVeKzrXAzT16PxTqNyG-hAnNEfu885ZY63H5Fg@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Also see http://aclweb.org/anthology//W/W12/W12-3159.pdf
(And thanks for those pointers, Barry.)
Kevin



On Fri, Jan 17, 2014 at 3:31 PM, Barry Haddow <bhaddow@staffmail.ed.ac.uk>wrote:

> Hi Prashant
>
> PRO also assumes that the model score depends linearly on the weights, as
> do the mira variants. There have been some attempts at "tuning" of
> hyperparameters, but it's a tricky problem. For example, see:
>
> http://books.nips.cc/papers/files/nips25/NIPS2012_0650.pdf (for parsing)
> http://www.aclweb.org/anthology/W13-2229.pdf
>
> cheers - Barry
>
>
>
> On 17/01/14 18:08, Hieu Hoang wrote:
>
> interesting. I don't think you can use MERT because it assumes that
> there is a linear relationship between weights and model scores.
>
> However, I think you can use PRO.
>
> I added a feature function
> HyperParameterAsWeight
> that uses it's weights to change some hyper-parameters such as stack size
> and beam width.
>
> It's a hack, but I hope it's a start of what you want to do. Change it
> however you want. I haven't tested it so it's up to you to test it
>
> Fyi, the commit is here
>
> https://github.com/moses-smt/mosesdecoder/commit/20e7d078a6ed23e6a293801955ba09e622f993df
>
>
>
>
>
> On 17 January 2014 16:10, Prashant Mathur <prashant@fbk.eu> wrote:
>
>> Hi All,
>>
>> I am tuning the standard models with an extended feature.
>> As far as I understood, we can tune only the feature weights in mert.
>> In my case I have some more hyper parameters that I want to tune
>> together with the feature weights .. for example. feature learning rate.
>> Once the nbest-list is created these parameters cannot have any effect
>> on the list, but they are useful for generating a better nbest-list.
>> Is there a way to tune all the parameters and hyper parameters together?
>>
>> Thanks,
>> Prashant
>>
>> PS: Till now I was tuning these hyper parameters using Simplex.
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140117/19de6878/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 87, Issue 43
*********************************************

Tweet