Moses-support Digest, Vol 97, Issue 22

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: using sparse features (Philipp Koehn)
2. Re: Load LD_LIBRARY_PATH in ems with -cluster (sge)
(Raymond W. M. Ng)


----------------------------------------------------------------------

Message: 1
Date: Thu, 13 Nov 2014 11:26:56 -0500
From: Philipp Koehn <pkoehn@inf.ed.ac.uk>
Subject: Re: [Moses-support] using sparse features
To: Prashant Mathur <prashant@fbk.eu>
Cc: moses-support@mit.edu, Hieu Hoang <hieu.hoang@ed.ac.uk>, Barry
Haddow <bhaddow@staffmail.ed.ac.uk>
Message-ID:
<CAAFADDAjg9nm+FHTAY839M2ZrEVYGkodA4Z8fpor8GeuXcFBRg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

Typically you want to learn these feature weights when tuning. The current
setup supports and produces a sparse feature file.

-phi
On Nov 13, 2014 11:18 AM, "Prashant Mathur" <prashant@fbk.eu> wrote:

> what if I don't know the feature names before hand?
> In that case, can I set the weights directly during decoding?
>
> On Thu, Nov 13, 2014 at 4:59 PM, Barry Haddow <bhaddow@staffmail.ed.ac.uk>
> wrote:
>
>> Hi Prashant
>>
>> You add something like this to your moses.ini:
>>
>> [weight-file]
>> /path/to/sparse/weights/file
>>
>> The sparse weights file has the form:
>>
>> name1 weight1
>> name2 weight2
>> name3 weight3
>> .
>> .
>> .
>>
>> At least that's how it works in Moses v2.
>>
>> cheers - Barry
>>
>> On 13/11/14 15:42, Prashant Mathur wrote:
>>
>>> Thanks a lot Barry for your answers.
>>>
>>> I have another question.
>>> When I print these sparse features at the end of decoding, all sparse
>>> features are assigned a weight of 0 because all of them were initialized
>>> during decoding.
>>> How can I set these weights for sparse features before they are
>>> evaluated?
>>>
>>>
>>> Thanks Hieu for the link..
>>> I am going to update the code as soon as I can.. but it will take some
>>> time.. will get back to you when I do that.
>>>
>>> --Prashant
>>>
>>>
>>> On Thu, Nov 13, 2014 at 2:34 PM, Hieu Hoang <Hieu.Hoang@ed.ac.uk
>>> <mailto:Hieu.Hoang@ed.ac.uk>> wrote:
>>>
>>> re-iterating what Barry said, you should use the github moses if
>>> you want to create your own feature functions, especially with
>>> sparse features. The reasons:
>>> 1. Adding new feature functions is a pain in v 0.91. It's
>>> trivial now. You can watch my talk to find out why
>>> http://lectures.ms.mff.cuni.cz/video/recordshow/index/44/184
>>> 2. It's confusing exactly when the feature functions are
>>> computed. It's clear now (hopefully!)
>>> 3. I think you had to set special flags somewhere to use sparse
>>> features. Now, all feature functions can use sparse features as
>>> well as dense features
>>> 4. I don't remember the 0.91 code very well. So I can't help you
>>> if you get stuck
>>>
>>>
>>> On 13 November 2014 11:06, Barry Haddow
>>> <bhaddow@staffmail.ed.ac.uk <mailto:bhaddow@staffmail.ed.ac.uk>>
>>>
>>> wrote:
>>>
>>> Hi Prashant
>>>
>>> I tried to answer your questions inline:
>>>
>>>
>>> On 12/11/14 20:27, Prashant Mathur wrote:
>>> > Hi All,
>>> >
>>> > I have a question about implementing sparse feature function.
>>> > I went through the details on its implementation, still
>>> somethings are
>>> > not clear.
>>> > FYI, I am using an old version of moses which dates back to
>>> Release
>>> > 0.91 I guess. So, I am sorry if my questions don't relate to
>>> the
>>> > latest implementation.
>>>
>>> This is a bad idea. The FF interface has changed a lot since
>>> 0.91.
>>>
>>> >
>>> > 1. I was looking at the TargetNgramFeature where
>>> MakePrefixNgrams adds
>>> > features in Evaluate function. From the code it seems
>>> MakePrefixNgrams
>>> > is adding sparse features on the fly. Is it correct?
>>>
>>> Yes, you can add sparse features on the fly. That's really
>>> what makes
>>> them sparse features.
>>>
>>> >
>>> > what is the weight assigned to this newly added feature? 1 or
>>> 0?
>>>
>>> The weight comes from the weights file that you provide at
>>> start-up. If
>>> the feature is not in the weights file, then it gets a weight
>>> of 0.
>>>
>>> >
>>> > 2. What is the difference between these two functions?
>>> >
>>> > /void PlusEquals(const ScoreProducer*sp, const std::string&
>>> name,
>>> > float score)/
>>> > /
>>> > /
>>> > /void SparsePlusEquals(const std::string& full_name, float
>>> score)
>>> > /
>>>
>>> In the first, a string from the ScoreProducer is prepended to
>>> the name,
>>> whilst in the second the string full_name is used as the name.
>>> I think
>>> we should really use the first form to keep features in their own
>>> namespace, but the second form has pervaded Moses.
>>>
>>> >
>>> > It seems like both of them are used for updating sparse feature
>>> > values.. correct?
>>> > Or, do the first one points to sparse features of a
>>> particular FF and
>>> > second one to generic sparse features?
>>> >
>>> > 3. How is the structure like if I use one
>>> StatelessFeatureFunction
>>> > with unlimited scores? Is it different from having unlimited
>>> sparse
>>> > features?
>>> >
>>> > I assume if there is one FF then there is one weight
>>> assigned to it
>>> > but in the case of sparse features I have one weight for
>>> each feature.
>>> FFs can be dense or sparse. What that really means is that the
>>> number of
>>> feature values for a dense FF is known in advance (and so space
>>> is
>>> allocated in the feature value array) but for sparse FFs the
>>> number of
>>> feature values are not known in advance. So even dense FFs can
>>> have
>>> several weights associated with them - e.g. the phrase table
>>> features.
>>> In more recent versions of Moses a given FF can have both
>>> dense and
>>> sparse values.
>>>
>>> >
>>> > 4. In general when should I compute the sparse features?
>>>
>>> In general, computing them as soon as you can will probably
>>> make your
>>> code more efficient. When you are able to compute your sparse
>>> feature
>>> depends on the feature itself. For example, if the feature
>>> depends on
>>> only on the phrase pair then it could be computed and stored
>>> in the
>>> phrase table. This makes the phrase table bigger (which could
>>> slow you
>>> down) but saves on computation at decoding. On the other hand,
>>> a sparse
>>> reordering feature has to be mainly computed during decoding,
>>> since we
>>> do not know the ordering of segments until decoding. When I
>>> implemented
>>> sparse reordering features though, I precomputed the feature
>>> names since
>>> you don't want to do string concatenation during decoding.
>>>
>>>
>>> cheers - Barry
>>>
>>> >
>>> > Thanks for the patience,
>>> > --Prashant
>>> >
>>> > PS: I am still trying to figure out stuff, so questions
>>> might seem stupid.
>>> >
>>> >
>>> > _______________________________________________
>>> > Moses-support mailing list
>>> > Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> > http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>> --
>>> The University of Edinburgh is a charitable body, registered in
>>> Scotland, with registration number SC005336.
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>
>>>
>>> -- Hieu Hoang
>>> Research Associate
>>> University of Edinburgh
>>> http://www.hoang.co.uk/hieu
>>>
>>>
>>>
>>
>> --
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>>
>>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141113/bf6df8fb/attachment-0001.htm

------------------------------

Message: 2
Date: Thu, 13 Nov 2014 16:59:49 +0000
From: "Raymond W. M. Ng" <wm.ng@sheffield.ac.uk>
Subject: Re: [Moses-support] Load LD_LIBRARY_PATH in ems with -cluster
(sge)
To: undisclosed-recipients:;
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CANkx1N78fHwpYcAss098CxmMaf01ZEranfvmcJZNpjTadHpGAg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Hieu,

I just recompiled moses to link boost statically.
When I did this last time months ago, I remember I failed to link boost
statically. I didn't trace the error and just had a dynamic build.

A minor trick may be necessary for others to link boost statically when
compiling moses....

In my boost lib folder I only have
libbost_thread-mt.a
but *NOT* libboost_thread.a

I need to link the two
$ ln -s libboost_thread-mt.a libboost_thread.a
before moving on compiling moses.

Otherwise moses build will complain
/usr/bin/ld: cannot find -lboost_thread


best
raymond


This may serve a note to others


On 13 November 2014 14:44, Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:

> you don't need to set LD_LIBRARY_PATH if the boost lib statically linked.
> This is how I always use it, I never set this variable.
>
> You need to make sure there's no .so file in the folder
> /export/sw/std/boost/v1.56.0/x86_64/lib
> /export/sw/std/boost/v1.56.0/x86_64/lib64
>
>
> On 13 November 2014 14:39, Raymond W. M. Ng <wm.ng@sheffield.ac.uk> wrote:
>
>> Thanks Hieu,
>>
>> I did link boost (and sirlm, irstlm) when i built moses.
>>
>> ./bjam --with-boost=/export/sw/std/boost/v1.56.0/x86_64
>> --with-srilm=/export/sw/spl/srilm/v1.6.0/srilm
>> --with-irstlm=/export/sw/spl/irstlm/v5.80.03/x86_64 -j8
>>
>> But I think at run time I still need to load the path.
>> setenv LD_LIBRARY_PATH /export/sw/std/boost/v1.56.0/x86_64/lib
>>
>> It won't cause an issue if everything is run locally, because I could
>> source the environment just once. Maybe this problem is, again, specific to
>> computing environment
>>
>> cheers
>> raymond
>>
>>
>> On 13 November 2014 13:36, Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:
>>
>>> personally, I would try to ensure that moses was statically linked to
>>> the boost libraries so it won't go looking for the libboost*.so at runtime.
>>>
>>> I do this by building my own boost library and linking to it, rather
>>> than the system boost. The instructions are here
>>> http://www.statmt.org/moses/?n=Development.GetStarted
>>>
>>> On 13 November 2014 13:23, Raymond W. M. Ng <wm.ng@sheffield.ac.uk>
>>> wrote:
>>>
>>>> Dear moses,
>>>>
>>>> I have the following error in training my translation model with ems on
>>>> SGE.
>>>>
>>>> /home/users/moses/v-2014-08-15/mosesdecoder/bin/build_binary: error
>>>> while loading shared libraries: libboost_thread-mt.so.1.56.0: cannot open
>>>> shared object file: No such file or directory
>>>>
>>>> In the offline mode, I can run this without problem by first sourcing
>>>> an environment file which contains something like
>>>> setenv LD_LIBRARY_PATH /export/sw/boost/v1.56.0/x86_64/lib
>>>>
>>>> So, I think the LD_LIBRARY_PATH environment was not loaded in the sge
>>>> parallel scripts,
>>>>
>>>> grep 'ENV' /home/moses/mosesdecoder/scripts/generic/qsub-wrapper.pl
>>>>
>>>> returns nothing.
>>>>
>>>> Is there something to modify in the code or did I miss any setting so
>>>> the environment variable is not loaded?
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>>
>>>
>>>
>>> --
>>> Hieu Hoang
>>> Research Associate
>>> University of Edinburgh
>>> http://www.hoang.co.uk/hieu
>>>
>>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141113/18de51cb/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 97, Issue 22
*********************************************

Tweet