Moses-support Digest, Vol 97, Issue 17

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. BLEU score Help (Maria Marpaung)
2. Re: using sparse features (Barry Haddow)
3. Load LD_LIBRARY_PATH in ems with -cluster (sge) (Raymond W. M. Ng)
4. Re: using sparse features (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Thu, 13 Nov 2014 10:46:43 +0000 (UTC)
From: Maria Marpaung <maria_marpaung@yahoo.co.id>
Subject: [Moses-support] BLEU score Help
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<1042738263.257507.1415875603083.JavaMail.yahoo@jws10922.mail.sg3.yahoo.com>

Content-Type: text/plain; charset="utf-8"

Hi please help me,
I have been can run Moses and get a score of BLUE is:? BLEU= 74.25, 86.2/77.1/70.7/64.8 (BP=1.000, ratio= 1.000, hyp_len= 59134, ref_len= 59144)

but I dont understand these values. Can you explain the meaning of each of these values?Therefore, I have to explain my current thesis presentation
Please, I need you help

Thank you,
Maria
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141113/f528b796/attachment-0001.htm

------------------------------

Message: 2
Date: Thu, 13 Nov 2014 11:06:55 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] using sparse features
To: Prashant Mathur <prashant@fbk.eu>, "moses-support@mit.edu"
<moses-support@mit.edu>
Message-ID: <546490CF.8090104@staffmail.ed.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Hi Prashant

I tried to answer your questions inline:


On 12/11/14 20:27, Prashant Mathur wrote:
> Hi All,
>
> I have a question about implementing sparse feature function.
> I went through the details on its implementation, still somethings are
> not clear.
> FYI, I am using an old version of moses which dates back to Release
> 0.91 I guess. So, I am sorry if my questions don't relate to the
> latest implementation.

This is a bad idea. The FF interface has changed a lot since 0.91.

>
> 1. I was looking at the TargetNgramFeature where MakePrefixNgrams adds
> features in Evaluate function. From the code it seems MakePrefixNgrams
> is adding sparse features on the fly. Is it correct?

Yes, you can add sparse features on the fly. That's really what makes
them sparse features.

>
> what is the weight assigned to this newly added feature? 1 or 0?

The weight comes from the weights file that you provide at start-up. If
the feature is not in the weights file, then it gets a weight of 0.

>
> 2. What is the difference between these two functions?
>
> /void PlusEquals(const ScoreProducer*sp, const std::string& name,
> float score)/
> /
> /
> /void SparsePlusEquals(const std::string& full_name, float score)
> /

In the first, a string from the ScoreProducer is prepended to the name,
whilst in the second the string full_name is used as the name. I think
we should really use the first form to keep features in their own
namespace, but the second form has pervaded Moses.

>
> It seems like both of them are used for updating sparse feature
> values.. correct?
> Or, do the first one points to sparse features of a particular FF and
> second one to generic sparse features?
>
> 3. How is the structure like if I use one StatelessFeatureFunction
> with unlimited scores? Is it different from having unlimited sparse
> features?
>
> I assume if there is one FF then there is one weight assigned to it
> but in the case of sparse features I have one weight for each feature.
FFs can be dense or sparse. What that really means is that the number of
feature values for a dense FF is known in advance (and so space is
allocated in the feature value array) but for sparse FFs the number of
feature values are not known in advance. So even dense FFs can have
several weights associated with them - e.g. the phrase table features.
In more recent versions of Moses a given FF can have both dense and
sparse values.

>
> 4. In general when should I compute the sparse features?

In general, computing them as soon as you can will probably make your
code more efficient. When you are able to compute your sparse feature
depends on the feature itself. For example, if the feature depends on
only on the phrase pair then it could be computed and stored in the
phrase table. This makes the phrase table bigger (which could slow you
down) but saves on computation at decoding. On the other hand, a sparse
reordering feature has to be mainly computed during decoding, since we
do not know the ordering of segments until decoding. When I implemented
sparse reordering features though, I precomputed the feature names since
you don't want to do string concatenation during decoding.


cheers - Barry

>
> Thanks for the patience,
> --Prashant
>
> PS: I am still trying to figure out stuff, so questions might seem stupid.
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support


--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



------------------------------

Message: 3
Date: Thu, 13 Nov 2014 13:23:50 +0000
From: "Raymond W. M. Ng" <wm.ng@sheffield.ac.uk>
Subject: [Moses-support] Load LD_LIBRARY_PATH in ems with -cluster
(sge)
To: moses-support <moses-support@mit.edu>
Message-ID:
<CANkx1N50th0To7O7=kxR3T4HD8RA9Vn7yDJz4OYXcQC4pPxihg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Dear moses,

I have the following error in training my translation model with ems on SGE.

/home/users/moses/v-2014-08-15/mosesdecoder/bin/build_binary: error while
loading shared libraries: libboost_thread-mt.so.1.56.0: cannot open shared
object file: No such file or directory

In the offline mode, I can run this without problem by first sourcing an
environment file which contains something like
setenv LD_LIBRARY_PATH /export/sw/boost/v1.56.0/x86_64/lib

So, I think the LD_LIBRARY_PATH environment was not loaded in the sge
parallel scripts,

grep 'ENV' /home/moses/mosesdecoder/scripts/generic/qsub-wrapper.pl

returns nothing.

Is there something to modify in the code or did I miss any setting so the
environment variable is not loaded?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141113/c44e319d/attachment-0001.htm

------------------------------

Message: 4
Date: Thu, 13 Nov 2014 13:34:01 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] using sparse features
To: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAEKMkbi4FjnbR1GhrA1i=6K7MWgwmoUk1d9kJJmJGkUbc5C6xQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

re-iterating what Barry said, you should use the github moses if you want
to create your own feature functions, especially with sparse features. The
reasons:
1. Adding new feature functions is a pain in v 0.91. It's trivial now.
You can watch my talk to find out why
http://lectures.ms.mff.cuni.cz/video/recordshow/index/44/184
2. It's confusing exactly when the feature functions are computed. It's
clear now (hopefully!)
3. I think you had to set special flags somewhere to use sparse features.
Now, all feature functions can use sparse features as well as dense features
4. I don't remember the 0.91 code very well. So I can't help you if you
get stuck


On 13 November 2014 11:06, Barry Haddow <bhaddow@staffmail.ed.ac.uk> wrote:

> Hi Prashant
>
> I tried to answer your questions inline:
>
>
> On 12/11/14 20:27, Prashant Mathur wrote:
> > Hi All,
> >
> > I have a question about implementing sparse feature function.
> > I went through the details on its implementation, still somethings are
> > not clear.
> > FYI, I am using an old version of moses which dates back to Release
> > 0.91 I guess. So, I am sorry if my questions don't relate to the
> > latest implementation.
>
> This is a bad idea. The FF interface has changed a lot since 0.91.
>
> >
> > 1. I was looking at the TargetNgramFeature where MakePrefixNgrams adds
> > features in Evaluate function. From the code it seems MakePrefixNgrams
> > is adding sparse features on the fly. Is it correct?
>
> Yes, you can add sparse features on the fly. That's really what makes
> them sparse features.
>
> >
> > what is the weight assigned to this newly added feature? 1 or 0?
>
> The weight comes from the weights file that you provide at start-up. If
> the feature is not in the weights file, then it gets a weight of 0.
>
> >
> > 2. What is the difference between these two functions?
> >
> > /void PlusEquals(const ScoreProducer*sp, const std::string& name,
> > float score)/
> > /
> > /
> > /void SparsePlusEquals(const std::string& full_name, float score)
> > /
>
> In the first, a string from the ScoreProducer is prepended to the name,
> whilst in the second the string full_name is used as the name. I think
> we should really use the first form to keep features in their own
> namespace, but the second form has pervaded Moses.
>
> >
> > It seems like both of them are used for updating sparse feature
> > values.. correct?
> > Or, do the first one points to sparse features of a particular FF and
> > second one to generic sparse features?
> >
> > 3. How is the structure like if I use one StatelessFeatureFunction
> > with unlimited scores? Is it different from having unlimited sparse
> > features?
> >
> > I assume if there is one FF then there is one weight assigned to it
> > but in the case of sparse features I have one weight for each feature.
> FFs can be dense or sparse. What that really means is that the number of
> feature values for a dense FF is known in advance (and so space is
> allocated in the feature value array) but for sparse FFs the number of
> feature values are not known in advance. So even dense FFs can have
> several weights associated with them - e.g. the phrase table features.
> In more recent versions of Moses a given FF can have both dense and
> sparse values.
>
> >
> > 4. In general when should I compute the sparse features?
>
> In general, computing them as soon as you can will probably make your
> code more efficient. When you are able to compute your sparse feature
> depends on the feature itself. For example, if the feature depends on
> only on the phrase pair then it could be computed and stored in the
> phrase table. This makes the phrase table bigger (which could slow you
> down) but saves on computation at decoding. On the other hand, a sparse
> reordering feature has to be mainly computed during decoding, since we
> do not know the ordering of segments until decoding. When I implemented
> sparse reordering features though, I precomputed the feature names since
> you don't want to do string concatenation during decoding.
>
>
> cheers - Barry
>
> >
> > Thanks for the patience,
> > --Prashant
> >
> > PS: I am still trying to figure out stuff, so questions might seem
> stupid.
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141113/6b91fc8c/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 97, Issue 17
*********************************************

Tweet