Moses-support Digest, Vol 105, Issue 32

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Getting started - Baseline basic question (Hieu Hoang)
2. Re: Moses build failed (Manuela Sanguinetti)
3. Re: Sparse phrase table, is still supported? (jian zhang)


----------------------------------------------------------------------

Message: 1
Date: Thu, 16 Jul 2015 09:40:58 +0400
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Getting started - Baseline basic question
To: Vincent Nguyen <vnguyen@neuf.fr>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbhMwK2Fp-z=OgB0mUt9t34ErCniUUTPjoeRQt9fqnAJjg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

see the example under 'Comparison With Other Toolkits'

if you have a corpus file called 'text' and you want to create a 5-gram
language model, the command is
lmplz -o 5 --interpolate_unigrams 0 <text >text.arpa

Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

On 15 July 2015 at 23:50, Vincent Nguyen <vnguyen@neuf.fr> wrote:

> you need to create a language model. You can do it with kenlm using the
> program lmplz. The program is described here:
>
>> https://kheafield.com/code/kenlm/estimation/
>> We haven't updated the tutorial with more info about lmplz yet, but we
>> should do at some point
>>
>
>
> I have read this page, but honestly I have no clue about the sequence of
> what to do to create the LM and the training
> once I have prepared the Corpus.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150716/426df661/attachment-0001.htm

------------------------------

Message: 2
Date: Thu, 16 Jul 2015 09:58:23 +0200
From: Manuela Sanguinetti <msanguin@di.unito.it>
Subject: Re: [Moses-support] Moses build failed
To: Hieu Hoang <hieuhoang@gmail.com>, Rico Sennrich
<rico.sennrich@gmx.ch>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <55A7641F.5000508@di.unito.it>
Content-Type: text/plain; charset="utf-8"

Thank you,

I'll try with this one.

Just out of curiosity ... how did you install the required dependecies
on cygwin? Did you use the command line?


best wishes,
manuela

Il 16/07/2015 07:24, Hieu Hoang ha scritto:
> i just compiled on Windows/cygwin ok. Unfortunately, you do need to be
> quite experienced with computer science and linux to get very far with
> moses.
>
> The company, Precision Translation Tools, have adapted Moses
> specifically for Windows (without cygwin).
> http://www.precisiontranslationtools.com/smt-for-windows-and-more/
> Perhaps thats a little more user-friendly
>
> Hieu Hoang
> Researcher
> New York University, Abu Dhabi
> http://www.hoang.co.uk/hieu
>
> On 15 July 2015 at 20:30, Rico Sennrich <rico.sennrich@gmx.ch
> <mailto:rico.sennrich@gmx.ch>> wrote:
>
> Hi Manuela,
>
> I don't know enough about cygwin to know how the installation of
> the dependencies could have gone wrong.
>
> Can someone who uses cygwin help?
>
> best wishes,
> Rico
>
> On 15.07.2015 14:56, Manuela Sanguinetti wrote:
>> Hi Rico,
>>
>> first of all, thanks for your feedback.
>> I first installed all the dependencies listed on the Moses page
>> by running the Cygwin 'set_up-x86_64' executable file. However,
>> checking if everything was actually there (using the command
>> "which ..."), it appeared that for a large number of those
>> packages this wasn't the case. So I've just tried to install
>> them using the command "apt-cyg install ...", but I got the
>> message "Package ... is already installed, skipping"!
>> I guess that all these "missing" dependecies should then be
>> properly linked, by setting some path or environment variable ...
>> but unfortunately this goes far beyond my competences in computer
>> science :( so now I'm totally clueless as to what to do.
>>
>> Thank you anyway,
>> Manuela
>>
>>
>>
>> Il 15/07/2015 15:26, Rico Sennrich ha scritto:
>>> Hello Manuela,
>>>
>>> have you installed all the dependencies listed here for Cygwin
>>> installation? http://www.statmt.org/moses/?n=Development.GetStarted
>>>
>>> it seems like you're missing a system-wide installation of
>>> boost. mosesdecoder/bjam currently isn't set-up to use the
>>> boost-jam of a local boost installation (even if you pass a
>>> "--with-boost" argument).
>>>
>>> best wishes,
>>> Rico
>>>
>>>
>>> On 15.07.2015 12:19, Manuela Sanguinetti wrote:
>>>> Hi,
>>>>
>>>> I'm experiencing some problems in compiling Moses on Windows
>>>> using Cygwin. I manually installed Boost (v. 1.58), but when I
>>>> run the command
>>>>
>>>> ~/mosesdecoder
>>>> $ ./bjam --with-boost=~/boost_1_58_0/ -j4
>>>>
>>>> the build fails.
>>>> I'm attaching the 'build.log.gz' file, as suggested in the
>>>> error message, hoping that someone could help me figuring out
>>>> what the problem is and how I can fix it.
>>>>
>>>>
>>>> Thank you in advance for any hint you may provide,
>>>> MS
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150716/48ec2e32/attachment-0001.htm

------------------------------

Message: 3
Date: Thu, 16 Jul 2015 10:03:01 +0100
From: jian zhang <zhangj@computing.dcu.ie>
Subject: Re: [Moses-support] Sparse phrase table, is still supported?
To: Matthias Huck <mhuck@inf.ed.ac.uk>
Cc: moses-support@mit.edu
Message-ID:
<CALA=z0Dc8vsizT5k7GEhg6oaJFBPUuE2J9hza6UEpWZNOMgzUA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Matthias,

Thanks for the information.

I tested on moses 3.0, adding phrase table sparse feature is seems working.

However, I did not add any flag into ini, like suggested "If a phrase table
contains sparse features, then this needs to be flagged in the
configuration file by adding the word sparse after the phrase table file
name.". Did i miss anything?

Regards,

Jian




On Thu, Jul 16, 2015 at 3:23 AM, Matthias Huck <mhuck@inf.ed.ac.uk> wrote:

> Hi Jian,
>
> That depends on the nature of the features you're planning to
> implement.
>
> In order to produce sparse features, you need to write a feature
> function anyway.
>
> But if it's only a handful of scores and they can be calculated during
> extraction time, then go for dense features and add the scores directly
> to the phrase table.
>
> If the scores cannot be precalculated, for instance because you need
> non-local information that is only available during decoding, then a
> feature function implementation becomes necessary.
>
> When you write a feature function that calculates scores during decoding
> time, it can produce dense scores, sparse scores, or both types. That's
> up to you.
>
> If it's plenty of scores which are fired rarely, then sparse is the
> right choice. And you certainly need a sparse feature function
> implementation in case you are not aware in advance of the overall
> amount of feature scores it can produce.
>
> If you need information from phrase extraction in order to calculate
> scores during decoding time, then we have something denoted as "phrase
> properties". Phrase properties give you a means of storing arbitrary
> additional information in the phrase table. You have to extend the
> extraction pipeline to retrieve and store the phrase properties you
> require. The decoder can later read this information from the phrase
> table, and your feature function can utilize it in some way.
>
> A large amount of sparse feature scores can somewhat slow down decoding
> and tuning. Also, you have to use MIRA or PRO for tuning, not MERT.
>
> Cheers,
> Matthias
>
>
> On Thu, 2015-07-16 at 02:18 +0100, jian zhang wrote:
> > Hi Matthias,
> >
> >
> > Not for domain feature.
> >
> >
> > I want to implement some sparse features, so there are two options:
> > 1, add to phrase table, if it is supported
> > 2, implement sparse feature functions,
> >
> >
> > I'd like to know are there any difference between these two options,
> > for example, tuning, compute sentence translation scores ...
> >
> >
> > Regards,
> >
> >
> >
> > Jian
> >
> >
> >
> > On Thu, Jul 16, 2015 at 2:06 AM, Matthias Huck <mhuck@inf.ed.ac.uk>
> > wrote:
> > Hi,
> >
> > Are you planning to use binary domain indicator features? I'm
> > not sure
> > whether a sparse feature function for this is currently
> > implemented. If
> > you're working with a small set of domains, you can employ
> > dense
> > indicators instead (domain-features = "indicator" in EMS).
> > You'll have
> > to re-extract the phrase table, though. Or process it with a
> > script to
> > add dense indicator values to the scores field.
> >
> > I believe that there might also be some bug in the extraction
> > pipeline
> > when both domain-features = "sparse indicator" and
> > score-settings =
> > "--GoodTuring" are active in EMS. At least it caused me
> > trouble a couple
> > of weeks ago. However, I must admit that I didn't investigate
> > it further
> > at that point.
> >
> > Anyway, the bottom line is that I recommend re-extracting with
> > dense
> > indicators.
> >
> > But let me know what you find regarding a sparse
> > implementation.
> >
> > Cheers,
> > Matthias
> >
> >
> > On Thu, 2015-07-16 at 00:48 +0100, jian zhang wrote:
> > > Hi,
> > >
> > >
> > > Is the sparse features at phrase table, like
> > >
> > >
> > >
> > > das Haus ||| the house ||| 0.8 0.5 0.8 0.5 2.718 ||| 0-0 1-1
> > ||| 5000
> > > 5000 2500 ||| dom_europarl 1
> > >
> > >
> > >
> > > still supported? If yes, what should I set to the ini file
> > based on
> > > the example above?
> > >
> > >
> > > Thank,
> > >
> > >
> > > Jian
> > >
> > >
> > > --
> > > Jian Zhang
> > > Centre for Next Generation Localisation (CNGL)
> > > Dublin City University
> >
> > > _______________________________________________
> > > Moses-support mailing list
> > > Moses-support@mit.edu
> > > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> >
> >
> > --
> > The University of Edinburgh is a charitable body, registered
> > in
> > Scotland, with registration number SC005336.
> >
> >
> >
> >
> >
> > --
> > Jian Zhang
> > Centre for Next Generation Localisation (CNGL)
> > Dublin City University
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>


--
Jian Zhang
Centre for Next Generation Localisation (CNGL)
<http://www.cngl.ie/index.html>
Dublin City University <http://www.dcu.ie/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150716/87f4cbe6/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 105, Issue 32
**********************************************

Tweet