Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: windows cygwin moses build (jfrm.maurel@gmail.com)
2. Re: windows cygwin moses build (jfrm.maurel@gmail.com)
3. Re: Implementation of pre-reordering for German and other
languages (Hieu Hoang)
4. Re: error running the software (Hieu Hoang)
5. how to integrate a long-span LM into moses (David Mrva)
----------------------------------------------------------------------
Message: 1
Date: Thu, 03 Apr 2014 12:25:41 +0200
From: "jfrm.maurel@gmail.com" <jfrm.maurel@gmail.com>
Subject: Re: [Moses-support] windows cygwin moses build
To: moses-support@mit.edu
Cc: Hieu Hoang <hieu.hoang@ed.ac.uk>
Message-ID: <533D3725.4070700@gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Le 30/03/2014 00:28, Hieu Hoang a ?crit :
Hi,
> When I compiled moses 2 months ago, it seems fine
I could obtain most of .exe files however I have still an error in
BackwardTest when building moses:
./b2 -j4 --prefix=$PWD --libdir=$PWD/lib64 --layout=tagged --without-mpi
link=static threading=single install >build.log
Could you please tell me if you can see what is wrong ?
Regards
--
Jean-Fran?ois MAUREL
PIMECA
http://www.pimeca.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: buildlog.tar.bz2
Type: application/octet-stream
Size: 4475 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140403/4d4c0f78/attachment-0001.obj
------------------------------
Message: 2
Date: Thu, 03 Apr 2014 12:33:01 +0200
From: "jfrm.maurel@gmail.com" <jfrm.maurel@gmail.com>
Subject: Re: [Moses-support] windows cygwin moses build
To: moses-support@mit.edu
Cc: Hieu Hoang <hieu.hoang@ed.ac.uk>
Message-ID: <533D38DD.5040906@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Le 30/03/2014 00:28, Hieu Hoang a ?crit :
Hi,
Contrary to my preceeding e-mail, the command line I used to build moses is:
./bjam -j4 --without-tcmalloc --with-boost=/usr/local/boost_1_55_0 -d2
--debug-configuration >build.log
Sorry.
Regards
--
Jean-Fran?ois MAUREL
PIMECA
http://www.pimeca.com
------------------------------
Message: 3
Date: Thu, 3 Apr 2014 11:50:22 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Implementation of pre-reordering for
German and other languages
To: Maxim Khalilov <maxkhalilov@gmail.com>
Cc: moses-support <moses-support@mit.edu>, mt-list@eamt.org
Message-ID:
<CAEKMkbgKyY4q182iTKXw_zc3SD-YzeVhAqok7ddk6zArJdPmRQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
I don't know of an implementation that's available for download and I seem
to remember that the parser they used was extremely difficult to use or
compile.
If you find out more, please let everyone know
On 25 March 2014 17:07, Maxim Khalilov <maxkhalilov@gmail.com> wrote:
> Dear Moses community,
>
> I am looking for a ready to use or easily customizable implementation of
> pre-reordering algorithms for Moses? In particular, I'm interested in
> language pairs with German as a source language and a variety of languages
> as target, so probably the best solution is syntax based.
>
> As a starting point, I would consider the algorithm described in (Collins
> et al., 2005), but I don't know if there is an implementation available and
> which parser it relies on.
>
> Thanks for your help beforehand,
> Maxim Khalilov
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140403/d04b95ee/attachment-0001.htm
------------------------------
Message: 4
Date: Thu, 3 Apr 2014 11:54:07 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] error running the software
To: arpit gupta <arpitg1991@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbgGJZB9HO7999cM+KCqkNtfc7-B2dJO0Ah-8qraayZxYQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
it seems the linker can't find the boost library program_options.
What is the exact bjam command you ran when you got this error? What OS are
you running? Are you trying to link to your own version of boost, and if
so, did the boost compile have any errors?
On 1 April 2014 04:46, arpit gupta <arpitg1991@gmail.com> wrote:
> I am attaching the log files of error. Can you please help ?
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140403/73ae6ed7/attachment-0001.htm
------------------------------
Message: 5
Date: Thu, 03 Apr 2014 11:55:28 +0100
From: David Mrva <davidm@cantabresearch.com>
Subject: [Moses-support] how to integrate a long-span LM into moses
To: moses-support@mit.edu
Message-ID: <533D3E20.8080203@cantabresearch.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hi moses developers,
I have integrated an LM with unlimited history into moses code by
implementing the StatefulFeatureFunction class following the example in
moses/FF/SkeletonStatefulFF.cpp. My LM code can also handle ngrams
through using kenlm. As kenlm is integrated into moses, I compared my
new code with the standard moses with cs-en models. Standard moses gave
17.12 BLEU points and my new code with the same ini file and the same LM
file resulted in BLEU 13.4. This lead me to read the Kenlm.cpp
implementation in much more detail and I found a number of interesting
aspects that I would like to understand to integrate a long-span model
correctly.
1/ In the Evaluate(const Hypothesis &hypo, const FFState *ps,
ScoreComponentCollection *out) const method moses/LM/Kenlm.cpp
calculates the score over only the first N - 1 words, where N is the
ngram order. Why not calculate the score for all words in the target
phrase from hypo.GetCurrTargetWordsRange().GetEndPos() to
hypo.GetCurrTargetWordsRange().GetStartPos()? Is this an optimisation or
is it required to make the translation work?
2/ At the time of the load of the translation phrase table, moses calls:
void LanguageModel::Evaluate(const Phrase &source, const TargetPhrase
&targetPhrase, ScoreComponentCollection &scoreBreakdown,
ScoreComponentCollection &estimatedFutureScore) const
where LanguageModel is the ancestor of the LanguageModelKen<Model>
template implemented in Kenlm.cpp. Moses calls this Evaluate() method
for each source-target translation phrase and assigns the accumulated
score of the first N-1 words to estimatedFutureScore and the accumulated
score over the rest of the phrase into scoreBreakdown. What is the
purpose of this split of the ngram scores into the first N-1 words and
the rest of the phrase? Is the scoreBreakdown value added with the score
calculated with the Evaluate() method from point 1/ during the search to
get the total phrase score?
3/ The method LanguageModelKen<Model>::CalcScore(const Phrase &phrase,
float &fullScore, float &ngramScore, size_t &oovCount) const used also
in the calculation in point 2/ above, distinguishes between
terminal/non-terminal words. What are these? Is the distinction relevant
to long-span LMs? Why is the terminal/non-terminal distinction not
necessary in the Evaluate() method described in point 1/ above?
After changing my implementation to incorporate the behaviour described
above I got an exact match of the output with moses's "native" Kenlm
implementation. However, the behaviour in point 1/ is not suitable for
long-span LMs and point 2/ does not make sense for non-ngram models. I
would expect my long-span LM to operate in such a way that an LM state
assigned to a hypothesis covers all target words from the start to the
end of the hypothesis. And when a hypothesis is being extended, its LM
state is extended by one target word at a time in a loop over the new
phrase from start to finish. Ngram LM implementation does not work in
this way and it seems to harm ngram performance. Can anyone shed some
light on the motivation behind the behaviour described above in points 1-3?
I used moses with its default, a.k.a. "normal", search algorithm (no
[search-algorithm] variable specified in my config). For completeness,
my config when using moses with its Kenlm class is pasted below.
Best regards,
David
# input factors
[input-factors]
0
# mapping steps
[mapping]
0 T 0
[distortion-limit]
6
# feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryMemory name=TranslationModel0 table-limit=20
num-features=4 path=model/phrase-table.1.gz input-factor=0 output-factor=0
LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=model/reordering-table.1.wbe-msd-bidirectional-fe.gz
Distortion
KENLM lazyken=1 name=LM0 factor=0 path=lm/europarl.binlm.1 order=5
# dense weights for feature functions
[weight]
UnknownWordPenalty0= 1
WordPenalty0= -1
PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2
LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
Distortion0= 0.3
LM0= 0.5
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 90, Issue 5
********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 90, Issue 5"
Post a Comment