Moses-support Digest, Vol 100, Issue 9

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Using factor in the hierarchical model (Hieu Hoang)
2. Re: facored LMs in Moses (Hieu Hoang)
3. Re: Suggested change to train-model.perl (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Tue, 03 Feb 2015 11:12:40 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Using factor in the hierarchical model
To: ekkim214 <ekkim214@gmail.com>, moses-support@mit.edu
Message-ID: <54D0AD28.5010608@gmail.com>
Content-Type: text/plain; charset="windows-1252"

as ken mentioned, don't use the = sign in the name of your LM file. I
think it's fixed in the latest version of moses. Which version do you
have, from when ago?

moses_chart should handle factors. Again, if you version is from long
ago, I suggest your update

On 02/02/15 10:06, ekkim214 wrote:
>
> I want to use some factor (class) in the hierarchical model.
> I am using the ems script and using KenLM for surface and factor both.
> The building of LM is successful with settings "-discount_fallback" but
> I got an error during tuning phase like this.
>
> Exception: moses/LM/Ken.cpp:399 in Moses::LanguageModel*
> Moses::ConstructKenLM(const string&) threw util::Exception because
> `args.size() != 2'.
> Incorrect format of KenLM property:
> path=/home3/ekkim/working/E2K_1501/hierarchical_class/lm/travel22=mkcls.binlm.5
> Exit code: 1
>
> Does not "moses_chart" decoder which is distributed in the package
> support factored training?
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150203/9b2bad97/attachment-0001.htm

------------------------------

Message: 2
Date: Tue, 03 Feb 2015 11:34:31 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] facored LMs in Moses
To: joerg <tiedeman@gmail.com>, moses-support <moses-support@mit.edu>
Message-ID: <54D0B247.6090402@gmail.com>
Content-Type: text/plain; charset="windows-1252"

There's LanguageModelJoint that models several factors. However, it
hasn't been used for a long time. From the code, I know it won't work
anymore.

There's an abstract class (LanguageModelMultiFactor) ready for SRILM
factored models. But no-one's gotten round to actually integrating
SRILM's model in the past 10 years. Hopefully with interest in all
things bilingual LM, someone might do it.

On 02/02/15 10:49, joerg wrote:
>
> I would like to have an LM over several factors (in a factored
> phrase-based model). What would be the proper way to train and
> integrate such a model in Moses? Is it straightforward to use the
> factored models in SRILM? Can they be binarized in the same way as
> other models?
>
> Could I otherwise also train a standard LM (say KenLM) over several
> factors and the decoder can deal with it? If that is possible, how do
> I have to format my input for KenLM parameter estimation? Simply
> concatenating the factors I need with '|'?
>
> Thanks for helping!
> J?rg
>
> **********************************************************************************
> J?rg Tiedemann http://stp.lingfil.uu.se/~joerg/
> <http://stp.lingfil.uu.se/%7Ejoerg/>
>
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

--
Hieu Hoang
Research Associate (until March 2015)
** searching for interesting commercial MT position **
University of Edinburgh
http://www.hoang.co.uk/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150203/0c385595/attachment-0001.htm

------------------------------

Message: 3
Date: Tue, 3 Feb 2015 11:43:42 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Suggested change to train-model.perl
To: Tom Hoar <tahoar@precisiontranslationtools.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbg1OS1iJ2az-7krJRKHvfaD6srwNz1jZRKr42uM3b1N0w@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

personally, I prefer to keep that part simple and neutral to what aligner
is used. It currently supports giza++, mgiza, fast align, berkeley aligner
(probably).

it might be better to change the 'make install' in mgiza to put exe and
scripts together

On 1 February 2015 at 06:11, Tom Hoar <tahoar@precisiontranslationtools.com>
wrote:

> I acknowledge changes to train-model.perl are done with caution. Here's
> one suggestion I think will benefit anyone using MGIZA++.
>
> In the current RC-3 branch, the code looks like this starting at line 253:
>
> # supporting binaries from other packages
> my $MKCLS = "$_EXTERNAL_BINDIR/mkcls";
> my $MGIZA_MERGE_ALIGN = "$_EXTERNAL_BINDIR/merge_alignment.py";
> my $GIZA;
> my $SNT2COOC;
>
>
> A "standard" build of MGIZA includes these steps and creates a directory
> tree with $MGIZAPP_PREFIX/bin, $MGIZAPP_PREFIX/lib and
> $MGIZAPP_PREFIX/scripts.
>
> cmake . -DCMAKE_INSTALL_PREFIX="$MGIZAPP_PREFIX"
> make
> make install
>
>
> Today, the user must copy (symlink) the merge_alignment.py to complete
> the setup:
>
> $MGIZAPP_PREFIX/scripts/merge_alignment.py -->
> $MGIZAPP_PREFIX/bin/merge_alignment.py
>
> Then, the user sets the --external-bin-dir on the train-model.perl
> command line:
>
> --external-bin-dir $MGIZAPP_PREFIX/bin
>
>
> I propose this modification to train-model.perl
>
> # supporting binaries from other packages
> my $MKCLS = "$_EXTERNAL_BINDIR/mkcls";
> my $MGIZA_MERGE_ALIGN = "$_EXTERNAL_BINDIR/merge_alignment.py";
> # added to fall back to MGIZA++ default install directory
> if (! -x $MGIZA_MERGE_ALIGN) {
> $MGIZA_MERGE_ALIGN="$_EXTERNAL_BINDIR/../scripts/merge_alignment.py";
> }
> my $GIZA;
> my $SNT2COOC;
>
>
> By adding this fall back, users who setup with a "standard" MGIZA++
> `make install` don't have to copy the merge_alignment.py file, without
> losing today's functionality.
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
Hieu Hoang
Research Associate (until March 2015)
** searching for interesting commercial MT position **
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150203/119ffe10/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 100, Issue 9
*********************************************

Tweet