Moses-support Digest, Vol 89, Issue 38

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Exception: bitset::set (Philipp Koehn)
2. Re: train-model.perl creates moses.ini that moses can't
handle (Peter Kleiweg)
3. Failed to compile moses (Steven Xu)
4. Re: Exception: bitset::set (Rajen Chatterjee)
5. Re: train-model.perl creates moses.ini that moses can't
handle (Peter Kleiweg)
6. Re: Fwd: Moses-support post from
shachar.mirkin@xrce.xerox.com requires approval (Mirkin, Shachar)


----------------------------------------------------------------------

Message: 1
Date: Mon, 17 Mar 2014 15:25:25 -0400
From: Philipp Koehn <pkoehn@inf.ed.ac.uk>
Subject: Re: [Moses-support] Exception: bitset::set
To: Rajen Chatterjee <rajen.k.chatterjee@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDB-iV3E=6EwtdnQEEu+5MnR3ZEuc-T43SR=V6ySXo+y2A@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

Hi,

see the same answer to the earlier email:

By default, the maximum number of factors is 4, but you are using 5
output factors.

You can recompile Moses with an additional switch:
bjam --max-factors=5

-phi

On Sun, Mar 16, 2014 at 12:33 PM, Rajen Chatterjee
<rajen.k.chatterjee@gmail.com> wrote:
> Hi All,
>
> While running decoder I am getting following exception:
>
> Start loading PhraseTable
> /home/rajen/Public/SMT/experiments/Project/result/gorn/en-hi/moses_data/model/phrase-table.0,4-0.gz
> : [32.062] seconds
> filePath:
> /home/rajen/Public/SMT/experiments/Project/result/gorn/en-hi/moses_data/model/phrase-table.0,4-0.gz
> ScoreProducer: PhraseModel start: 10 end: 15
> Exception: bitset::set
>
> Any idea how to solve this?
>
>
> --
> -Regards,
> Rajen Chatterjee.
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

Message: 2
Date: Mon, 17 Mar 2014 20:45:31 +0100 (CET)
From: Peter Kleiweg <p.c.j.kleiweg@rug.nl>
Subject: Re: [Moses-support] train-model.perl creates moses.ini that
moses can't handle
To: moses-support@mit.edu
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <alpine.DEB.2.00.1403172025090.3975@pebbe>
Content-Type: TEXT/PLAIN; charset=US-ASCII

Philipp Koehn schreef op de 17e dag van de lentemaand van het jaar 2014:

> Hi,
>
> by default, the maximum number of factors is 4, but you are using 5
> output factors.
>
> You can recompile Moses with an additional switch:
> bjam --max-factors=5

I tried that. I get the same error.

Hmm, tried that again, and now it does work without error.
Strange.


> I am a bit concerned about your model since it generates a large number
> of output factors independently, which may lead to an explosion of possible
> translation options where all good choices get pruned out before you get
> to the generation step which will eliminate all the impossible combinations.
>
> It would be probably a good idea to use this complex factored setup only
> as a back-off for unknown words. For details, please look at the following
> paper: http://amta2012.amtaweb.org/AMTA2012Files/papers/147.pdf


Thanks. I will look into that. I am still trying to understand
how to use factors to get a better translator.

I have two goals.

1. I want to know what Moses is capable of, to compare its best
result to the result of a new translation system that is still
being develloped. For this, I need to get an understanding of
factored models and hierarchical models (First result with
hierarchical model: it works, but worse than basic model).

2. I want to get word alignments that are as good as possible,
because we need it for building the new system. And I have seen
some very bad word alignments produced by Giza++. Currently,
Moses uses factors only in the model. I am wondering if you
could use factors to get better word alignments. Another thing I
was thinking about is if you could use a trained (and
tuned) model to guide the word alignment, that way improving on
the original alignment, possably in an iterative process.

At the moment, I am still strugling with goal 1.


--
Peter Kleiweg
http://pkleiweg.home.xs4all.nl/


------------------------------

Message: 3
Date: Mon, 17 Mar 2014 16:33:42 -0400
From: Steven Xu <steven.xu@lba.ca>
Subject: [Moses-support] Failed to compile moses
To: moses-support@mit.edu
Message-ID: <20140317163342.Horde.lbcfDOc3P8sHrE_Kf5A7fA1@smtp.lba.ca>
Content-Type: text/plain; charset="utf-8"

Hi, All

I have problem to compile moses. I followed exactly the steps
described in the online document(
http://www.statmt.org/moses/?n=Development.GetStarted):

1) I compiled boost successfully.
wget
http://downloads.sourceforge.net/project/boost/boost/1.55.0/boost_1_55_0.tar.gz?r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fboost%2Ffiles%2Fboost%2F1.55.0%2F&ts=1389613041&use_mirror=kent
tar zxvf boost_1_55_0.tar.gz
cd boost_1_55_0/
./bootstrap.sh
./b2 -j8 --prefix=$PWD --libdir=$PWD/lib64 --layout=tagged
link=static threading=multi,single install || echo FAILURE

2) I run into probem at
./bjam --with-boost=/root/Desktop/boost_1_55_0 -j8

This is the command I collected the logs:
./jam-files/bjam --with-boost=/root/Desktop/boost_1_55_0 -j8
--debug-configuration -d2 |gzip >build.log.gz

Thanks for your answer.

Steven Xu

--
Thanks
Steven Xu
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log.gz
Type: application/x-gzip
Size: 229 bytes
Desc: build.log.gz
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140317/c363f769/attachment-0001.bin

------------------------------

Message: 4
Date: Mon, 17 Mar 2014 22:14:14 +0000
From: Rajen Chatterjee <rajen.k.chatterjee@gmail.com>
Subject: Re: [Moses-support] Exception: bitset::set
To: Philipp Koehn <pkoehn@inf.ed.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAC4-+NyWTdCiOpH-RdnHYqVEDONP0=sntdbb1yQQFRz=T-COvw@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Thanks.


On Mon, Mar 17, 2014 at 7:25 PM, Philipp Koehn <pkoehn@inf.ed.ac.uk> wrote:

> Hi,
>
> see the same answer to the earlier email:
>
> By default, the maximum number of factors is 4, but you are using 5
> output factors.
>
> You can recompile Moses with an additional switch:
> bjam --max-factors=5
>
> -phi
>
> On Sun, Mar 16, 2014 at 12:33 PM, Rajen Chatterjee
> <rajen.k.chatterjee@gmail.com> wrote:
> > Hi All,
> >
> > While running decoder I am getting following exception:
> >
> > Start loading PhraseTable
> >
> /home/rajen/Public/SMT/experiments/Project/result/gorn/en-hi/moses_data/model/phrase-table.0,4-0.gz
> > : [32.062] seconds
> > filePath:
> >
> /home/rajen/Public/SMT/experiments/Project/result/gorn/en-hi/moses_data/model/phrase-table.0,4-0.gz
> > ScoreProducer: PhraseModel start: 10 end: 15
> > Exception: bitset::set
> >
> > Any idea how to solve this?
> >
> >
> > --
> > -Regards,
> > Rajen Chatterjee.
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
>



--
-Regards,
Rajen Chatterjee.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140317/3fcdc7fb/attachment-0001.htm

------------------------------

Message: 5
Date: Mon, 17 Mar 2014 23:31:15 +0100 (CET)
From: Peter Kleiweg <p.c.j.kleiweg@rug.nl>
Subject: Re: [Moses-support] train-model.perl creates moses.ini that
moses can't handle
To: moses-support@mit.edu
Message-ID: <alpine.DEB.2.00.1403172329270.3975@pebbe>
Content-Type: TEXT/PLAIN; charset=US-ASCII

Peter Kleiweg schreef op de 17e dag van de lentemaand van het jaar 2014:

> Philipp Koehn schreef op de 17e dag van de lentemaand van het jaar 2014:
>
> > Hi,
> >
> > by default, the maximum number of factors is 4, but you are using 5
> > output factors.
> >
> > You can recompile Moses with an additional switch:
> > bjam --max-factors=5
>
> I tried that. I get the same error.
>
> Hmm, tried that again, and now it does work without error.
> Strange.

But now I get a segmentation fault when it tries to translate
the first sentence.

Never mind. The specifications I used for training were probably
not very sensible.


--
Peter Kleiweg
http://pkleiweg.home.xs4all.nl/


------------------------------

Message: 6
Date: Mon, 17 Mar 2014 17:17:10 +0100
From: "Mirkin, Shachar" <shachar.mirkin@xrce.xerox.com>
Subject: Re: [Moses-support] Fwd: Moses-support post from
shachar.mirkin@xrce.xerox.com requires approval
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>, moses-support
<moses-support@mit.edu>, Ulrich Germann <ugermann@inf.ed.ac.uk>
Message-ID: <53272006.7000704@xrce.xerox.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi,

I'm now subscribed also from this email address.

Let me give more details about the problems that I encountered.
Trying to load the Moses server with the modified ini file, after
replacing the PhraseDictionaryBinary line with:

PhraseDictionaryDynSuffixArray source=<path-to-source-corpus> target=<path-to-target-corpus> alignment=<path-to-alignments>

(with the correct paths, of course), I got:

Feature function PhraseDictionaryDynSuffixArray0 specified 1 dense
scores or weights. Actually has 0

This was solved by adding "num-features=0" to the
PhraseDictionaryDynSuffixArray line.

The next error was:

...
Loading source corpus...
terminate called after throwing an instance of 'Moses::StrayFactorException'
what(): moses/Word.cpp:112 in void
Moses::Word::CreateFromString(Moses::FactorDirection, const
std::vector<long unsigned int, std::allocator<long unsigned int> >&,
const StringPiece&, bool) threw StrayFactorException because `fit'.
You have configured 0 factors but the word le contains factor delimiter
| too many times.

In this test my source, target and alignment files consist each of a
single line with no "|"s, and the word "le" is the first one in the source.

Is there anything else I should do in the ini file?

Thanks,
Shachar



On 03/17/2014 02:58 PM, Hieu Hoang wrote:
> Hi Shachar
>
> can you please subscribe to the mailing list before posting to it.
> It's a public email address so there's a lot of automated spammers.
> You can subscribe here
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
> To answer you question - the webpage does document it in the new ini
> format, eg.
> PhraseDictionaryDynSuffixArray source=<path-to-source-corpus> ...
> Do you have a printout of the old version?
>
> Also, the dynamic suffix array is undergoing updates as Uli Germann
> (cc'ed) is updating it with more features. He can tell you more about it
>
>
> ---------- Forwarded message ----------
> From: <moses-support-owner@mit.edu <mailto:moses-support-owner@mit.edu>>
> Date: 17 March 2014 12:13
> Subject: Moses-support post from shachar.mirkin@xrce.xerox.com
> <mailto:shachar.mirkin@xrce.xerox.com> requires approval
> To: moses-support-owner@mit.edu <mailto:moses-support-owner@mit.edu>
>
>
> As list administrator, your authorization is requested for the
> following mailing list posting:
>
> List: Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> From: shachar.mirkin@xrce.xerox.com
> <mailto:shachar.mirkin@xrce.xerox.com>
> Subject: Incremental training and the new ini format
> Reason: Post by non-member to a members-only list
>
> At your convenience, visit:
>
> http://mailman.mit.edu/mailman/admindb/moses-support
>
> to approve or deny the request.
>
>
> ---------- Forwarded message ----------
> From: "Mirkin, Shachar" <shachar.mirkin@xrce.xerox.com
> <mailto:shachar.mirkin@xrce.xerox.com>>
> To: moses-support@mit.edu <mailto:moses-support@mit.edu>
> Cc:
> Date: Mon, 17 Mar 2014 13:06:47 +0100
> Subject: Incremental training and the new ini format
> Hi,
>
> I'm trying to use incremental training with the latest Moses version,
> but the documentation refers to the old ini format
> (http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc34).
> Can you please explain what changes are required to get the
> incremental training working with the new ini format?
>
> Thanks,
> Shachar
>
>
>
>
> ---------- Forwarded message ----------
> From: moses-support-request@mit.edu <mailto:moses-support-request@mit.edu>
> To:
> Cc:
> Date:
> Subject: confirm 2701c5fb8f659b6037c9e0bf07ad70095ba4ffe2
> If you reply to this message, keeping the Subject: header intact,
> Mailman will discard the held message. Do this if the message is
> spam. If you reply to this message and include an Approved: header
> with the list password in it, the message will be approved for posting
> to the list. The Approved: header can also appear in the first line
> of the body of the reply.
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140317/0e7f6a3d/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 89, Issue 38
*********************************************

Tweet