Moses-support Digest, Vol 103, Issue 64

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: New source formatter & checker (Ulrich Germann)
2. Re: Tuning error - Feature name IRSTLM is not registered
(Ulrich Germann)
3. MML and IRSTLM (Tomas Fulajtar)


----------------------------------------------------------------------

Message: 1
Date: Tue, 26 May 2015 17:30:49 +0100
From: Ulrich Germann <ulrich.germann@gmail.com>
Subject: Re: [Moses-support] New source formatter & checker
To: Jeroen Vermeulen <jtv@precisiontranslationtools.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAHQSRUoapkXp=ix0nUBaaSd_DW9FC=Y-20FQRXj6-6ob4qjJsg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Jeroen,

on the matter of style I'd like to point out that the official style
guidelines for Moses code require opening braces on a separate line.

http://www.statmt.org/moses/?n=Moses.CodeStyle

The official style has always required this since the first entry about
this in the Wiki back in 2009. Recently (Jan 16 this year) the page was
vandalized by an anonymous editor to claim something to the contrary, but
the page has now been restored to what it should be. So while you're
fiddling with the beautify scripts, please fix this as well.

I'm personally no friend of automatic "beautification", because it does far
more harm than good in my opinion, but if you insist on doing it, it should
be done right.

Best regards - Uli

On Sun, May 17, 2015 at 2:13 PM, Jeroen Vermeulen <
jtv@precisiontranslationtools.com> wrote:

> Hi all,
>
> We have a replacement for the old beautify.perl script:
> scripts/other/beautify.py.
>
> It does one of two things, or both:
> * Re-format C/C++ source code, just like the old script did.
> * Check for style errors and such.
>
> This last thing is called a "lint" check. For this I chose Pocketlint,
> a checker I have good experiences with, although if people want a
> different one (or additional checks) I can change that.
>
> I fixed most of the lint that got reported, except in JavaScript code.
> We may add automatic reformatting for additional languages later. I
> sincerely hope all of this does not cause any serious merge problems for
> your branches.
>
> Ideally, everyone would get in the habit of installing Pocketlint and
> running this script regularly whether they accidentally added any lint.
> To see how the script works, run:
>
> ./scripts/other/beautify.py -h
>
> The lint check processes a few files at a time. By default it stops
> when it sees lint. If you want to see a full check, use the -i option.
>
>
> Jeroen
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Ulrich Germann
Senior Researcher
School of Informatics
University of Edinburgh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150526/d9784bec/attachment-0001.htm

------------------------------

Message: 2
Date: Tue, 26 May 2015 17:55:02 +0100
From: Ulrich Germann <ulrich.germann@gmail.com>
Subject: Re: [Moses-support] Tuning error - Feature name IRSTLM is not
registered
To: Marco Damonte <mdtux89@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAHQSRUoVJYxh5Z6ge4N4JkhtqFexP2W8wvT=d8+_E0ihhcERhA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Marco,

IRSTLM is an external library and needs to be compiled into moses
explicitly. It's fairly rivial to accomplish that.

1. Get and unpack the IRSTLM tarball (that's a big 'o' and not a zero after
the -q

wget -qO-
http://downloads.sourceforge.net/project/irstlm/irstlm/irstlm-5.80/irstlm-5.80.08.tgz
| tar xzf-

2. cd irstlm-5.80.08/trunk
3. ./regenerate-makefiles.sh
4. ./configure --prefix=/some/path/irstlm
5. make && make install

6. Run bjam with --with-irstlm=/some/path/irstlm to compile moses and -a to
make sure everything gets recompiled.

Obviously /some/path is only a placeholder here.

Once moses is compiled, you can definitely remove the directory
irstlm-5.80.08 and everything in it.
If you plan to recompile moses at some point, keep /some/path/irstlm,
otherwise it shouldn't be needed any more, either.

I suppose you are using EMS to run your experiments. I don't use it, so
others will have to chime in on how to set it up for irstlm (or check the
online Moses documentation (e.g.
http://www.statmt.org/moses/?n=FactoredTraining.EMS).

Cheers - Uli



On Tue, May 26, 2015 at 3:43 PM, Marco Damonte <mdtux89@gmail.com> wrote:

> Hi all,
>
> I am new with Moses. My experiment crashed during the tuning phase with
> the error "Feature name IRSTLM is not registered". I understand from a
> previous post on this mailing list (
> http://comments.gmane.org/gmane.comp.nlp.moses.user/10344) that I should
> either recompile moses with irstlm or use kenlm.
>
> My question is: how exactly should I replace irstlm with kenlm? My
> configuration file doesn't mention irstlm at all.
>
> Thanks in advance.
> Marco
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Ulrich Germann
Senior Researcher
School of Informatics
University of Edinburgh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150526/abe59b7f/attachment-0001.htm

------------------------------

Message: 3
Date: Tue, 26 May 2015 17:00:32 +0000
From: Tomas Fulajtar <TomasFu@moravia.com>
Subject: [Moses-support] MML and IRSTLM
To: "moses-support (moses-support@mit.edu)" <moses-support@mit.edu>
Message-ID:
<BY1PR0201MB09655A25D3816E70F2D7E9F5ADCC0@BY1PR0201MB0965.namprd02.prod.outlook.com>

Content-Type: text/plain; charset="utf-8"

Dear all,

I am trying to perform the Modified Moore-Lewis filtering with the IRSTLM framework (version 5.80.06). Looking into ems/support/mml-train.perl, there is the ?vocab param for SRILM which does following:


-vocab file
Read a vocabulary from file. Subsequently, out-of-vocabulary words in both counts or text are replaced with the unknown-word token. If this option is not specified all words found are implicitly added to the vocabulary.

See http://www.speech.sri.com/projects/srilm/manpages/ngram-count.1.html.

I have found that most similar approach in IRSTLM is with the ?sd option for ngt command tool. My question for experts is if this is comparable with SRILM tool. I have read the source ,but just got a basic idea.

>From IRSTLM manual:

A new n-gram table for the limited dictionary can be computed with ngt by specifying the sub-dictionary:
$> ngt -i=train.www -sd=top10k -n=3 -o=train.10k.www -b=yes
The commandreplaces all words outside top10K with the special out-of-vocabulary symbol unk .

I think it would be useful to also update the train-irst2.pl accordingly. I can provide patched file as well once verify the functionality.


Thank you,

Tom?? Fulajt?r | Researcher
T: +420-545-552-340
tomasfu@moravia.com<mailto:tomasfu@moravia.com> | moravia.com<http://www.moravia.com/> | Skype: tomasfulajtar

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150526/946059f9/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 103, Issue 64
**********************************************

Tweet