Moses-support Digest, Vol 103, Issue 65

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Tuning error - Feature name IRSTLM is not registered
(Ulrich Germann)
2. Re: New source formatter & checker (Hieu Hoang)
3. Re: New source formatter & checker (Ulrich Germann)


----------------------------------------------------------------------

Message: 1
Date: Tue, 26 May 2015 18:39:19 +0100
From: Ulrich Germann <ulrich.germann@gmail.com>
Subject: Re: [Moses-support] Tuning error - Feature name IRSTLM is not
registered
To: Marco Damonte <mdtux89@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAHQSRUo-bmzPJGWpXtXAPSqvqHmh67JGAjSunD1=-8=xLW1+OA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Follow-up (thanks to Matthias Huck for the info):

If you want to use KenLM, your [LM] section in your EMS config file should
look similar to this. Most likely your setup specifies the wrong type.

[LM]

### tool to be used for language model training
# kenlm training
lm-training = "$moses-script-dir/ems/support/lmplz-wrapper.perl -bin
$moses-bin-dir/lmplz"
settings = "--prune '0 0 1' -T $working-dir/lm -S 10%"
lm-binarizer = $moses-src-dir/bin/build_binary
type = 8



On Tue, May 26, 2015 at 5:55 PM, Ulrich Germann <ulrich.germann@gmail.com>
wrote:

> Hi Marco,
>
> IRSTLM is an external library and needs to be compiled into moses
> explicitly. It's fairly rivial to accomplish that.
>
> 1. Get and unpack the IRSTLM tarball (that's a big 'o' and not a zero
> after the -q
>
> wget -qO-
> http://downloads.sourceforge.net/project/irstlm/irstlm/irstlm-5.80/irstlm-5.80.08.tgz
> | tar xzf-
>
> 2. cd irstlm-5.80.08/trunk
> 3. ./regenerate-makefiles.sh
> 4. ./configure --prefix=/some/path/irstlm
> 5. make && make install
>
> 6. Run bjam with --with-irstlm=/some/path/irstlm to compile moses and -a
> to make sure everything gets recompiled.
>
> Obviously /some/path is only a placeholder here.
>
> Once moses is compiled, you can definitely remove the directory
> irstlm-5.80.08 and everything in it.
> If you plan to recompile moses at some point, keep /some/path/irstlm,
> otherwise it shouldn't be needed any more, either.
>
> I suppose you are using EMS to run your experiments. I don't use it, so
> others will have to chime in on how to set it up for irstlm (or check the
> online Moses documentation (e.g.
> http://www.statmt.org/moses/?n=FactoredTraining.EMS).
>
> Cheers - Uli
>
>
>
> On Tue, May 26, 2015 at 3:43 PM, Marco Damonte <mdtux89@gmail.com> wrote:
>
>> Hi all,
>>
>> I am new with Moses. My experiment crashed during the tuning phase with
>> the error "Feature name IRSTLM is not registered". I understand from a
>> previous post on this mailing list (
>> http://comments.gmane.org/gmane.comp.nlp.moses.user/10344) that I should
>> either recompile moses with irstlm or use kenlm.
>>
>> My question is: how exactly should I replace irstlm with kenlm? My
>> configuration file doesn't mention irstlm at all.
>>
>> Thanks in advance.
>> Marco
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
>
> --
> Ulrich Germann
> Senior Researcher
> School of Informatics
> University of Edinburgh
>



--
Ulrich Germann
Senior Researcher
School of Informatics
University of Edinburgh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150526/ee1ecd7a/attachment-0001.htm

------------------------------

Message: 2
Date: Tue, 26 May 2015 21:39:50 +0400
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] New source formatter & checker
To: Ulrich Germann <ugermann@inf.ed.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAEKMkbg7UynnJPiEW1YW8EAHmunqz9bekSLVRmT_suWc1MBj+A@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

ah, the style guide that no-one has ever bothered to follow since I
originally wrote it 10 years ago.

I changed it in Jan to reflect the style that people actually check in, so
presumably prefer. The beautifier tries to mimick this style. I will change
the style guide back.

fyi - the original style is based on microsoft, eg
http://blogs.msdn.com/b/brada/archive/2005/01/26/361363.aspx
i guess people prefer a unix-based style, eg
https://www.kernel.org/doc/Documentation/CodingStyle


I think beautifying is a necessary 'cos we have a distributed project of
committers who use different editors and OSes. It provides some consistency.

I regularly see you commit changes where you've manually wrap long lines.
Youre welcome to change the beautifier to enforce this. However, I prefer
that the beautifier isn't changed too much otherwise the code will flip
flop for no good reason.



Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

On 26 May 2015 at 20:30, Ulrich Germann <ulrich.germann@gmail.com> wrote:

> Hi Jeroen,
>
> on the matter of style I'd like to point out that the official style
> guidelines for Moses code require opening braces on a separate line.
>
> http://www.statmt.org/moses/?n=Moses.CodeStyle
>
> The official style has always required this since the first entry about
> this in the Wiki back in 2009. Recently (Jan 16 this year) the page was
> vandalized by an anonymous editor to claim something to the contrary, but
> the page has now been restored to what it should be. So while you're
> fiddling with the beautify scripts, please fix this as well.
>
> I'm personally no friend of automatic "beautification", because it does
> far more harm than good in my opinion, but if you insist on doing it, it
> should be done right.
>
> Best regards - Uli
>
> On Sun, May 17, 2015 at 2:13 PM, Jeroen Vermeulen <
> jtv@precisiontranslationtools.com> wrote:
>
>> Hi all,
>>
>> We have a replacement for the old beautify.perl script:
>> scripts/other/beautify.py.
>>
>> It does one of two things, or both:
>> * Re-format C/C++ source code, just like the old script did.
>> * Check for style errors and such.
>>
>> This last thing is called a "lint" check. For this I chose Pocketlint,
>> a checker I have good experiences with, although if people want a
>> different one (or additional checks) I can change that.
>>
>> I fixed most of the lint that got reported, except in JavaScript code.
>> We may add automatic reformatting for additional languages later. I
>> sincerely hope all of this does not cause any serious merge problems for
>> your branches.
>>
>> Ideally, everyone would get in the habit of installing Pocketlint and
>> running this script regularly whether they accidentally added any lint.
>> To see how the script works, run:
>>
>> ./scripts/other/beautify.py -h
>>
>> The lint check processes a few files at a time. By default it stops
>> when it sees lint. If you want to see a full check, use the -i option.
>>
>>
>> Jeroen
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
>
> --
> Ulrich Germann
> Senior Researcher
> School of Informatics
> University of Edinburgh
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150526/99d788ba/attachment-0001.htm

------------------------------

Message: 3
Date: Tue, 26 May 2015 20:47:02 +0100
From: Ulrich Germann <ulrich.germann@gmail.com>
Subject: Re: [Moses-support] New source formatter & checker
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAHQSRUqOiZR36LQEb866PKjre=5W5YNhJPWK79qRKoup8kab3A@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Well, the second document does indeed give some darn good arguments for
trailing braces: 1. "
as shown to us by the prophets Kernighan and Ritchie
"; 2. "
K&R are _right_ and [...] K&R are right
"; 3. "
think 25-line terminal screens here
"

The document also has other great suggestions that we may want to adopt."
Tabs are 8 characters, and thus indentations are also 8 characters
"

The fact is: with trailing braces and steadfast refusal to use vertical
space to improve
code readibility, you do need a tab width of 8 or so to be able to
recognize code blocks.

Incidentally, there's actually one recommendation in that document that IS
worth
adopting *(and in which pretty much all style guides agree plus-minus 5
characters or so):*

The limit on the length of lines is 80 columns and this is a
strongly preferred limit.

Let me repeat that:

The limit on the length of lines is 80 columns and this is a
strongly preferred limit.

And again:

The limit on the length of lines is 80 columns and this is a
strongly preferred limit.

That is the only thing worth really considering from that document.

We can safely consider the 25-line limit a thing of the past unless you
program on
your mobile phone. However, there is a good justification for the 80-column
limit: merging code, where it's useful to see the code side by side. When
resolving merge conflicts in git, we are usually dealing with 3 versions
that need
to be compared. Those three columns won't fit on my screen when lines are
120+
characters long.

I digress. Back to the braces issue. Unlike the arguments for trailing
braces
(K&R did it in a printed book to save paper; I'm a retro programmer who
likes to
do things old school; it obscures my code so people will think I'm really
awesome
because they can't really understand what I'm doing without lots of effort,
just like
Derrida and Piaget didn't write to be understood --- they wrote NOT to be
understood),
so unlike these arguments, there are very good arguments for having opening
braces
on a new line and vertically aligned with the closing bracket.

1. It's much easier to recognize logical code blocks. It is. Try it out.
Read code that you
haven't written and be honest with yourself. It may seem unfamiliar at
first, but once
you are used to it, there's no turning back.

2. If you want to temporarily disable a condition (e.g. in debugging), you
just comment
out the if statement. You do not have to THEN put an opening brace at
the beginning
of the line (extra editing) and remove it later (extra editing again)
--- which also
FORCES you to be inconsistent. (You can't always remove the braces
because
of variable scoping). You have the same issue when you want to use
preprocessor
commands to have conditions only under certain circumstances, so instead
of
#ifdef ENABLE_CONDITION
if (condition)

Tweet