Moses-support Digest, Vol 85, Issue 42

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Segfault when using sigtest-filtering for hierarchical mt
(Hieu Hoang)
2. Re: Estimating probabilities with KenLM (Kenneth Heafield)
3. Re: Estimating probabilities with KenLM (Hieu Hoang)
4. Re: Estimating probabilities with KenLM (Prasanth K)


----------------------------------------------------------------------

Message: 1
Date: Mon, 25 Nov 2013 17:39:06 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Segfault when using sigtest-filtering for
hierarchical mt
To: Fabienne Braune <braune.fabienne@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbh8TQbTnO_nhH6M4NpTO4RRkGUrO_4JEK=akB8Hxq=i_g@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

found the problem line, number 24795
" [X][X] , [X] ||| [ [X][X] [X] ||| 0.000135865 0.00410465 3.36251e-05
0.0001047 ||| 0-0 1-1 ||| 18.4006 74.3493 0.25

you should escape the opening [ character on the target. tokenizer.perl and
escape-special-chars.perl will do this for you


On 22 November 2013 13:54, Fabienne Braune <braune.fabienne@gmail.com>wrote:

> Hi all,
>
> Sigtest filtering segfaults when I try to filter my rule-table. It looks
> like the line causing the crash is :
>
> [X][X] , [X][X] ! [X] ||| " [X][X] , [X][X] ! [X] ||| 0.01 0.414966 0.01
> 0.21847 ||| 0-0 1-1 2-2 3-3 4-4 ||| 0.0425902 0.0425902 0.0425902
>
> I attached a toy rule-table containing this line and on which sigtest
> crashes (test). Here is also the command line i used to lauch sigtest
> filtering :
>
> cat path-to-rule-table | mosesdecoder/contrib/sigtest-filter/filter-pt -e
> working-dir/train.en -f working-dir/train.de -l a+e -n 30 -h
>
> A strange thing is that when I create a toy table containing only a few
> entries but also the nasty line then everything runs fine. I also attached
> this file (onlycrashing).
>
> Would be great if someone could help me on this, i am out of ideas.
>
> Cheers,
> Fabienne
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131125/ecfb3dc1/attachment-0001.htm

------------------------------

Message: 2
Date: Mon, 25 Nov 2013 11:07:32 -0800
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] Estimating probabilities with KenLM
To: Prasanth K <prasanthk.ms09@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <52939FF4.2070803@kheafield.com>
Content-Type: text/plain; charset=ISO-8859-1

Hi,

I've taken a shot in the dark based on physmem.c to support physical
memory estimation on BSD and OS X. Please clone

github.com/kpu/kenlm

and compile with

./bjam

If that fails, please let Hieu and I know (maybe Hieu can help since he
has OS X). If it doesn't fail, run

bin/lmplz

with no argument. The help message will include a line e.g.

"This machine has 135224176640 bytes of memory."

or

"Unable to determine the amount of memory on this machine."

If it works, then I'll push to Moses. Trying to not break Moses master
for OS X.

Kenneth

On 11/24/13 22:40, Prasanth K wrote:
> Hi Kenneth,
>
> Thanks for the clarification w.r.t. calculating the memory size. But I
> am running these on a Mac (10.9 Mavericks). Do you think I should still
> port the lmplz code to Mac for the estimation of probabilities?
>
> One thing though, I did change the default clang compiler that comes
> with this new Mac to a gcc-4.8 (not sure that changes anything in this
> context).
>
> - Prasanth
>
>
>
>
> On Fri, Nov 22, 2013 at 6:50 PM, Kenneth Heafield <moses@kheafield.com
> <mailto:moses@kheafield.com>> wrote:
>
> Hi,
>
> What OS are you on? Cygwin? Apparently every OS reports
> memory size
> in a different way:
>
> http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=lib/physmem.c;h=2629936146e3042f927523322f18aca76996cd7f;hb=HEAD
>
> The good news is that the above code is LGPLv2:
>
> http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=modules/physmem;h=9644522e0493a85a9fb4ae7c4449741c2c1500ea;hb=HEAD
>
> But currently I'm just using this short function that will fail on some
> platforms:
>
> uint64_t GuessPhysicalMemory() {
> #if defined(_WIN32) || defined(_WIN64)
> return 0;
> #elif defined(_SC_PHYS_PAGES) && defined(_SC_PAGESIZE)
> long pages = sysconf(_SC_PHYS_PAGES);
> if (pages == -1) return 0;
> long page_size = sysconf(_SC_PAGESIZE);
> if (page_size == -1) return 0;
> return static_cast<uint64_t>(pages) *
> static_cast<uint64_t>(page_size);
> #else
> return 0;
>

Tweet