Moses-support Digest, Vol 102, Issue 54

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Working with big models (liling tan)
2. Re: Working with big models (Marcin Junczys-Dowmunt)
3. Re: Working with big models (Jon D.)
4. Re: support (Abdelfetah Boumerdas)


----------------------------------------------------------------------

Message: 1
Date: Sun, 26 Apr 2015 21:20:15 +0200
From: liling tan <alvations@gmail.com>
Subject: Re: [Moses-support] Working with big models
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAKzPaJ+fN5=5sYK4hxMyuXY0hfKquvGNry2bnQ-B98DG_8u__A@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Dear Moses devs/users,

@Ken, I'm working with 128 RAM, the default binarized LM works but it's
kind of slow when tuning.

I've tried the trie and it's wonderful!! Effectively, it brought down the
size of the LM:

Text: 16GB
ARPA: 38G
Binary (no trie): 71GB
Trie Binary: 17GB

*Does the small trie binary look normal? *(it looks magically small, but
i've tested and it didn't give me a difference in BLEU).

The decoding speed reduces, I'm waiting to see how much time saved when
tuning.

The only issue is that using trie binarization takes much longer than
normal binary. It took a night's sleep (5-6 hours) to complete while the
normal binarization takes 1-2 hours. Still it's worth the time since it
reduces the size of the model and decoding speed.

@Marcin, @Ken thanks for the tips on working with big LM and phrase models.

Regards,
Liling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150426/d217aa77/attachment-0001.htm

------------------------------

Message: 2
Date: Sun, 26 Apr 2015 21:25:24 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Working with big models
To: moses-support@mit.edu
Message-ID: <553D3BA4.8020700@amu.edu.pl>
Content-Type: text/plain; charset="windows-1252"

Size reduction and binarizing time are normal. At WIPO we use only
quantized models, with no quality loss so far. Your speed issues were
caused by insufficient RAM then. Interesting, since I was able to use
98GB models on my 128GB server with several moses instances running in
parallel, but maybe you had other stuff running.

W dniu 26.04.2015 o 21:20, liling tan pisze:
> Dear Moses devs/users,
>
> @Ken, I'm working with 128 RAM, the default binarized LM works but
> it's kind of slow when tuning.
>
> I've tried the trie and it's wonderful!! Effectively, it brought down
> the size of the LM:
>
> Text: 16GB
> ARPA: 38G
> Binary (no trie): 71GB
> Trie Binary: 17GB
>
> *Does the small trie binary look normal? *(it looks magically small,
> but i've tested and it didn't give me a difference in BLEU).
>
> The decoding speed reduces, I'm waiting to see how much time saved
> when tuning.
>
> The only issue is that using trie binarization takes much longer than
> normal binary. It took a night's sleep (5-6 hours) to complete while
> the normal binarization takes 1-2 hours. Still it's worth the time
> since it reduces the size of the model and decoding speed.
>
> @Marcin, @Ken thanks for the tips on working with big LM and phrase
> models.
>
> Regards,
> Liling
>
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150426/218ab5ed/attachment-0001.htm

------------------------------

Message: 3
Date: Sun, 26 Apr 2015 22:37:36 +0200
From: "Jon D." <jonsafari@gmail.com>
Subject: Re: [Moses-support] Working with big models
To: moses-support <moses-support@mit.edu>
Message-ID:
<CA+Y6dkcYmYSGNOqRmQ-GqN5WgK2kZu5W+PKwWpEbzSgMU-3kVg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

The size of the trie looks unusually small unless you did quantization as
well. I usually see non-quantized KenLM tries being about 50-70% the
filesize of probing hash tables. The trie binarization time sounds normal
relative to the hash time.

Best,
-Jon



On Sun, Apr 26, 2015 at 9:20 PM, liling tan <alvations@gmail.com> wrote:

> Dear Moses devs/users,
>
> @Ken, I'm working with 128 RAM, the default binarized LM works but it's
> kind of slow when tuning.
>
> I've tried the trie and it's wonderful!! Effectively, it brought down the
> size of the LM:
>
> Text: 16GB
> ARPA: 38G
> Binary (no trie): 71GB
> Trie Binary: 17GB
>
> *Does the small trie binary look normal? *(it looks magically small, but
> i've tested and it didn't give me a difference in BLEU).
>
> The decoding speed reduces, I'm waiting to see how much time saved when
> tuning.
>
> The only issue is that using trie binarization takes much longer than
> normal binary. It took a night's sleep (5-6 hours) to complete while the
> normal binarization takes 1-2 hours. Still it's worth the time since it
> reduces the size of the model and decoding speed.
>
> @Marcin, @Ken thanks for the tips on working with big LM and phrase
> models.
>
> Regards,
> Liling
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150426/a74fe175/attachment-0001.htm

------------------------------

Message: 4
Date: Sun, 26 Apr 2015 21:38:22 +0100
From: Abdelfetah Boumerdas <aa_boumerdas@esi.dz>
Subject: Re: [Moses-support] support
To: Dawit Mulugeta <mulugetadawit@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CABJLC3c=75G6vinxbsM5Wu041W8ceAxsKE+e8E4utMnMaMcrZg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hello
I had the same problem once and i have been told to recompile moses with
the "-a" option and it worked just fine. Just notice that from what i've
noticed the "-a" option recompile moses from scratch so if you compiled
moses with some other libraries like for example the boost library you
should add it as well.

I wish this helps you. Good luck.
On 26 Apr 2015 16:46, "Dawit Mulugeta" <mulugetadawit@gmail.com> wrote:

> Dear Sir/madam,
>
> I was trying to compile IRSTLM with moses. I was doing my project on
> IRSTLM language model with moses. I synthax I have used is
>
> ./bjam --with-irstlm=/home/dawit/irstlm-5.80.03 -j5
>
> But I fails to build.
>
> have properly followed the instruction on http://www.statmt.org/moses/?n=Development.GetStarted
>
> . I have tried it on ubuntu 12.04(32bit) and 14.04(64bit).I have attached herewith the log file.
>
> Please help me on this issue.
>
> Best wishes,
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150426/27827f0c/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 102, Issue 54
**********************************************

Tweet