Moses-support Digest, Vol 102, Issue 53

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Working with big models (Kenneth Heafield)
2. Re: Working with big models (liling tan)
3. Stanford lemmetizer (Marwa Refaie)
4. support (Dawit Mulugeta)


----------------------------------------------------------------------

Message: 1
Date: Sat, 25 Apr 2015 16:15:34 -0400
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] Working with big models
To: moses-support@mit.edu
Message-ID: <553BF5E6.5050802@kheafield.com>
Content-Type: text/plain; charset=windows-1252

How much RAM do you have?

Have you tried Marcin's suggestion regarding the trie?

To filter you need to 1) have a test set 2) match that test set to the
phrase table to produce a target vocabulary and 3) provide that target
vocabulary to the filter. See
https://github.com/kpu/kenlm/blob/master/lm/filter/phrase_table_vocab_main.cc
which should work if you don't have factors .

Kenneth

On 04/25/2015 04:08 PM, liling tan wrote:
> Dear Moses devs/users,
>
> @Ken, MERT with 100 iterations might be an overkill but MERT with 20
> also wouldn't finished in a day on the 71GB binarzied LM. It was at the
> 5th iteration when i killed it.
>
> @Marcin, I'll try to retune with the trie LM and see how far it goes.
>
> *Does the build_binary come with thread options?*
> *
> *
> *Also, would the KenLM filter help? How does the filter work? *
>
> I've tried on the 71GB LM:
>
> ~/moses/bin/filter union lm.en.binary lm.filter.binary
>
>
> After waiting for 15 mins, it sort of looks like there nothing going on
> and looks stuck.
> More details
> on: http://stackoverflow.com/questions/29869607/how-to-tune-a-machine-translation-model-with-huge-language-model
>
> Regards,
> Liling
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

Message: 2
Date: Sat, 25 Apr 2015 22:37:49 +0200
From: liling tan <alvations@gmail.com>
Subject: Re: [Moses-support] Working with big models
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAKzPaJLExZm6cw4By3n+9MOKKSv6R6s2uWfPdJmUjq_+5WG8qQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Dear Moses devs/users

@Ken, thanks for the link to the vocab extraction code and the explanation.

Building trie binaries takes a little long on a single core. Is there some
-threads options for build_binary?

Regards,
Liling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150425/ced35057/attachment-0001.htm

------------------------------

Message: 3
Date: Sat, 25 Apr 2015 22:45:27 +0000
From: Marwa Refaie <basmallah@hotmail.com>
Subject: [Moses-support] Stanford lemmetizer
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <DUB118-W22C0114B37C1EBD0890B34BAEB0@phx.gbl>
Content-Type: text/plain; charset="windows-1256"

Hi all,
I need to use word lemma & pos, here example of stanford output for lemmtizer & pos

<pos><sentence id="0"> <word wid="0" pos="JJ" lemma="artificial">Artificial</word> <word wid="1" pos="NN" lemma="intelligence">intelligence</word></sentence><sentence id="1"> <word wid="0" pos="JJ" lemma="artificial">Artificial</word> <word wid="1" pos="NN" lemma="intelligence">intelligence</word> <word wid="2" pos="-LRB-" lemma="-lrb-">-LRB-</word>
Any one knows how to convert this to the Moses factored models
Artificial | artificial | NN intelligence |intelligence | NN |
Marwa N. Refaie


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150425/964c38bf/attachment-0001.htm

------------------------------

Message: 4
Date: Sun, 26 Apr 2015 18:42:09 +0300
From: Dawit Mulugeta <mulugetadawit@gmail.com>
Subject: [Moses-support] support
To: moses-support@mit.edu
Message-ID:
<CAJ17VqBBh2Z9wcqv7GMKkAtHJzR7cw_idBSg7cV-=NZzddd2UQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Dear Sir/madam,

I was trying to compile IRSTLM with moses. I was doing my project on IRSTLM
language model with moses. I synthax I have used is

./bjam --with-irstlm=/home/dawit/irstlm-5.80.03 -j5

But I fails to build.

have properly followed the instruction on
http://www.statmt.org/moses/?n=Development.GetStarted

. I have tried it on ubuntu 12.04(32bit) and 14.04(64bit).I have
attached herewith the log file.

Please help me on this issue.

Best wishes,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150426/47e0b34b/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Untitled 1.odt
Type: application/vnd.oasis.opendocument.text
Size: 19086 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20150426/47e0b34b/attachment.bin

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 102, Issue 53
**********************************************

Tweet