Moses-support Digest, Vol 102, Issue 63

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Adding a language model built on Google Web (Hieu Hoang)
2. Re: Adding a language model built on Google Web
(Marcin Junczys-Dowmunt)


----------------------------------------------------------------------

Message: 1
Date: Tue, 28 Apr 2015 18:06:23 +0400
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Adding a language model built on Google
Web
To: Alla Rozovskaya <sigaliyah@gmail.com>, moses-support@mit.edu
Message-ID: <553F93DF.80607@gmail.com>
Content-Type: text/plain; charset="windows-1252"

I spoke to Ken about using KenLM to train a standard backoff LM with the
n-gram corpus. It's not supported yet or recommended.

I'm not sure whether the moses' SRILM wrapper will support the
count-based LM. And how much memory it will consume. Try it and please
let us know.

People have also been using the Common Crawl corpus to build huge
backoff LM. They're very difficult to use as it consumes a lot of memory

On 25/04/2015 20:24, Alla Rozovskaya wrote:
> Hello,
>
> I have built an interpolated count-based LM on the Google Web N-gram
> corpus using SRILM toolkit, as specified here:
> http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html
>
> Is it possible to use it in moses? In particular, since this model
> uses count files and a file specifying weights, what is the right way
> to specify the path in moses.ini?
>
> Thank you,
>
> Alla
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

--
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150428/62714441/attachment-0001.htm

------------------------------

Message: 2
Date: Tue, 28 Apr 2015 16:13:22 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Adding a language model built on Google
Web
To: moses-support@mit.edu
Message-ID: <553F9582.3000604@amu.edu.pl>
Content-Type: text/plain; charset="windows-1252"

Hi,

W dniu 28.04.2015 o 16:06, Hieu Hoang pisze:
>
> People have also been using the Common Crawl corpus to build huge
> backoff LM. They're very difficult to use as it consumes a lot of memory
>
That's what I added pruning to KenLM for :) Also if you combine that
with some domain-filtering you get nice models form the common crawl
data. You might need a couble of TV of free disk space though.
Best,
Marcin

> On 25/04/2015 20:24, Alla Rozovskaya wrote:
>> Hello,
>>
>> I have built an interpolated count-based LM on the Google Web N-gram
>> corpus using SRILM toolkit, as specified here:
>> http://www.speech.sri.com/projects/srilm/manpages/srilm-faq.7.html
>>
>> Is it possible to use it in moses? In particular, since this model
>> uses count files and a file specifying weights, what is the right way
>> to specify the path in moses.ini?
>>
>> Thank you,
>>
>> Alla
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
> --
> Hieu Hoang
> Researcher
> New York University, Abu Dhabi
> http://www.hoang.co.uk/hieu
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150428/218fd2cb/attachment-0001.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 102, Issue 63
**********************************************

Tweet