Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Is there multithread option for KenLM's build_binary?
(liling tan)
2. Please remove me from the mailing list (Cook Richard)
3. Re: Please remove me from the mailing list (Hieu Hoang)
4. File not found in Step 5 (extract phrases) (Kun Wang)
----------------------------------------------------------------------
Message: 1
Date: Thu, 13 Aug 2015 05:08:17 +0200
From: liling tan <alvations@gmail.com>
Subject: Re: [Moses-support] Is there multithread option for KenLM's
build_binary?
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAKzPaJKM3wObCLcUz1pZBWo=U7tv0BFA0gtzCdBiZ371jQwStA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Dear Hieu,
Thanks for the info on the KenLM,
Regards,
Liling
On Tue, Aug 11, 2015 at 5:57 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
>
> On 10/08/2015 17:22, liling tan wrote:
>
> Dear Moses devs/users,
>
> @Marcin @Ken , Thanks for the tips on the -S for build_binary, RAM
> estimation and the probing vs trie explanations.
>
> Just to do a check, currently, is there an option for lmplz to output
> binarized directly without going through ARPA? If there is, is there also a
> binary to arpa dumping mechanism?
>
> as far as i know, neither of these options are available in the current
> version of kenlm
>
>
> Regards,
> LIling
>
>
>
>
> On Fri, Aug 7, 2015 at 9:31 PM, liling tan <alvations@gmail.com> wrote:
>
>> Dear Moses dev/users,
>>
>> On a related note, without multi-threads, can anyone give a gauge of how
>> much RAM is required to binarized a 80GB (compressed .gz) 6gram arpa file?
>> The no. of ngrams are:
>>
>> \data\
>> ngram 1=7503209
>> ngram 2=131003943
>> ngram 3=671005861
>> ngram 4=1510529519
>> ngram 5=2165163610
>> ngram 6=2477533666
>>
>>
>> Also, how long would it take (single-threadedly) on a 2.4Ghz core with
>> 128GB RAM? Is there a way to mathematically estimate the time taken and RAM
>> required to binarize a language model?
>>
>> Also, is binarized and quantized LM from KenLM lossy? If so how lossy?
>> The KenLM paper states "To conserve memory at the expense of accuracy,
>> values may be quantized using q bits per probability and r bits per
>> backoff". Can someone help point us to papers that quanitfy how lossy it
>> gets in terms of MT experiments or word perplexity task?
>>
>> Thanks in advance for the pointers!
>>
>> Regards,
>> Liling
>>
>> On Fri, Aug 7, 2015 at 8:56 PM, liling tan < <alvations@gmail.com>
>> alvations@gmail.com> wrote:
>>
>>> Dear Moses dev/users,
>>>
>>> Is there multithread option for KenLM's build_binary?
>>>
>>> Regards,
>>> Liling
>>>
>>
>>
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> --
> Hieu Hoang
> Researcher
> New York University, Abu Dhabihttp://www.hoang.co.uk/hieu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150813/764e1161/attachment-0001.htm
------------------------------
Message: 2
Date: Thu, 13 Aug 2015 09:18:35 +0000
From: Cook Richard <RichardCook@mt-g.com>
Subject: [Moses-support] Please remove me from the mailing list
To: "'moses-support@mit.edu'" <moses-support@mit.edu>
Message-ID:
<8E3B44A63D3EE840BF0D38A84ACD06212803455B@MTGSRV11.mt-g.local>
Content-Type: text/plain; charset="us-ascii"
Thank you
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150813/cfd4c902/attachment-0001.htm
------------------------------
Message: 3
Date: Thu, 13 Aug 2015 13:56:33 +0400
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Please remove me from the mailing list
To: Cook Richard <RichardCook@mt-g.com>, "'moses-support@mit.edu'"
<moses-support@mit.edu>
Message-ID: <55CC69D1.2010305@gmail.com>
Content-Type: text/plain; charset="windows-1252"
ok.
however, you subscribed yourself so you should have been able to remove
yourself.
If you're still getting emails, email the admins rather than spam
everyone on the list. You can find the name at the bottom of the
subscribing page
http://mailman.mit.edu/mailman/listinfo/moses-support
ie. hieuhoang at gmail.com, phkoehn at users.sourceforge.net, swadey at
mit.edu <mailto:moses-support-owner@mit.edu>
On 13/08/2015 13:18, Cook Richard wrote:
>
> Thank you
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
--
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150813/f684c997/attachment-0001.htm
------------------------------
Message: 4
Date: Thu, 13 Aug 2015 20:55:28 +0800
From: "Kun Wang" <kunwang@nlpr.ia.ac.cn>
Subject: [Moses-support] File not found in Step 5 (extract phrases)
To: moses-support <moses-support@mit.edu>
Message-ID: <2015081320552686987510@nlpr.ia.ac.cn>
Content-Type: text/plain; charset="gb2312"
Dear all,
Does anyone know why moses cannot find the files in step 5?
Enclosed please find the log and train.sh. Thank you very much in advance.
I am sure there are enough disk space.
(5) extract phrases @ Thu Aug 13 20:45:54 CST 2015
/home/kwang/kw2T/decoder/mosesdecoder/scripts/generic/extract-parallel.perl 32 split "sort " /home/kwang/kw2T/decoder/mosesdecoder/scripts/../bin/extract-rules /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/corpus/fbis.eng /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/corpus/fbis.chn /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/aligned.grow-diag-final-and /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/extract --GlueGrammar /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/glue-grammar --MaxSpan 10 --GZOutput
Executing: /home/kwang/kw2T/decoder/mosesdecoder/scripts/generic/extract-parallel.perl 32 split "sort " /home/kwang/kw2T/decoder/mosesdecoder/scripts/../bin/extract-rules /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/corpus/fbis.eng /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/corpus/fbis.chn /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/aligned.grow-diag-final-and /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/extract --GlueGrammar /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/glue-grammar --MaxSpan 10 --GZOutput
Started Thu Aug 13 20:45:54 2015
using gzip
USAGE: split <training-set> <num-shards> <shard-stem>
isBSDSplit=1
Executing: mkdir -p /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516; ls -l /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516
total=1000 line-per-split=32
split -l 32 -a 7 /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/corpus/fbis.eng /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/target.split -l 32 -a 7 /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/corpus/fbis.chn /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/source.split -l 32 -a 7 /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/aligned.grow-diag-final-and /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/align.USAGE: split <training-set> <num-shards> <shard-stem>
USAGE: split <training-set> <num-shards> <shard-stem>
USAGE: split <training-set> <num-shards> <shard-stem>
merging extract / extract.inv
gunzip -c /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaaa.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaab.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaac.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaad.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaae.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaaf.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaag.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaah.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaai.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaaj.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaak.inv.gz !
/home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaal.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaam.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaan.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaao.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaap.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaaq.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaar.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaas.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaat.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaau.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaav.inv.gz /home/kwa!
ng/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaa
aw.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaax.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaay.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaaz.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaba.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaabb.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaabc.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaabd.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaabe.inv.gz /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaabf.inv.gz | LC_ALL=C sort -T /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516 2>> /dev/stderr | gzip -c > /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/ext!
ract.inv.sorted.gz 2>> /dev/stderr
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaaa.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaab.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaac.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaad.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaae.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaaf.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaag.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaah.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaai.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaaj.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaak.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaal.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaam.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaan.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaao.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaap.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaaq.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaar.gz: No such file or directory
gzip: /home/kwang/kw2T/smt-work-dir/moses-hiero-fbis-dir/model/tmp.2516/extract.aaaaaas.gz: No such file or directory
Best Regards,
Kun Wang
2015-08-13
=========================================================
???? WANG Kun
National Laboratory of Pattern Recognition (NLPR)
Institute of Automation, Chinese Academy of Sciences
Beijing, China
Tel??8610-82544588
Email: kunwang@nlpr.ia.ac.cn
=========================================================
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150813/ab793148/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: train.log
Type: application/octet-stream
Size: 90957 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20150813/ab793148/attachment.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: train.sh
Type: application/octet-stream
Size: 829 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20150813/ab793148/attachment-0001.obj
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 106, Issue 28
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 106, Issue 28"
Post a Comment