Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Format of binarized phrase tables (Barry Haddow)
2. Re: Format of binarized phrase tables (Marcin Junczys-Dowmunt)
3. Re: how to test whether tcmalloc is used? (Rico Sennrich)
4. Re: Format of binarized phrase tables (Raj Dabre)
5. Re: how to test whether tcmalloc is used? (Hieu Hoang)
----------------------------------------------------------------------
Message: 1
Date: Wed, 26 Nov 2014 09:45:48 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Format of binarized phrase tables
To: Raj Dabre <prajdabre@gmail.com>, "moses-support@mit.edu"
<moses-support@mit.edu>
Message-ID: <5475A14C.2080503@staffmail.ed.ac.uk>
Content-Type: text/plain; charset=windows-1252; format=flowed
Hi Raj
The format of these tables is not described anywhere. You'd have to read
the code in moses/TranslationModel/PhraseDictionaryTree.cpp, and then
try to convert it it Java.
A better plan would be to use JNI to call the C++ code -- a similar
approach has been followed in the python interface in contrib/python.
This would insulate you from the low-level details, and from changes in
the format,
cheers - Barry
On 26/11/14 03:22, Raj Dabre wrote:
> Hello All,
>
> I know that Moses allows for binarization of a phrase table which can
> be read on demand at decoding time.
> We get 5 files named: phrase-table.binphr.*
> I want to write my own routine in Java to read phrase pairs from these
> on demand.
> Can anyone guide me ?
>
> PS: If an explanation of the same for binary reordering tables can be
> done then it would be great too.
>
> Thanks in advance.
>
> --
> Raj Dabre.
> Research Student,
> Graduate School of Informatics,
> Kyoto University.
> CSE MTech, IITB., 2011-2014
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
------------------------------
Message: 2
Date: Wed, 26 Nov 2014 10:50:50 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Format of binarized phrase tables
To: moses-support@mit.edu
Message-ID: <f85ba397cee2dc9a30fb696ceebab393@amu.edu.pl>
Content-Type: text/plain; charset="utf-8"
Hi,
I have a JNI interface to my compact phrase table somewhere, I guess I
can put that in contrib within a day or two if there is interest.
best,
Marcin
W dniu 2014-11-26 10:45, Barry Haddow napisa?(a):
> Hi Raj
>
> The format of these tables is not described anywhere. You'd have to read
> the code in moses/TranslationModel/PhraseDictionaryTree.cpp, and then
> try to convert it it Java.
>
> A better plan would be to use JNI to call the C++ code -- a similar
> approach has been followed in the python interface in contrib/python.
> This would insulate you from the low-level details, and from changes in
> the format,
>
> cheers - Barry
>
> On 26/11/14 03:22, Raj Dabre wrote:
>
>> Hello All, I know that Moses allows for binarization of a phrase table which can be read on demand at decoding time. We get 5 files named: phrase-table.binphr.* I want to write my own routine in Java to read phrase pairs from these on demand. Can anyone guide me ? PS: If an explanation of the same for binary reordering tables can be done then it would be great too. Thanks in advance. -- Raj Dabre. Research Student, Graduate School of Informatics, Kyoto University. CSE MTech, IITB., 2011-2014 _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support [1]
Links:
------
[1] http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141126/8ee3c882/attachment-0001.htm
------------------------------
Message: 3
Date: Wed, 26 Nov 2014 10:50:34 +0000 (UTC)
From: Rico Sennrich <rico.sennrich@gmx.ch>
Subject: Re: [Moses-support] how to test whether tcmalloc is used?
To: moses-support@mit.edu
Message-ID: <loom.20141126T114037-634@post.gmane.org>
Content-Type: text/plain; charset=us-ascii
Li Xiang <lixiang.ict@...> writes:
>
> I compile Moses with tcmalloc. How can I test whether tcmalloc is used and
evaluate the performance ?
>
there's probably many ways, but here's three:
at compile time, you will see the following message if tcmalloc is not enabled:
"Tip: install tcmalloc for faster threading. See BUILD-INSTRUCTIONS.txt for
more information."
you can also use '--without-tcmalloc' to disable tcmalloc and compare speed
to a binary that is compiled with tcmalloc.
If you use profiling tools (such as 'perf'), you can see which malloc is
being called. 'perf top' shows me this line, among others:
1.75% moses_chart moses [.]
tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
unsigned long, int
------------------------------
Message: 4
Date: Wed, 26 Nov 2014 20:00:36 +0900
From: Raj Dabre <prajdabre@gmail.com>
Subject: Re: [Moses-support] Format of binarized phrase tables
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAB3gfjC=Uw1W9Bi3YvqcKSNCDZwsxnFZ9LL98jkxvwzkkL63iA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hello Marcin,
Yes please.
It would save me lots of time.
Thanks.
Regards.
On Wed, Nov 26, 2014 at 6:50 PM, Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
wrote:
> Hi,
>
> I have a JNI interface to my compact phrase table somewhere, I guess I can
> put that in contrib within a day or two if there is interest.
>
> best,
>
> Marcin
>
> W dniu 2014-11-26 10:45, Barry Haddow napisa?(a):
>
> Hi Raj
>
> The format of these tables is not described anywhere. You'd have to read
> the code in moses/TranslationModel/PhraseDictionaryTree.cpp, and then
> try to convert it it Java.
>
> A better plan would be to use JNI to call the C++ code -- a similar
> approach has been followed in the python interface in contrib/python.
> This would insulate you from the low-level details, and from changes in
> the format,
>
> cheers - Barry
>
> On 26/11/14 03:22, Raj Dabre wrote:
>
> Hello All, I know that Moses allows for binarization of a phrase table
> which can be read on demand at decoding time. We get 5 files named:
> phrase-table.binphr.* I want to write my own routine in Java to read phrase
> pairs from these on demand. Can anyone guide me ? PS: If an explanation of
> the same for binary reordering tables can be done then it would be great
> too. Thanks in advance. -- Raj Dabre. Research Student, Graduate School
> of Informatics, Kyoto University. CSE MTech, IITB., 2011-2014
> _______________________________________________ Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
--
Raj Dabre.
Research Student,
Graduate School of Informatics,
Kyoto University.
CSE MTech, IITB., 2011-2014
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141126/e94eb972/attachment-0001.htm
------------------------------
Message: 5
Date: Wed, 26 Nov 2014 11:34:12 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] how to test whether tcmalloc is used?
To: Rico Sennrich <rico.sennrich@gmx.ch>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbia_d_s9YP0p3VvN-VDSxzwsjdn-dCRmyfYdG8K9cPsYg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Best to do what Rico says, but....
If the tcmalloc library is dynamically linked to Moses, running ldd will
show it is linked into moses:
#ldd bin/moses
.....
libtcmalloc_minimal.so.4 => /usr/lib/libtcmalloc_minimal.so.4
(0x00007ff49f5a2000)
...
You can force it to statically link by deleting
rm /usr/lib/libtcmalloc*.a
On 26 November 2014 at 10:50, Rico Sennrich <rico.sennrich@gmx.ch> wrote:
> Li Xiang <lixiang.ict@...> writes:
>
> >
> > I compile Moses with tcmalloc. How can I test whether tcmalloc is used
> and
> evaluate the performance ?
> >
>
> there's probably many ways, but here's three:
>
> at compile time, you will see the following message if tcmalloc is not
> enabled:
>
> "Tip: install tcmalloc for faster threading. See BUILD-INSTRUCTIONS.txt
> for
> more information."
>
> you can also use '--without-tcmalloc' to disable tcmalloc and compare speed
> to a binary that is compiled with tcmalloc.
>
> If you use profiling tools (such as 'perf'), you can see which malloc is
> being called. 'perf top' shows me this line, among others:
>
> 1.75% moses_chart moses [.]
>
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
> unsigned long, int
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141126/476fe907/attachment.htm
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 97, Issue 81
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 97, Issue 81"
Post a Comment