Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Format of binarized phrase tables (Marcin Junczys-Dowmunt)
2. Re: Format of binarized phrase tables (Raj Dabre)
----------------------------------------------------------------------
Message: 1
Date: Thu, 27 Nov 2014 12:03:14 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Format of binarized phrase tables
To: Raj Dabre <prajdabre@gmail.com>
Cc: moses-support@mit.edu
Message-ID: <606ede123bf472c4ac4c313354d788ea@amu.edu.pl>
Content-Type: text/plain; charset="utf-8"
The code in the wipo branch is not segfaulting, it's just an old moses
version. I can put my recent attempts to make it run with current master
into a separate branch (that's segfaulting). It's best if you start
there when you try to fix it. I will let you know once I pushed it.
Probably this evening.
W dniu 2014-11-27 11:57, Raj Dabre napisa?(a):
> Hello Marcin,
>
> You just saved me a lot of time since I was planning to write this code from scratch. Many thanks for that. I will try to fix the reasons for the segfaults.
>
> Many thanks again!
>
> Regards.
>
> On Thu, Nov 27, 2014 at 7:43 PM, Marcin Junczys-Dowmunt <junczys@amu.edu.pl> wrote:
>
> Hi,
>
> I tried to port it to the newest Moses version, but I still get segfaults. This used to work with Moses 1.* before the feature function format was changed. However, since the binary format of the compact phrase table has not changed since then, just use the old interface.
>
> Checkout the "wipo" branch (not wipoNew!) and compile with
>
> ./bjam --with-cmph=/usr/include -j8 --with-java=/usr/lib/jvm/java-7-openjdk-amd64
>
> You see you need to specifiy the location of your Java includes, specifically the directories jni.h resides in. This should build the stuff in misc/jni/* including java classes and stuff. You need ant for that. Then you can run the query tool like this:
>
> echo "test" | ./bin/JniQueryPt_example.sh /some_path/phrase-table.0-0.minphr 4
>
> You need to specify the path and the number of scores in the phrase table. Look inside JniQueryPt_example.sh how to call the jar and inside misc/jni/java/example.java how to call the code directly from Java. I apologize for the code, I am not a Java programmer, so this may be crude.
>
> Best,
>
> Marcin
>
> W dniu 2014-11-26 12:00, Raj Dabre napisa?(a):
>
> Hello Marcin,
>
> Yes please. It would save me lots of time. Thanks.
>
> Regards.
>
> On Wed, Nov 26, 2014 at 6:50 PM, Marcin Junczys-Dowmunt <junczys@amu.edu.pl> wrote:
>
> Hi,
>
> I have a JNI interface to my compact phrase table somewhere, I guess I can put that in contrib within a day or two if there is interest.
>
> best,
>
> Marcin
>
> W dniu 2014-11-26 10:45, Barry Haddow napisa?(a):
>
> Hi Raj
>
> The format of these tables is not described anywhere. You'd have to read
> the code in moses/TranslationModel/PhraseDictionaryTree.cpp, and then
> try to convert it it Java.
>
> A better plan would be to use JNI to call the C++ code -- a similar
> approach has been followed in the python interface in contrib/python.
> This would insulate you from the low-level details, and from changes in
> the format,
>
> cheers - Barry
>
> On 26/11/14 03:22, Raj Dabre wrote:
> Hello All, I know that Moses allows for binarization of a phrase table which can be read on demand at decoding time. We get 5 files named: phrase-table.binphr.* I want to write my own routine in Java to read phrase pairs from these on demand. Can anyone guide me ? PS: If an explanation of the same for binary reordering tables can be done then it would be great too. Thanks in advance. -- Raj Dabre. Research Student, Graduate School of Informatics, Kyoto University. CSE MTech, IITB., 2011-2014 _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support [1]
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support [1]
--
Raj Dabre.
Research Student, Graduate School of Informatics,
Kyoto University.
CSE MTech, IITB., 2011-2014
--
Raj Dabre.
Research Student, Graduate School of Informatics,
Kyoto University.
CSE MTech, IITB., 2011-2014
Links:
------
[1] http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141127/949135fe/attachment-0001.htm
------------------------------
Message: 2
Date: Thu, 27 Nov 2014 21:43:39 +0900
From: Raj Dabre <prajdabre@gmail.com>
Subject: Re: [Moses-support] Format of binarized phrase tables
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAB3gfjBmq5V3Anou59JJjXKCqs1m=472eGXfTCU3gHasdF81Dw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Marcin,
I just finished installing everything and the code works like a charm. I
did have to modify the LD_LIBRARY_PATH since libcmph.so.0 did not get
linked to libJniQueryPt.so in the lib folder. But thats a small thing.
I will study the code in detail and try to make it work for querying the
reordering models also (Unless that is also taken care of???).
If you have anything new you wanted to add to this then please let me know.
Some experience in JNI would help a Java programmer like me play with the
internals of moses.
Thanks again.
On Thu, Nov 27, 2014 at 8:03 PM, Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
wrote:
> The code in the wipo branch is not segfaulting, it's just an old moses
> version. I can put my recent attempts to make it run with current master
> into a separate branch (that's segfaulting). It's best if you start there
> when you try to fix it. I will let you know once I pushed it. Probably this
> evening.
>
> W dniu 2014-11-27 11:57, Raj Dabre napisa?(a):
>
> Hello Marcin,
>
> You just saved me a lot of time since I was planning to write this code
> from scratch. Many thanks for that.
> I will try to fix the reasons for the segfaults.
>
> Many thanks again!
>
> Regards.
>
> On Thu, Nov 27, 2014 at 7:43 PM, Marcin Junczys-Dowmunt <
> junczys@amu.edu.pl> wrote:
>
>> Hi,
>>
>> I tried to port it to the newest Moses version, but I still get
>> segfaults. This used to work with Moses 1.* before the feature function
>> format was changed. However, since the binary format of the compact phrase
>> table has not changed since then, just use the old interface.
>>
>> Checkout the "wipo" branch (not wipoNew!) and compile with
>>
>> ./bjam --with-cmph=/usr/include -j8
>> --with-java=/usr/lib/jvm/java-7-openjdk-amd64
>>
>> You see you need to specifiy the location of your Java includes,
>> specifically the directories jni.h resides in. This should build the stuff
>> in misc/jni/* including java classes and stuff. You need ant for that. Then
>> you can run the query tool like this:
>>
>> echo "test" | ./bin/JniQueryPt_example.sh
>> /some_path/phrase-table.0-0.minphr 4
>>
>> You need to specify the path and the number of scores in the phrase
>> table. Look inside JniQueryPt_example.sh how to call the jar and inside
>> misc/jni/java/example.java how to call the code directly from Java. I
>> apologize for the code, I am not a Java programmer, so this may be crude.
>>
>> Best,
>>
>> Marcin
>>
>> W dniu 2014-11-26 12:00, Raj Dabre napisa?(a):
>>
>> Hello Marcin,
>>
>> Yes please.
>> It would save me lots of time.
>> Thanks.
>>
>> Regards.
>>
>> On Wed, Nov 26, 2014 at 6:50 PM, Marcin Junczys-Dowmunt <
>> junczys@amu.edu.pl> wrote:
>>
>>> Hi,
>>>
>>> I have a JNI interface to my compact phrase table somewhere, I guess I
>>> can put that in contrib within a day or two if there is interest.
>>>
>>> best,
>>>
>>> Marcin
>>>
>>> W dniu 2014-11-26 10:45, Barry Haddow napisa?(a):
>>>
>>> Hi Raj
>>>
>>> The format of these tables is not described anywhere. You'd have to read
>>> the code in moses/TranslationModel/PhraseDictionaryTree.cpp, and then
>>> try to convert it it Java.
>>>
>>> A better plan would be to use JNI to call the C++ code -- a similar
>>> approach has been followed in the python interface in contrib/python.
>>> This would insulate you from the low-level details, and from changes in
>>> the format,
>>>
>>> cheers - Barry
>>>
>>> On 26/11/14 03:22, Raj Dabre wrote:
>>>
>>> Hello All, I know that Moses allows for binarization of a phrase table
>>> which can be read on demand at decoding time. We get 5 files named:
>>> phrase-table.binphr.* I want to write my own routine in Java to read phrase
>>> pairs from these on demand. Can anyone guide me ? PS: If an explanation of
>>> the same for binary reordering tables can be done then it would be great
>>> too. Thanks in advance. -- Raj Dabre. Research Student, Graduate School
>>> of Informatics, Kyoto University. CSE MTech, IITB., 2011-2014
>>> _______________________________________________ Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>>
>> --
>> Raj Dabre.
>> Research Student,
>> Graduate School of Informatics,
>> Kyoto University.
>> CSE MTech, IITB., 2011-2014
>>
>>
>>
>>
>
>
>
> --
> Raj Dabre.
> Research Student,
> Graduate School of Informatics,
> Kyoto University.
> CSE MTech, IITB., 2011-2014
>
>
>
>
--
Raj Dabre.
Research Student,
Graduate School of Informatics,
Kyoto University.
CSE MTech, IITB., 2011-2014
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141127/c24441cc/attachment.htm
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 97, Issue 92
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 97, Issue 92"
Post a Comment