Moses-support Digest, Vol 88, Issue 66

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Moses in server mode (Tom Hoar)
2. Binarising the phrase table (Per Tunedal)
3. Re: Moses in server mode (Sarah Schulz)


----------------------------------------------------------------------

Message: 1
Date: Fri, 28 Feb 2014 07:02:30 +0700
From: Tom Hoar <tahoar@precisiontranslationtools.com>
Subject: Re: [Moses-support] Moses in server mode
To: moses-support@MIT.edu
Message-ID: <530FD216.2020403@precisiontranslationtools.com>
Content-Type: text/plain; charset="iso-8859-1"

XML-RPC limits are specific to both client and server implementations.
We use the Python xmlrpclib library as the xml-rpc proxy in our client
applications.

We researched its limits extensively before committing to the solution.
My Google searches found no overall size limit. It uses http as the
underlying transport protocol, which has no inherent size limits. We
found some implied references to 25 MB, but can't find those references
again.

The xmlrpc-c library documentation
(http://xmlrpc-c.sourceforge.net/doc/libxmlrpc.html#limit mentions a
nesting limit of 64 elements deep and a character-per-element limit is
512K. It looks like the 512K limit is related to the size of the element
tag and attributes, not the text content within the tags. Our testing
consistently handles xml-rpc much larger in the text within the element.
Overall, we have sent hundreds of megabytes to our server (not
mosesserver) in one transaction. So, I'm not even 100% certain that
Python's xmlrpclib uses this particular xmlrpc-c library, but it might
help with troubleshooting.

I'd look at

1. how long is the segment you're sending to mosesserver?
2. why is your source language segment is so long?
3. how much xml-markup is in that segment?
4. how large are the tags and attributes within that markup?



On 02/28/2014 02:17 AM, Hieu Hoang wrote:
> Is the problem reproduceable with a particular sentence, and how long
> is the sentence? Can you reproduce the error with a different sentence
> with the same number of words or characters?
>
> If it's reproduceable, perhaps you can make your files available for
> download and I'll try and debug it
>
>
> On 26 February 2014 08:16, Sarah Schulz <sarah.schulz@ugent.be
> <mailto:sarah.schulz@ugent.be>> wrote:
>
> Hello,
>
> I am running Moses in server mode using the python lib xmlrpclib.
> When I
> send long messages to Moses, it crashes. I guess it has to do with a
> limitation of the message size. I could find a possibility to increase
> the allowed message size for the XML-RPC implementation for C/C++
> (which
> shows me that there is something like a maximum length) but not
> for the
> python implementation.
>
> That is the error I get:
>
> File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
> return self.__send(self.__name, args)
> File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
> verbose=self.__verbose
> File "/usr/lib/python2.7/xmlrpclib.py", line 1264, in request
> return self.single_request(host, handler, request_body, verbose)
> File "/usr/lib/python2.7/xmlrpclib.py", line 1294, in
> single_request
> response = h.getresponse(buffering=True)
> File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
> response.begin()
> File "/usr/lib/python2.7/httplib.py", line 407, in begin
> version, status, reason = self._read_status()
> File "/usr/lib/python2.7/httplib.py", line 371, in _read_status
> raise BadStatusLine(line)
> httplib.BadStatusLine: ''
>
>
> Does anybody have experience with that?
>
> Cheers,
>
> Sarah
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140228/cd13ffff/attachment-0001.htm

------------------------------

Message: 2
Date: Fri, 28 Feb 2014 07:47:15 +0100
From: Per Tunedal <per.tunedal@operamail.com>
Subject: [Moses-support] Binarising the phrase table
To: moses-support@mit.edu
Message-ID:
<1393570035.773.88831217.7AC5AC71@webmail.messagingengine.com>
Content-Type: text/plain


Hi,
tried to binarise the phrase table and got in to trouble.

1. Error messages, as below. What's that?
distinct source phrases: 439511 distinct first words of source phrases:
67600 number of phrase pairs (line count): 940438
Count of lines with missing alignments: 0/940438
WARNING: there are src voc entries with no phrase translation: count
2168
There exists phrase translations for 65432 entries

2. Modify the moses.ini file. I've found this on the Moses/Baseline
page:
1. Change PhraseDictionaryMemory to PhraseDictionaryBinary
2. Set the path of the PhraseDictionary feature to point to
$HOME/working/train/binarised-model/Kryptering1.sv-fr.phrase-table
3. Set the path of the LexicalReordering feature to point to
$HOME/working/train/binarised-model/Kryptering1.sv-fr.reordering-table

But I cannot find any such entries in my moses.ini - maybe because I'm
running a somewhat older version of Moses. I've found e.g. the following
lines in my ini-file:

[ttable-file]
0 0 0 5 /home/per/working/train/model/phrase-table.gz

# distortion (reordering) files
[distortion-file]
0-0 wbe-msd-bidirectional-fe-allff 6
/home/per/working/train/model/reordering-table.wbe-msd-bidirectional-fe.gz

How should I change the entries to use my binarised model?

Yours,
Per Tunedal



------------------------------

Message: 3
Date: Fri, 28 Feb 2014 08:54:23 +0100
From: Sarah Schulz <sarah.schulz@ugent.be>
Subject: Re: [Moses-support] Moses in server mode
To: moses-support@mit.edu
Message-ID: <531040AF.9070207@ugent.be>
Content-Type: text/plain; charset="iso-8859-1"

Thanks for the answers.

I had a look at the python library files yesterday and there is indeed a
limit hard coded there. It's 10MB (which is already quite high). The
comment there says that this is due to the inability of some OS to deal
with larger chunks.
A solution here would be to set it higher and recompile the library.
I asked the developer of this library to add the possibility to increase
this maximum size in case you really need it.
However, I found out that the chunks I send are indeed this large by
mistake. So my problem is solved anyways.

Cheers,

Sarah

On 02/28/2014 01:02 AM, Tom Hoar wrote:
> XML-RPC limits are specific to both client and server implementations.
> We use the Python xmlrpclib library as the xml-rpc proxy in our client
> applications.
>
> We researched its limits extensively before committing to the
> solution. My Google searches found no overall size limit. It uses http
> as the underlying transport protocol, which has no inherent size
> limits. We found some implied references to 25 MB, but can't find
> those references again.
>
> The xmlrpc-c library documentation
> (http://xmlrpc-c.sourceforge.net/doc/libxmlrpc.html#limit mentions a
> nesting limit of 64 elements deep and a character-per-element limit is
> 512K. It looks like the 512K limit is related to the size of the
> element tag and attributes, not the text content within the tags. Our
> testing consistently handles xml-rpc much larger in the text within
> the element. Overall, we have sent hundreds of megabytes to our server
> (not mosesserver) in one transaction. So, I'm not even 100% certain
> that Python's xmlrpclib uses this particular xmlrpc-c library, but it
> might help with troubleshooting.
>
> I'd look at
>
> 1. how long is the segment you're sending to mosesserver?
> 2. why is your source language segment is so long?
> 3. how much xml-markup is in that segment?
> 4. how large are the tags and attributes within that markup?
>
>
>
> On 02/28/2014 02:17 AM, Hieu Hoang wrote:
>> Is the problem reproduceable with a particular sentence, and how long
>> is the sentence? Can you reproduce the error with a different
>> sentence with the same number of words or characters?
>>
>> If it's reproduceable, perhaps you can make your files available for
>> download and I'll try and debug it
>>
>>
>> On 26 February 2014 08:16, Sarah Schulz <sarah.schulz@ugent.be
>> <mailto:sarah.schulz@ugent.be>> wrote:
>>
>> Hello,
>>
>> I am running Moses in server mode using the python lib xmlrpclib.
>> When I
>> send long messages to Moses, it crashes. I guess it has to do with a
>> limitation of the message size. I could find a possibility to
>> increase
>> the allowed message size for the XML-RPC implementation for C/C++
>> (which
>> shows me that there is something like a maximum length) but not
>> for the
>> python implementation.
>>
>> That is the error I get:
>>
>> File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
>> return self.__send(self.__name, args)
>> File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
>> verbose=self.__verbose
>> File "/usr/lib/python2.7/xmlrpclib.py", line 1264, in request
>> return self.single_request(host, handler, request_body, verbose)
>> File "/usr/lib/python2.7/xmlrpclib.py", line 1294, in
>> single_request
>> response = h.getresponse(buffering=True)
>> File "/usr/lib/python2.7/httplib.py", line 1030, in getresponse
>> response.begin()
>> File "/usr/lib/python2.7/httplib.py", line 407, in begin
>> version, status, reason = self._read_status()
>> File "/usr/lib/python2.7/httplib.py", line 371, in _read_status
>> raise BadStatusLine(line)
>> httplib.BadStatusLine: ''
>>
>>
>> Does anybody have experience with that?
>>
>> Cheers,
>>
>> Sarah
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>>
>> --
>> Hieu Hoang
>> Research Associate
>> University of Edinburgh
>> http://www.hoang.co.uk/hieu
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140228/5a7c041f/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 88, Issue 66
*********************************************

0 Response to "Moses-support Digest, Vol 88, Issue 66"

Post a Comment