Moses-support Digest, Vol 93, Issue 17

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Detokenizer (Judah Schvimer)


----------------------------------------------------------------------

Message: 1
Date: Mon, 14 Jul 2014 11:53:20 -0400
From: Judah Schvimer <judah.schvimer@mongodb.com>
Subject: [Moses-support] Detokenizer
To: moses-support <moses-support@mit.edu>
Message-ID:
<CALF9aB4035pLqT6Ygij6dMtMiMmbZWf3bQwpRdDEKe6oiOoXQA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

When I'm using the decoder I have to tokenize my target sentences before I
translate them. However, when I detokenize them it leaves awkward spaces
around what was tokenized. is there any way to fix this? It seems to be
mainly around slashes and colons

Source: :doc:`/tutorial/aggregation-zip-code-data-set`
Target: : Doc: '/ tutorial / aggregation-zip-code-data-set'

Thanks,
Judah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140714/1fabb77b/attachment-0001.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 93, Issue 17
*********************************************

Tweet