Moses-support Digest, Vol 139, Issue 10

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Regarding cleaning corpus (Aqlan fares)


----------------------------------------------------------------------

Message: 1
Date: Fri, 18 May 2018 02:31:09 +0800
From: Aqlan fares <alforsan930@gmail.com>
Subject: [Moses-support] Regarding cleaning corpus
To: moses-support@mit.edu
Message-ID:
<CAFd8pYoQXaObKDKj-w2aRN4r2s3qS_ZnR4Djm90PgZpOdMJHCA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,
I am working on Arabic MT, using different tokenization schemes.
The different schemes result in different line lengths, which might cause
imbalances among the different options when I clean corpus, to eliminate
the lines beyond the length of 85 words.
In order to avoid this imbalance, let's say that I have 4 scheme ( A, B,
C,D), and I need to eliminate the lines across all files whose B scheme
exceeds 85 words.
How can I do that using clean-corpus-n.perl ?

Thanks for any help you may offer.

Best Regards.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20180517/00da8e8a/attachment-0001.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 139, Issue 10
**********************************************

Tweet