Moses-support Digest, Vol 137, Issue 10

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. moses tokenizer (Jigyasa Sakhuja)
2. Re: moses tokenizer (Debajyoty Banik)


----------------------------------------------------------------------

Message: 1
Date: Fri, 23 Mar 2018 18:49:31 +0100
From: Jigyasa Sakhuja <jigyasasakhuja@icloud.com>
Subject: [Moses-support] moses tokenizer
To: moses-support@mit.edu
Message-ID: <0A355CEE-F77F-404F-9508-E14851D76C4F@icloud.com>
Content-Type: text/plain; charset=us-ascii

hi i want to tokenize my data and i want to use the tokenizer.perl file but i dont know how to use that tokenizer for tokenization . can you help me ?


------------------------------

Message: 2
Date: Fri, 23 Mar 2018 23:59:12 +0530
From: Debajyoty Banik <debajyoty.banik@gmail.com>
Subject: Re: [Moses-support] moses tokenizer
To: Jigyasa Sakhuja <jigyasasakhuja@icloud.com>
Cc: moses-support@mit.edu
Message-ID:
<CAK0HAkgVS83Zo8-=e20rXZ5f3E-dF_AUd5aaBQZOS5EyZo3npQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

You can use the following command to use tokenizer which exists inside
of mosesdecoder
folder. Lets your input file (corpora.en) exists inside the "data".folder
and the generated tokenized output file name is corpora.tok.en.

~/mosesdecoder/scripts/tokenizer/tokenizer.perl -l en <~/data/corpora.en
>~/data/corpora.tok.en




On Fri, Mar 23, 2018 at 11:19 PM, Jigyasa Sakhuja <jigyasasakhuja@icloud.com
> wrote:

> hi i want to tokenize my data and i want to use the tokenizer.perl file
> but i dont know how to use that tokenizer for tokenization . can you help
> me ?
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--

Thanks and Regards

Debajyoty Banik
IIT Patna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20180323/b3ab0860/attachment-0001.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 137, Issue 10
**********************************************

Tweet