Moses-support Digest, Vol 127, Issue 45

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Chinese tokenizer / detokenizer (segmenter / unsegmenter)
(Dingyuan Wang)
2. Re: Request for help w/ "The build failed." (Tom Hoar)


----------------------------------------------------------------------

Message: 1
Date: Wed, 31 May 2017 14:40:53 +0800
From: Dingyuan Wang <abcdoyle888@gmail.com>
Subject: Re: [Moses-support] Chinese tokenizer / detokenizer
(segmenter / unsegmenter)
To: Vincent Nguyen <vnguyen@neuf.fr>, moses-support
<moses-support@mit.edu>
Message-ID: <e7a539bb-b1e4-ba5d-812b-8c40dd2429fb@gmail.com>
Content-Type: text/plain; charset=utf-8

Hi,

I personally use the jieba tokenizer (https://github.com/fxsjy/jieba).
Install the python package and use `python -mjieba -d ' '`.

For detokenizer, I wrote my own script
(https://github.com/The-Orizon/nlputils/blob/master/detokenizer.py).
Install `pangu` python package, and use `python3 detokenizer.py`. The
idea is to remove spaces in CJK/fullwidth characters using regex.

The above can't deal with numbers, abbreviations with dots, n't etc. though.

2017-05-30 02:31, Vincent Nguyen:
> Hello team,
>
> I have read many post and it looks like most people tend to use the
> Stanford segmenter.
>
> Do you have some good experience with other tools ?
>
> Also, what "detokenizer" do you actually use. It seems, that it is not
> just a question of removing space, especially when Chinese target
> contains some non chinese words / symbols.
>
> Thanks for your insight,
>
> Vincent.
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>

--
Dingyuan Wang



------------------------------

Message: 2
Date: Wed, 31 May 2017 19:36:14 +0700
From: Tom Hoar <tahoar@pttools.net>
Subject: Re: [Moses-support] Request for help w/ "The build failed."
To: moses-support@mit.edu
Message-ID: <90b855c2-a9a6-32a1-1acd-faafb2336627@pttools.net>
Content-Type: text/plain; charset="utf-8"

Thanks, Ken, for mentioning us.

Hi Chaz. I don't normally talk about Slate Desktop on the Moses list but
I'll fill Ken's opening.

We first launched a Windows-based Moses toolkit (called Slate) about 2
years ago. I took toolkit off the website a while back, and I'm about to
re-post it for download. The new toolkit will include an installer
includes the necessary Perl and Python runtimes, supporting GNU Windows
utilities (sort/gzip, etc), post-Release 3 Moses binaries and
Perl/Python scripts updated for cross-platform operations.

To make our Windows binaries, we did not use the bjam tools in the open
source GIT repository. Instead, we made our own proprietary Makefiles
that cross-build the Windows binaries on an Ubuntu host. As they are
proprietary, we do not distribute the Makefiles or instructions to
create our build environment. However, we checked-in all of our changes
to Moses source code to trunk and I can share the commit ID's with
anyone who's interested. Please contact me privately if you're
interested in licensing our proprietary Makefiles and instructions. In
the next week or so, I'll re-post our packaged Moses toolkit on our
website's e-commerce store. The binaries and scripts in this package are
licensed under their respective open source licenses. You'll need to
build your own Windows-based applications, an arduous task at best.
That's where Slate Desktop comes in.

Our Slate Desktop product includes the Moses toolkit distribution. It
also includes our proprietary tools for corpus management, corpus
preparation, engine management and connectors to popular CAT tools
(memoQ, Trados Studio, OmegaT, and CafeTran Espresso). Our EULA and
one-time perpetual license fee govern the use of these proprietary
tools. We showcase them in action here:
http://pttools.net/portfolio/showcase/

So, if you're looking for free Windows software, Slate and Slate Desktop
are not for you. You'll need to follow Ken's advice for "pure
contributor-based (support) where you'd be expected to submit patches."
If you're more interested in immediate productivity that gets you up and
running in less than 10 minutes, Slate Desktop is the answer. It's an
off-the-shelf Windows application at a reasonable fee with excellent
technical support. Judging by your Moses list support requests, you've
already spent more time on the of your engineering effort at a cost of
more than the price of a Slate Desktop license.

Regards,
Tom



On 5/30/2017 11:00 PM, moses-support-request@mit.edu wrote:
> Date: Tue, 30 May 2017 12:12:08 +0100
> From: Kenneth Heafield<moses@kheafield.com>
> Subject: Re: [Moses-support] Request for help w/ "The build failed."
> To: daideqi<daideqi@yahoo.com>,moses-support@mit.edu, Hieu Hoang
> <hieuhoang@gmail.com>
>
> Cygwin is even worse. Windows support is either a commercial product
> (http://pttools.net/) or pure contributor-based where you'd be expected
> to submit patches.
>
> On 05/30/2017 12:03 PM, daideqi wrote:
>> Hi Kenneth,
>>
>> Thanks! I must confess that I fought and fought over the past 3-day weekend trying to get Moses and Moses for Mere Mortals to work on my Windows 10 Ubuntu install. It was so painful and overheated my box too (a dell inspirion i7-6700 16GB RAM and running Windows 10). And nothing worked! Try as I might, something always failed at the end.
>>
>> So, it's back to the drawing board for me. I'm thinking of installing Cygwin and trying Moses on that. My *nix-foo isn't the best, but I'm not a novice. Trying to get Moses to work has been surprisingly difficult, at least for me. Back to work...
>>
>> Wax on...wax off...
>>
>> All the Best,
>> Chaz
>> --------------------------------------------
>> On Mon, 5/29/17, Kenneth Heafield<moses@kheafield.com> wrote:
>>
>> Subject: Re: [Moses-support] Request for help w/ "The build failed."
>> To:moses-support@mit.edu, "Hieu Hoang"<hieuhoang@gmail.com>, "daideqi"<daideqi@yahoo.com>
>> Date: Monday, May 29, 2017, 3:08 AM
>>
>> A symlink
>> for CreateProbingPT2 has nothing to do with KenLM. The
>> symlink already exists and the build system is trying to
>> make it again (this also means not windows). I suppose we
>> should be using ln -sf.
>>
>>
>>
>> Try deleting CreateProbingPT2 then rebuilding.
>>
>>
>>
>> Kenneth
>>
>> On May 29, 2017 1:04:25 AM
>> GMT+01:00, Hieu Hoang<hieuhoang@gmail.com>
>> wrote:
>> are you using Windows 10, or Ubuntu
>> 14? It seems you are using a Windows file system and this
>> is causing some unit tests in KenLM to fail.
>>
>> I would recommend
>> you do everything on Linux
>>
>> * Looking for MT/NLP opportunities *
>> Hieu Hoang
>> http://moses-smt.org/
>>
>>
>>
>> On 28
>> May 2017 at 00:50, daideqi<daideqi@yahoo.com>
>> wrote:
>> Hi,
>>
>>
>>
>> Respectfully request help with Moses. I'm trying to
>> compile it on Windows 10 Ubuntu 14.04.5 LTS. I successfully
>> built Moses yesterday but decided to delete the mosesdecoder
>> folder and rebuild for practice. Unfortunately, it's
>> not working anymore, and I'm following the instructions
>> that the system gave me. As an aside, I got this error
>> after running the recommended bjam command:
>>
>> ln: failed to create
>> symbolic link ?CreateProbingPT2?: File exists
>>
>>
>>
>> Would be grateful for assistance that can be provided.
>>
>>
>>
>> Very Respectfully,
>>
>> Chaz
>> ______________________________
>> _________________
>>
>> Moses-support mailing list
>>
>> Moses-support@mit.edu
>>
>> http://mailman.mit.edu/
>> mailman/listinfo/moses-support
>>
>>
>>
>>
>>
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170531/48f63e3e/attachment-0001.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 127, Issue 45
**********************************************

Tweet