Moses-support Digest, Vol 139, Issue 6

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Install moses on Windows (Tom Hoar)
2. Re: Install moses on Windows (Mohamed Amine MENACER)


----------------------------------------------------------------------

Message: 1
Date: Thu, 10 May 2018 13:01:41 +0700
From: Tom Hoar <tahoar@precisiontranslationtools.com>
Subject: Re: [Moses-support] Install moses on Windows
To: moses-support@mit.edu
Message-ID:
<7b37a573-0cf7-b36d-61be-841f97ef5a4e@precisiontranslationtools.com>
Content-Type: text/plain; charset="utf-8"

Hi everyone. Sorry I joined the conversation a bit late.

Here are all the options for Moses on Windows that I've tried:

*Virtual machine* - VMs are more efficient than they used to be. Any
good hypervisor will work. You need to build your own tools to create a
sockets layer to communicate from the host machine to the VM.

*Moses with Cygwin* - I never got it to work. There are too many
variables and the underlying open source components change drastically.
It could be one person's full-time job to track keep Moses updated for
those build tools.

*Moses with Visual Studio* - It's a good option if you're looking for
only the moses.exe runtime binary. This does not address the serious
challenges of updating all the other C++, Perl and Python components to
run on Windows. Nor does it address collecting and installing all the
GNU utilities.

*Windows Subsystem for Linux (WSL)* - WSL is native Linux running on the
Windows. Microsoft has done a great job of mapping the Linux kernel
calls to the Windows kernel. There's no virtual machine. It's very much
like running Windows applications on Linux using WINE... but without the
complexities of re-mapping Microsoft's GUI APIs.

From Moses' perspective, WSL is nothing special. It sees the
environment as any native Linux. You can compile Linux binaries on Linux
and copy them to your Windows 10 machine and run them with WSL. You will
need to verify/install all of the other dependencies because WSL
distributions are sparse at best. For example, the Ubuntu 16.04 distro
for WSL does not install Python 2.7. The `python` command will fail. It
installs Python 3.5 by default with `python3` as the executable name.
This will cause Moses to fail. You'll need to do a lot of troubleshooting.

WSL also falls short when it comes to inter-process communications
between WSL's native Linux apps and native Windows apps. I found one
Microsoft instruction that WSL and Windows apps share the network
sockets stack, and that is the only inter-process communication between
the two layers. So, STDIN/STDOUT/STDERR are not available between native
Windows apps and Linux apps on WSL. This means from a practical
standpoint, you're back to working with WSL as though it's a VM. That
is, you can build a sockets layer to communicate between the two. Maybe
you can use the moses binary in Moses Server mode and use XML-RPC, but I
never tried it.

There's another awkward inter-process strategy. If you're in a Windows
cmd.exe terminal or PowerShell terminal, you can execute a command line
that will launch a Linux app in the WSL space. (and vice-versa. From a
native Linux Bash terminal, you can launch a native Windows .exe, but
that's irrelevant to Moses)? So, you could have your Windows app save a
text file to the file system, then execute the Moses Linux binary with
the -i argument pointing to the input file. The challenge here is
translating the Windows file system "C:\Users\...." to the WSL file
system "/mnt/c/Users/...". You'll also have to figure out a way to pass
the STDOUT redirection to the moses binary to save it's output. I don't
know if this is possible. Then, your Windows application that spawned
the Linux Moses binary will need to detect when the binary is done, and
then retrieve the output file. WARNING: do not directly access the Linux
file system tree from Windows
[%LOCALAPPDATA%\Packages\CanonicalGroupLimited.UbuntuonWindows_xxxxxxxxxxxxx\LocalState\rootfs].
You are virtually guaranteed to corrupt the files. Only have Windows
apps save to it's native C:\ drives and then access them from Linux
using the mount director tree.

Bottom line... you should read what Microsoft's engineers have written
about why they created WSL. It is not intended to be a production tool.
It is designed as a development environment that facilitates developing
cross-platform applications/tools. Microsoft is still putting great
development effort into WSL, it will likely evolve to become a much
better production environment. WSL also has some funky quirks relating
to path lengths and permissions, but they were edge case for me and
you'll probably never encounter them.

*Slate Toolkit* - I have a free Windows and Linux package distributions
that you can download from my website
(https://slate.rocks/slate-toolkit-edition/). I won't repeat the README
here. In brief, the toolkit packages includes all of the the MGIZA++
tools and the Moses tools for each platform as described in the README.
The Windows package includes the required native Windows GNU tools.
These are the GIT commits used to compile the binaries from Moses github
trunk.

MGIZA_COMMIT = d643960de98565d208114780ba8025799208afa7
MOSES_COMMIT_LINUX = 3a0631a05b7f53a7f387ca8ddca432f5ddb22029 (Jan
2018?)
MOSES_COMMIT_WINDOWS = a64468a9919867174beba71962ada795a4ce12d3
(pre-merge of Moses server)

The Linux binaries were compiled using Moses' BJAM. The Windows binaries
were compiled using my proprietary build environment (Visual Studio or
Cygwin). Both Linux and Windows were compiled with these options enabled:

* 64-bit architecture only
* static compiled -- no .dll/.so dependencies
* KenLM
* XML-RPC
* legacy binarized translation and reordering tables

The packages include Perl and Python scripts from MGIZA++ and Moses that
I updated for cross-platform use. This means these packages are
full-stack phrase-based SMT that create SMT models and decode with them.
Models trained/tuned on Windows can be copied to Linux for decoding and
vice versa.

The files in the packages are laid out so that you unzip the package,
open a command line and execute the demo shell scripts. There is a
sample data set so you can confirm everything works. There are demo
shell scripts (Bash for Linux, .CMD batch files for Windows) that
execute complete toolchains to build a baseline SMT model. They are not
suitable for production, but they should be a good learning environment
to compare Linux vs Windows usage. Standard pipes, sockets, file
system... they're all available for inter-process communications between
the Moses binaries and other Windows apps.

Licensing: All of the components, including the binaries, are
distributed under their respective open source licenses. For the most
part, that's LGPL-3 for Moses. The package has a complete list of open
source licenses per component. This means you can freely re-distribute
the binaries and scripts within the terms of the respective open source
licenses. I.e., I don't make any money on them.

*Slate Desktop* - Lastly, I'll mention my our flagship application. For
the SMT component, Slate Desktop uses Slate Toolkit. It includes many
many more tools for corpus preparation and integration with popular CAT
tools. We distribute Slate Desktop under a proprietary EULA. License
fees are based per-host. There are no subscription fees or royalties.
Slate Desktop's underlying architecture includes a primitive API. You
could, for example, license our API and build your application around
it. This means you could have your application running and tested in a
few days. If this interests you, please contact me off-list.

I hope this helps everyone and puts the Windows support in perspective.

Regards,
Tom


On 5/9/2018 11:58 PM, moses-support-request@mit.edu wrote:
> Date: Wed, 9 May 2018 18:52:48 +0200
> From: Mohamed Amine MENACER<am_menacer@esi.dz>
> Subject: Re: [Moses-support] Install moses on Windows
> To: Nisheeth Joshi<njoshi.in@gmail.com>,moses-support@mit.edu
> Cc:fadighawanmeh@yahoo.com
>
> Hi Nisheeth,
>
> Great!! I will try it out right away.
> I also think that it is a good idea to put this solution on the official
> website.
>
> Warm regards.
> Amine.
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20180510/51fb1140/attachment-0001.html

------------------------------

Message: 2
Date: Thu, 10 May 2018 06:51:44 +0000
From: Mohamed Amine MENACER <am_menacer@esi.dz>
Subject: Re: [Moses-support] Install moses on Windows
To: Tom Hoar <tahoar@precisiontranslationtools.com>
Cc: moses-support@mit.edu, fadighawanmeh@yahoo.com
Message-ID:
<CABtP53OzY6HWn8CV2=QKJe4J--8-HM_L=WGDX3Ez4Ue_YVjnmg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Tom,

What a great answer!! Thank you so much for the several proposals.
For me, I already did the training on Linux, so I just need to do the
decoding on Windows.
I will take a look on your solution "Slate Toolkit", it seems interesting.
Anyway, I have multiple choices, I must choose the best/right one.

Thank you again for your help.
Warm regards.
Amine.

On Thu, May 10, 2018, 08:13 Tom Hoar <tahoar@precisiontranslationtools.com>
wrote:

> Hi everyone. Sorry I joined the conversation a bit late.
>
> Here are all the options for Moses on Windows that I've tried:
>
> *Virtual machine* - VMs are more efficient than they used to be. Any good
> hypervisor will work. You need to build your own tools to create a sockets
> layer to communicate from the host machine to the VM.
>
> *Moses with Cygwin* - I never got it to work. There are too many
> variables and the underlying open source components change drastically. It
> could be one person's full-time job to track keep Moses updated for those
> build tools.
>
> *Moses with Visual Studio* - It's a good option if you're looking for
> only the moses.exe runtime binary. This does not address the serious
> challenges of updating all the other C++, Perl and Python components to run
> on Windows. Nor does it address collecting and installing all the GNU
> utilities.
>
> *Windows Subsystem for Linux (WSL)* - WSL is native Linux running on the
> Windows. Microsoft has done a great job of mapping the Linux kernel calls
> to the Windows kernel. There's no virtual machine. It's very much like
> running Windows applications on Linux using WINE... but without the
> complexities of re-mapping Microsoft's GUI APIs.
>
> From Moses' perspective, WSL is nothing special. It sees the environment
> as any native Linux. You can compile Linux binaries on Linux and copy them
> to your Windows 10 machine and run them with WSL. You will need to
> verify/install all of the other dependencies because WSL distributions are
> sparse at best. For example, the Ubuntu 16.04 distro for WSL does not
> install Python 2.7. The `python` command will fail. It installs Python 3.5
> by default with `python3` as the executable name. This will cause Moses to
> fail. You'll need to do a lot of troubleshooting.
>
> WSL also falls short when it comes to inter-process communications between
> WSL's native Linux apps and native Windows apps. I found one Microsoft
> instruction that WSL and Windows apps share the network sockets stack, and
> that is the only inter-process communication between the two layers. So,
> STDIN/STDOUT/STDERR are not available between native Windows apps and Linux
> apps on WSL. This means from a practical standpoint, you're back to working
> with WSL as though it's a VM. That is, you can build a sockets layer to
> communicate between the two. Maybe you can use the moses binary in Moses
> Server mode and use XML-RPC, but I never tried it.
>
> There's another awkward inter-process strategy. If you're in a Windows
> cmd.exe terminal or PowerShell terminal, you can execute a command line
> that will launch a Linux app in the WSL space. (and vice-versa. From a
> native Linux Bash terminal, you can launch a native Windows .exe, but
> that's irrelevant to Moses) So, you could have your Windows app save a
> text file to the file system, then execute the Moses Linux binary with the
> -i argument pointing to the input file. The challenge here is translating
> the Windows file system "C:\Users\...." to the WSL file system
> "/mnt/c/Users/...". You'll also have to figure out a way to pass the STDOUT
> redirection to the moses binary to save it's output. I don't know if this
> is possible. Then, your Windows application that spawned the Linux Moses
> binary will need to detect when the binary is done, and then retrieve the
> output file. WARNING: do not directly access the Linux file system tree
> from Windows
> [%LOCALAPPDATA%\Packages\CanonicalGroupLimited.UbuntuonWindows_xxxxxxxxxxxxx\LocalState\rootfs].
> You are virtually guaranteed to corrupt the files. Only have Windows apps
> save to it's native C:\ drives and then access them from Linux using the
> mount director tree.
>
> Bottom line... you should read what Microsoft's engineers have written
> about why they created WSL. It is not intended to be a production tool. It
> is designed as a development environment that facilitates developing
> cross-platform applications/tools. Microsoft is still putting great
> development effort into WSL, it will likely evolve to become a much better
> production environment. WSL also has some funky quirks relating to path
> lengths and permissions, but they were edge case for me and you'll probably
> never encounter them.
>
> *Slate Toolkit* - I have a free Windows and Linux package distributions
> that you can download from my website (
> https://slate.rocks/slate-toolkit-edition/). I won't repeat the README
> here. In brief, the toolkit packages includes all of the the MGIZA++ tools
> and the Moses tools for each platform as described in the README. The
> Windows package includes the required native Windows GNU tools. These are
> the GIT commits used to compile the binaries from Moses github trunk.
>
> MGIZA_COMMIT = d643960de98565d208114780ba8025799208afa7
> MOSES_COMMIT_LINUX = 3a0631a05b7f53a7f387ca8ddca432f5ddb22029 (Jan 2018?)
> MOSES_COMMIT_WINDOWS = a64468a9919867174beba71962ada795a4ce12d3 (pre-merge
> of Moses server)
>
> The Linux binaries were compiled using Moses' BJAM. The Windows binaries
> were compiled using my proprietary build environment (Visual Studio or
> Cygwin). Both Linux and Windows were compiled with these options enabled:
>
> - 64-bit architecture only
> - static compiled -- no .dll/.so dependencies
> - KenLM
> - XML-RPC
> - legacy binarized translation and reordering tables
>
> The packages include Perl and Python scripts from MGIZA++ and Moses that I
> updated for cross-platform use. This means these packages are full-stack
> phrase-based SMT that create SMT models and decode with them. Models
> trained/tuned on Windows can be copied to Linux for decoding and vice
> versa.
>
> The files in the packages are laid out so that you unzip the package, open
> a command line and execute the demo shell scripts. There is a sample data
> set so you can confirm everything works. There are demo shell scripts (Bash
> for Linux, .CMD batch files for Windows) that execute complete toolchains
> to build a baseline SMT model. They are not suitable for production, but
> they should be a good learning environment to compare Linux vs Windows
> usage. Standard pipes, sockets, file system... they're all available for
> inter-process communications between the Moses binaries and other Windows
> apps.
>
> Licensing: All of the components, including the binaries, are distributed
> under their respective open source licenses. For the most part, that's
> LGPL-3 for Moses. The package has a complete list of open source licenses
> per component. This means you can freely re-distribute the binaries and
> scripts within the terms of the respective open source licenses. I.e., I
> don't make any money on them.
>
> *Slate Desktop* - Lastly, I'll mention my our flagship application. For
> the SMT component, Slate Desktop uses Slate Toolkit. It includes many many
> more tools for corpus preparation and integration with popular CAT tools.
> We distribute Slate Desktop under a proprietary EULA. License fees are
> based per-host. There are no subscription fees or royalties. Slate
> Desktop's underlying architecture includes a primitive API. You could, for
> example, license our API and build your application around it. This means
> you could have your application running and tested in a few days. If this
> interests you, please contact me off-list.
>
> I hope this helps everyone and puts the Windows support in perspective.
>
> Regards,
> Tom
>
> On 5/9/2018 11:58 PM, moses-support-request@mit.edu wrote:
>
> Date: Wed, 9 May 2018 18:52:48 +0200
> From: Mohamed Amine MENACER <am_menacer@esi.dz> <am_menacer@esi.dz>
> Subject: Re: [Moses-support] Install moses on Windows
> To: Nisheeth Joshi <njoshi.in@gmail.com> <njoshi.in@gmail.com>, moses-support@mit.edu
> Cc: fadighawanmeh@yahoo.com
>
> Hi Nisheeth,
>
> Great!! I will try it out right away.
> I also think that it is a good idea to put this solution on the official
> website.
>
> Warm regards.
> Amine.
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20180510/92e7395c/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 139, Issue 6
*********************************************

Tweet