Moses-support Digest, Vol 90, Issue 12

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Implementation of pre-reordering for German andother
languages (Hwidong Na)
2. Re: yum-installed Boost environment variables for MGIZA++ (Gideon)


----------------------------------------------------------------------

Message: 1
Date: Fri, 04 Apr 2014 12:09:48 +0900
From: Hwidong Na <leona@postech.ac.kr>
Subject: Re: [Moses-support] Implementation of pre-reordering for
German andother languages
To: Benjamin K?rner <b.koerner@stud.uni-heidelberg.de>,
maxkhalilov@gmail.com
Cc: 'moses-support' <moses-support@mit.edu>, mt-list@eamt.org
Message-ID: <533E227C.4040003@postech.ac.kr>
Content-Type: text/plain; charset="iso-8859-1"

Hi all,

If you do not have to use syntactic parser, it is possible to utilize a
discriminative parser for word reordering, such as STIR
(http://research.google.com/pubs/pub37163.html, but no implementation
available) or LADER
(http://www.phontron.com/paper/neubig12emnlp-reordering.pdf,
implementation available at https://github.com/neubig/lader).

Best,

Hwidong Na
POSTECH, Korea

On 04/04/2014 01:54 AM, Benjamin K?rner wrote:
>
> Hi Maxim, hi all,
>
> in an undergraduate class we were working on a automatic rule
> extraction and source-side preordering system as a preprocessing step
> for English- Japanese. At the moment I'm writing my BA thesis on this
> stuff. It works with Moses and cdec and since it is just preprocessing
> it should work with any other decoder.
>
> If you exchange the parser and maybe some of the parameters it should
> work for every language pair. We get BLEU score improvement in
> intrinsic evaluation (right now with a 1,7 mio sentence eng-jap patent
> corpus).
>
> The idea came from Genzel
> http://research.google.com/pubs/archive/36484.pdf
>
> Also check our final presentation slides of our implemention @
> https://www.cl.uni-heidelberg.de/studies/projects/poster/hitschler_koerner_ohta2013.pdf
>
> Best,
>
> Benjamin
>
> *Von:*moses-support-bounces@mit.edu
> [mailto:moses-support-bounces@mit.edu] *Im Auftrag von *Milos Stanojevic
> *Gesendet:* Donnerstag, 3. April 2014 16:12
> *An:* Hieu Hoang
> *Cc:* moses-support; mt-list@eamt.org
> *Betreff:* Re: [Moses-support] Implementation of pre-reordering for
> German and other languages
>
> Hi Maxim,
>
> You can check this paper:
> http://aclweb.org/anthology//P/P11/P11-2067.pdf
> <http://aclweb.org/anthology/P/P11/P11-2067.pdf>
>
> It says: "CKK uses the Dubey and Keller (2003) parser, which is
> trained on the Negra corpus (Skut et al., 1997)."
>
> Regards,
>
> Milos Stanojevic
>
> On Thu, Apr 3, 2014 at 12:50 PM, Hieu Hoang <Hieu.Hoang@ed.ac.uk
> <mailto:Hieu.Hoang@ed.ac.uk>> wrote:
>
> I don't know of an implementation that's available for download and I
> seem to remember that the parser they used was extremely difficult to
> use or compile.
>
> If you find out more, please let everyone know
>
> On 25 March 2014 17:07, Maxim Khalilov <maxkhalilov@gmail.com
> <mailto:maxkhalilov@gmail.com>> wrote:
>
> Dear Moses community,
>
> I am looking for a ready to use or easily customizable
> implementation of pre-reordering algorithms for Moses? In
> particular, I'm interested in language pairs with German as a
> source language and a variety of languages as target, so probably
> the best solution is syntax based.
>
> As a starting point, I would consider the algorithm described in
> (Collins et al., 2005), but I don't know if there is an
> implementation available and which parser it relies on.
>
> Thanks for your help beforehand,
>
> Maxim Khalilov
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140404/5f810a79/attachment-0001.htm

------------------------------

Message: 2
Date: Fri, 4 Apr 2014 10:09:23 +0200
From: Gideon <gidi8ster@gmail.com>
Subject: Re: [Moses-support] yum-installed Boost environment variables
for MGIZA++
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CALP7mvnq5k5kWHgpmoiMtDu8Q_SGQ3B3kjOMU9ds7WJn=v4VQQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Dear Hieu Hoang,

I have now tried to compile Boost manually instead. There are still errors,
and now Moses compilation also fails. (I assumed that I should re-compile
Moses since it uses Boost). These were my steps:

cd boost_1_55_0
./bootstrap

Following your advice:

./b2 -j4 -a --prefix=$PWD --libdir=$PWD/lib64 --layout=tagged link=static
threading=multi,single install || echo FAILURE
(-a to re-compile)

There is one message written to standard error:
link.jam: No such file or directory (it does not exist anywhere in the
downloaded version)

The rest is in standard output. I have attached the two files as a tarball.

Here is a typical warning:

./boost/concept/detail/general.hpp:71:20: warning: typedef
'boost_concept_check653' locally defined but not used
[-Wunused-local-typedefs]
BOOST_PP_CAT(boost_concept_check,__LINE__)

And an error:

libs/iostreams/src/bzip2.cpp:20:56: fatal error: bzlib.h: No such file or
directory
#include "bzlib.h" // Julian Seward's "bzip.h" header.
^
compilation terminated.

Eventually:
...failed updating 2 targets...
...skipped 6 targets...
...updated 10795 targets...

Using your command, "ERROR" is written to output. I don't know if it is
fatal (i.e. if I can still use Moses and MGIZA++).

For Moses, I assume the following is correct:

export BOOST_ROOT=/home/kotzegj/tools/boost_1_55_0
export BOOST_BUILD_PATH=/home/kotzegj/tools/boost_1_55_0/boost

I noticed that now, BOOST_ROOT does have "include", which means that
somehow the installation with yum is different.

I ran bjam like this:

./bjam --with-irstlm=/usr/local/irstlm
--with-boost=/home/kotzegj/tools/boost_1_55_0 -j4 -a

Here is a typical error:

/usr/bin/ld: cannot find -lboost_iostreams
collect2: error: ld returned 1 exit status
...failed gcc.link
moses-chart-cmd/bin/gcc-4.8.2/release/debug-symbols-on/link-static/threading-multi/moses_chart...
gcc.compile.c++
moses-cmd/bin/gcc-4.8.2/release/debug-symbols-on/link-static/threading-multi/mbr.o

I cannot find any file containing boost_iostreams under my Boost path,
however:

[kotzegj@lt-2433323 boost_1_55_0]$ fnd "iostreams"
./boost/iostreams
./doc/html/boost_asio/example/cpp03/iostreams
./libs/asio/example/cpp03/iostreams
./libs/iostreams
./bin.v2/libs/iostreams
./include/boost/iostreams

Perhaps this means that boost_iostreams wasn't built or something? I can
attach build.log.gz (of Moses) in a following email.

Thank you for your time.

Gideon



---
gidi8ster@gmail.com
www.gideonkotze.nl
+27 78 739 8923 (Mobile)


On Thu, Apr 3, 2014 at 6:43 PM, Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:

> I'm not too sure. Maybe you should try to compile boost yourself.
>
> The instructions are here
> http://www.statmt.org/moses/?n=Development.GetStarted
>
>
> On 3 April 2014 16:23, Gideon <gidi8ster@gmail.com> wrote:
>
>> I have done what you asked, and removed all occurrences of "-mt" in
>> compile.sh. However:
>>
>> /usr/bin/ld: cannot find -lboost_system
>> /usr/bin/ld: cannot find -lboost_thread
>>
>> /usr/bin/ld: cannot find -lpthread
>> /usr/bin/ld: cannot find -lstdc++
>> /usr/bin/ld: cannot find -lm
>> /usr/bin/ld: cannot find -lc
>> collect2: error: ld returned 1 exit status
>>
>> Two other things I noticed:
>>
>> - The script expects a $BOOST_ROOT/include, although there exists no
>> include directory in /home/kotzegj/tools/boost_1_54_0
>> - Even if I change $BOOST_ROOT to, say, "/usr" (which does contain
>> "include" which on its own contains "boost"), the error output is exactly
>> the same.
>>
>> I have run "locate boost" and attached the output in case you wanted to
>> look at it. Perhaps you can tell from it if I have a faulty installation?
>>
>> Thanks again.
>>
>> Gideon
>>
>>
>>
>> ---
>> gidi8ster@gmail.com
>> www.gideonkotze.nl
>> +27 78 739 8923 (Mobile)
>>
>>
>> On Thu, Apr 3, 2014 at 4:59 PM, Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:
>>
>>> change
>>> -lboost_*-mt
>>> to
>>> -lboost_*
>>>
>>> i'm not sure why you're linking to stdc++, m, c.
>>>
>>> Check there's not typo in your compile.sh
>>>
>>> On 3 April 2014 15:52, Gideon <gidi8ster@gmail.com> wrote:
>>>
>>>> Thank you, I have now managed with SVN, it turns out that I had to
>>>> change the configuration for my (university) proxy settings. However, there
>>>> are still compilation errors it seems. I have warnings such as this:
>>>>
>>>> /home/kotzegj/tools/mgiza/mgizapp-code/mgizapp/src/mkcls/KategProblem.cpp:446:9:
>>>> warning: deprecated conversion from string constant to 'char*'
>>>> [-Wwrite-strings]
>>>> in="other ";
>>>>
>>>> And perhaps more importantly, some files that are not found:
>>>>
>>>> /usr/bin/ld: cannot find -lboost_system-mt
>>>> /usr/bin/ld: cannot find -lboost_thread-mt
>>>> /usr/bin/ld: cannot find -lpthread
>>>> /usr/bin/ld: cannot find -lstdc++
>>>> /usr/bin/ld: cannot find -lm
>>>> /usr/bin/ld: cannot find -lc
>>>>
>>>> which would indicate that there are still some problems with (finding)
>>>> Boost.
>>>>
>>>> And still no mgiza binary.
>>>>
>>>> Some context (fnd is an alias for "find . -name"):
>>>>
>>>> [kotzegj@lt-2433323 usr]$ fnd "*boost_thread*"
>>>> ./lib64/libboost_thread.so
>>>> ./lib64/libboost_thread.so.1.54.0
>>>>
>>>> [kotzegj@lt-2433323 usr]$ fnd "*boost_system*"
>>>> ./lib64/libboost_system.so.1.54.0
>>>> ./lib64/libboost_system.so
>>>>
>>>> As mentioned, I installed Boost using yum, and these are my current
>>>> variables in compile.sh:
>>>>
>>>> SRC_DIR=/home/kotzegj/tools/mgiza/mgizapp-code/mgizapp/src
>>>> BOOST_ROOT=/home/kotzegj/tools/boost_1_54_0
>>>> BOOST_LIBRARYDIR=/usr/lib64
>>>>
>>>> BOOST_ROOT is pointing to the uncompiled unbuilt Boost - perhaps it
>>>> should point somewhere else?
>>>>
>>>> Thank you for your time.
>>>>
>>>> Best,
>>>>
>>>> Gideon Kotz?
>>>>
>>>>
>>>>
>>>>
>>>> ---
>>>> gidi8ster@gmail.com
>>>> www.gideonkotze.nl
>>>>
>>>>
>>>> On Thu, Apr 3, 2014 at 3:05 PM, Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:
>>>>
>>>>> You should use the version in svn, rather than a packaged version.
>>>>> mgiza isn't really looked after by anyone anymore, bug fixes go straight
>>>>> into the svn and nowhere else.
>>>>>
>>>>> I've just pushed a fix for a minor OSX compile error
>>>>> https://sourceforge.net/p/mgizapp/code/48/
>>>>>
>>>>> Try to download it via svn. If it doesn't work, i will consider moving
>>>>> it to github.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On 2 April 2014 14:34, Gideon <gidi8ster@gmail.com> wrote:
>>>>>
>>>>>> Dear Hieu Hoang
>>>>>>
>>>>>> Thank you for you reply. I made some changes:
>>>>>>
>>>>>> - uncommented out the top variables and commented out the Mac ones
>>>>>> - set BOOST_ROOT to /home/kotzegj/tools/boost_1_54_0 and
>>>>>> BOOST_LIBRARYDIR to /usr/lib64
>>>>>> - set SRC_DIR to /home/kotzegj/tools/mgiza/mgizapp/src
>>>>>>
>>>>>> and ran ./compile.sh.
>>>>>>
>>>>>> Two issues remain so far:
>>>>>>
>>>>>> 1) I can't find any MGIZA++/mgiza++/etc. binary (find . -name). Is
>>>>>> there a way to test if MGIZA++ has installed correctly? I did not notice
>>>>>> any other error messages, apart from the files the script tries to remove
>>>>>> at the beginning, which do not exist.
>>>>>>
>>>>> it should be called
>>>>> manual-compile/mgiza
>>>>> If not, there was a compile error
>>>>>
>>>>>>
>>>>>> 2) When compiling, this warning pops up all the time (you are
>>>>>> probably aware of it)
>>>>>>
>>>>>> /usr/include/c++/4.8.2/backward/backward_warning.h:32:2: warning:
>>>>>> #warning This file includes at least one deprecated or antiquated header
>>>>>> which may be removed without further notice at a future date. Please use a
>>>>>> non-deprecated interface with equivalent functionality instead. For a
>>>>>> listing of replacement headers and interfaces, consult the file
>>>>>> backward_warning.h. To disable this warning use -Wno-deprecated. [-Wcpp]
>>>>>>
>>>>>> So I assume this is OK for now but there needs to be an update in the
>>>>>> future?
>>>>>>
>>>>> i see it. Be my guest if you wanna update it
>>>>>
>>>>>>
>>>>>> and (just to test)
>>>>>>
>>>>>> 2) [kotzegj@lt-2433323 manual-compile]$ ./mkcls
>>>>>> ERROR: can not open file train.
>>>>>> Error: Could not read the file 'train'.
>>>>>>
>>>>>> But perhaps this is the default?
>>>>>>
>>>>> I don't know. Try running it with valid arguments, eg
>>>>>
>>>>> http://www.statmt.org/moses/RELEASE-2.1/models/cs-en/steps/1/TRAINING_prepare-data.1.STDERR
>>>>>
>>>>>
>>>>>>
>>>>>> 3) svn checkout http://svn.code.sf.net/p/mgizapp/code/trunkmgizapp-code
>>>>>> --> did not work (timeout). I downloaded 0.7.3 from here instead:
>>>>>> http://www.kyloo.net/software/doku.php/mgiza:overview
>>>>>>
>>>>>> svn: E000110: Unable to connect to a repository at URL '
>>>>>> http://svn.code.sf.net/p/mgizapp/code/trunk'
>>>>>> svn: E000110: Error running context: Connection timed out
>>>>>>
>>>>>> Thank you for your time.
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> Gideon
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---
>>>>>> gidi8ster@gmail.com
>>>>>> www.gideonkotze.nl
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 2, 2014 at 2:38 PM, Hieu Hoang <hieuhoang@gmail.com>wrote:
>>>>>>
>>>>>>> the cmake build system on mgiza is difficult to use.
>>>>>>>
>>>>>>> I create an alternative compile script for mgiza. If you've download
>>>>>>> mgiza via svn, you should find my script in
>>>>>>> mgizapp/manual-compile/compile.sh
>>>>>>> Look at it, change the paths for your own needs
>>>>>>>
>>>>>>>
>>>>>>> On 02/04/2014 11:34, Gideon wrote:
>>>>>>>
>>>>>>> Dear Moses support team
>>>>>>>
>>>>>>> I was wondering if MGIZA++ has been test-compiled on a Fedora
>>>>>>> Linux system where Boost has been installed using yum, as I'm encountering
>>>>>>> some problems. I'm working on a Fedora 20 system with x86_64 architecture.
>>>>>>> So far I have done the following:
>>>>>>>
>>>>>>> yum install boost.x86_64
>>>>>>> yum install boost-devel.x86_64 (version 1.54 was installed)
>>>>>>> yum install gcc
>>>>>>> yum install gcc-c++
>>>>>>> yum install gperftools
>>>>>>> Downloaded and installed IRSTLM
>>>>>>> ./bjam --with-irstlm=/home/kotzegj/tools/irstlm-5.80.03 -j4
>>>>>>> --with-boost=/usr/lib64
>>>>>>> export BOOST_ROOT=/usr/lib64
>>>>>>> export
>>>>>>> BOOST_BUILD_PATH=/home/kotzegj/tools/mosesdecoder/jam-files/boost-build
>>>>>>> ./bjam --with-irstlm=/usr/local/irstlm -j4
>>>>>>>
>>>>>>> I have tested Moses with sample-models and it seems OK.
>>>>>>>
>>>>>>> However, MGIZA++ does not find Boost on its own:
>>>>>>>
>>>>>>> -- Could NOT find Boost
>>>>>>> CMake Error at CMakeLists.txt:59 (MESSAGE):
>>>>>>> Boost not found, please set the BOOST_ROOT and BOOST_LIBRARYDIR
>>>>>>> environment
>>>>>>> variables
>>>>>>>
>>>>>>> I have no success with setting either of these variables to:
>>>>>>>
>>>>>>> - /usr/lib64
>>>>>>> - /usr/include
>>>>>>> - /usr/include/boost
>>>>>>> - /home/kotzegj/tools/mosesdecoder/jam-files/boost-build
>>>>>>> etc.
>>>>>>>
>>>>>>> or if I download Boost manually, set the decompressed directory as
>>>>>>> BOOST_ROOT, and symlink lib64 as a child and set that as BOOST_LIBRARYDIR.
>>>>>>> (I've tried all kinds of stuff.)
>>>>>>>
>>>>>>> I have the feeling (after some Googling) that the yum installation
>>>>>>> does not result in the file structure that cmake is looking for, and that
>>>>>>> instead I should try a manual installation. I hope that I'm wrong?
>>>>>>>
>>>>>>> Thank you for your time.
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> Gideon Kotz?
>>>>>>>
>>>>>>> ---
>>>>>>> gidi8ster@gmail.com
>>>>>>> www.gideonkotze.nl
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Moses-support mailing list
>>>>>>> Moses-support@mit.edu
>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Hieu Hoang
>>>>> Research Associate
>>>>> University of Edinburgh
>>>>> http://www.hoang.co.uk/hieu
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Hieu Hoang
>>> Research Associate
>>> University of Edinburgh
>>> http://www.hoang.co.uk/hieu
>>>
>>>
>>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140404/675cf8c6/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: boost-log.tar.gz
Type: application/x-gzip
Size: 155813 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140404/675cf8c6/attachment.bin

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 90, Issue 12
*********************************************

Tweet