Moses-support Digest, Vol 97, Issue 24

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Load LD_LIBRARY_PATH in ems with -cluster (sge) (Hieu Hoang)
2. train-model.perl returns 0 despite failure in
merge_alignments.py (Kenneth Heafield)
3. Basline systeme (Cyrine NASRI)
4. Re: Basline systeme (Philipp Koehn)


----------------------------------------------------------------------

Message: 1
Date: Thu, 13 Nov 2014 18:06:38 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Load LD_LIBRARY_PATH in ems with -cluster
(sge)
To: "Raymond W. M. Ng" <wm.ng@sheffield.ac.uk>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbhzZ0sXPPprmFdnZrjFyaheAPhOAnR7RheC5XV7KGFh9Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

this was due to a change a couple of months ago.
https://www.mail-archive.com/moses-support%40mit.edu/msg10653.html
The documentation to create the boost libraries has been updated
http://www.statmt.org/moses/?n=Development.GetStarted



On 13 November 2014 16:59, Raymond W. M. Ng <wm.ng@sheffield.ac.uk> wrote:

> Hi Hieu,
>
> I just recompiled moses to link boost statically.
> When I did this last time months ago, I remember I failed to link boost
> statically. I didn't trace the error and just had a dynamic build.
>
> A minor trick may be necessary for others to link boost statically when
> compiling moses....
>
> In my boost lib folder I only have
> libbost_thread-mt.a
> but *NOT* libboost_thread.a
>
> I need to link the two
> $ ln -s libboost_thread-mt.a libboost_thread.a
> before moving on compiling moses.
>
> Otherwise moses build will complain
> /usr/bin/ld: cannot find -lboost_thread
>
>
> best
> raymond
>
>
> This may serve a note to others
>
>
> On 13 November 2014 14:44, Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:
>
>> you don't need to set LD_LIBRARY_PATH if the boost lib statically linked.
>> This is how I always use it, I never set this variable.
>>
>> You need to make sure there's no .so file in the folder
>> /export/sw/std/boost/v1.56.0/x86_64/lib
>> /export/sw/std/boost/v1.56.0/x86_64/lib64
>>
>>
>> On 13 November 2014 14:39, Raymond W. M. Ng <wm.ng@sheffield.ac.uk>
>> wrote:
>>
>>> Thanks Hieu,
>>>
>>> I did link boost (and sirlm, irstlm) when i built moses.
>>>
>>> ./bjam --with-boost=/export/sw/std/boost/v1.56.0/x86_64
>>> --with-srilm=/export/sw/spl/srilm/v1.6.0/srilm
>>> --with-irstlm=/export/sw/spl/irstlm/v5.80.03/x86_64 -j8
>>>
>>> But I think at run time I still need to load the path.
>>> setenv LD_LIBRARY_PATH /export/sw/std/boost/v1.56.0/x86_64/lib
>>>
>>> It won't cause an issue if everything is run locally, because I could
>>> source the environment just once. Maybe this problem is, again, specific to
>>> computing environment
>>>
>>> cheers
>>> raymond
>>>
>>>
>>> On 13 November 2014 13:36, Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:
>>>
>>>> personally, I would try to ensure that moses was statically linked to
>>>> the boost libraries so it won't go looking for the libboost*.so at runtime.
>>>>
>>>> I do this by building my own boost library and linking to it, rather
>>>> than the system boost. The instructions are here
>>>> http://www.statmt.org/moses/?n=Development.GetStarted
>>>>
>>>> On 13 November 2014 13:23, Raymond W. M. Ng <wm.ng@sheffield.ac.uk>
>>>> wrote:
>>>>
>>>>> Dear moses,
>>>>>
>>>>> I have the following error in training my translation model with ems
>>>>> on SGE.
>>>>>
>>>>> /home/users/moses/v-2014-08-15/mosesdecoder/bin/build_binary: error
>>>>> while loading shared libraries: libboost_thread-mt.so.1.56.0: cannot open
>>>>> shared object file: No such file or directory
>>>>>
>>>>> In the offline mode, I can run this without problem by first sourcing
>>>>> an environment file which contains something like
>>>>> setenv LD_LIBRARY_PATH /export/sw/boost/v1.56.0/x86_64/lib
>>>>>
>>>>> So, I think the LD_LIBRARY_PATH environment was not loaded in the sge
>>>>> parallel scripts,
>>>>>
>>>>> grep 'ENV' /home/moses/mosesdecoder/scripts/generic/qsub-wrapper.pl
>>>>>
>>>>> returns nothing.
>>>>>
>>>>> Is there something to modify in the code or did I miss any setting so
>>>>> the environment variable is not loaded?
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Hieu Hoang
>>>> Research Associate
>>>> University of Edinburgh
>>>> http://www.hoang.co.uk/hieu
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>>
>> --
>> Hieu Hoang
>> Research Associate
>> University of Edinburgh
>> http://www.hoang.co.uk/hieu
>>
>>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141113/92b5e41a/attachment-0001.htm

------------------------------

Message: 2
Date: Thu, 13 Nov 2014 17:14:13 -0500
From: Kenneth Heafield <moses@kheafield.com>
Subject: [Moses-support] train-model.perl returns 0 despite failure in
merge_alignments.py
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <54652D35.9030004@kheafield.com>
Content-Type: text/plain; charset=ISO-8859-1

Dear Moses,

The PYTHONIOENCODING environment variable sets Python's default
encoding because Python is too lame to default to utf-8. When is not
set, and a particular flavor of python is installed, merge_alignment.py
from mgiza fails. That triggers a snippet of train-model.perl output below:

Executing: /external-bin/merge_alignment.py
/experiment/giza.f-e/f-e.A3.final.part*>/experiment/giza.f-e/f-e.A3.final
Traceback (most recent call last):
File "/external-bin/merge_alignment.py", line 57, in <module>
sys.stdout.write("%s%s%s"%(sents[i][0],sents[i][1],sents[i][2]));
UnicodeEncodeError: 'ascii' codec can't encode character '\ufeff' in
position 84: ordinal not in range(128)
Exit code: 1
Executing: rm -f /experiment/giza.f-e/f-e.A3.final.gz
Executing: gzip /experiment/giza.f-e/f-e.A3.final
Waiting for second GIZA process...

However, when run with -last-step 2 (so stop after GIZA), the
train-model.perl script still returns 0. So I think there are two bugs:

1. Set (or at least document) PYTHONIOENCODING=utf-8 in the environment.
Google doesn't see it anywhere on statmt.org.

2. train-model.perl should return non-zero on failure of a command like
this.

Kenneth


------------------------------

Message: 3
Date: Thu, 13 Nov 2014 23:58:59 +0100
From: Cyrine NASRI <cyrine.nasri@univ-lorraine.fr>
Subject: [Moses-support] Basline systeme
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAPg_V0h2dgkKACKz=Xk86zyZ5uRuBciTWUzFn+W7WCfxL7A3BQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hello,
I'am trying to build the baseline SMT for english to german language. I've
got a "strange" problem because i work for the 1st time on german language
and i've got a score Bleu on Dev corpus = 16 and on the test corpus = 9 !
i prepared the corpus before training and mert as usual like this:
tokenization + cleaning + lowercase

Is there somthing special for german language?
PS: training, dev and test corpus are in the same domain (Europarl)

Thank you in advance

Best regards

Cyrine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141113/efacd153/attachment-0001.htm

------------------------------

Message: 4
Date: Thu, 13 Nov 2014 18:54:43 -0500
From: Philipp Koehn <pkoehn@inf.ed.ac.uk>
Subject: Re: [Moses-support] Basline systeme
To: Cyrine NASRI <cyrine.nasri@univ-lorraine.fr>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDAy2c0EG5G8pkMF+eeK4V+yStsdZwyn3AzT9gOFWdoJTA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hi,

if the tuning and test set are sampled from the same data, you should expect
similar scores. You could swap the two sets and see what happens - maybe
it is just that they are of different difficulty.

Phrase-based models have some deficiencies when it comes to German, such as
translation of compounds, and long distance reordering. We are having
more success with syntax-based models for this language pair.

-phi


On Thu, Nov 13, 2014 at 5:58 PM, Cyrine NASRI
<cyrine.nasri@univ-lorraine.fr> wrote:
> Hello,
> I'am trying to build the baseline SMT for english to german language. I've
> got a "strange" problem because i work for the 1st time on german language
> and i've got a score Bleu on Dev corpus = 16 and on the test corpus = 9 !
> i prepared the corpus before training and mert as usual like this:
> tokenization + cleaning + lowercase
>
> Is there somthing special for german language?
> PS: training, dev and test corpus are in the same domain (Europarl)
>
> Thank you in advance
>
> Best regards
>
> Cyrine
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 97, Issue 24
*********************************************

Tweet