Home » Uncategories » Moses-support Digest, Vol 93, Issue 35

Moses-support Digest, Vol 93, Issue 35

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Issues with Incremental retraining using Moses (Sandipan Dandapat)
2. lmplz and pruning bug (Marcin Junczys-Dowmunt)
3. Issue with tc mallo con Ubuntu-14.04 (Nicola Bertoldi)
4. Re: Issue with tc mallo con Ubuntu-14.04 (Tom Hoar)
5. Re: Predetermined translations in training data (Hieu Hoang)
6. Re: Issue with tc mallo con Ubuntu-14.04 (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Mon, 28 Jul 2014 07:51:39 +0100
From: Sandipan Dandapat <sandipandandapat@gmail.com>
Subject: [Moses-support] Issues with Incremental retraining using
Moses
To: moses-support@MIT.EDU
Message-ID:
<CAGr2oZRHVbBJ1M7Q631TiMwQj6mnw_TUBb_MbY84uRPyHhPk_g@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,
I am trying to use Moses Incremental Retraining as described in
http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc33

I have two doubts:

1. I am able to generate the new-alignment-file using additional data on
top of previously used data. Once the new alignment file is generated, the
page says to update the model. I am unable to understand how to use the
same during decoding? Can you please help me to understand how can I
proceed once my new-allignment file is generated?

2. What is happening when we are updating the moses.ini file using

PhraseDictionaryDynSuffixArray source=<path-to-source-corpus>
target=<path-to-target-corpus> alignment=<path-to-alignments>

I am unable to see any reference to this updated moses.ini file in the
rest of the section.

Thanks and regards,
sandipan
--------------------------------
Sandipan Dandapat
Postdoctoral Researcher
CNGL, School of Computing
Dublin City University
Google Scholar Profile:
http://scholar.google.co.in/citations?user=DWD_FiQAAAAJ&hl=en
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140728/be15165d/attachment-0001.htm

------------------------------

Message: 2
Date: Mon, 28 Jul 2014 09:42:41 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: [Moses-support] lmplz and pruning bug
To: moses-support <moses-support@mit.edu>
Message-ID: <53D5FEF1.90309@amu.edu.pl>
Content-Type: text/plain; charset=UTF-8; format=flowed

Hi,
some time ago pruning has been added for lmplz, but it turns out there
was a bug that sometimes caused back-off weights to be replaced by 0. If
you have used pruning before please update to the recent Kenlm or Moses
version, both contain a fixed lmplz now.

Also, there is still a bug in lmplz and pruning that may cause lmplz to
fail after the fourth step with an error message concerning missing
n-grams or back-off weights. This happens rarely and rather for corpora
with atypical n-gram distributions (e.g. artificially altered data). I
am working on that.

If lmplz with pruning makes it through to the end, the resulting ARPA
file should be correct now. Comparing with SRILM, I am getting very
similar perplexity results for different corresponding pruning settings.

Sorry for the inconveniences that might have caused.
Best,
Marcin


------------------------------

Message: 3
Date: Mon, 28 Jul 2014 10:08:25 +0200
From: Nicola Bertoldi <bertoldi@fbk.eu>
Subject: [Moses-support] Issue with tc mallo con Ubuntu-14.04
To: moses-support support <moses-support@mit.edu>
Message-ID: <B45CC5EC-3FE5-4C0C-AEDC-9E5C18E96141@fbk.eu>
Content-Type: text/plain; charset=us-ascii

hi guys

I am compiling moses on Ubuntu-14.04, but I cannot link it with tcmalloc correctly

I installed the tcmalloc lib with the command
sudo apt install libtcmalloc-minimal4
and it went well, because the dynamic library exists as shown here

$> ls -ltr /usr/lib/libtcmalloc_minimal*
lrwxrwxrwx 1 root root 28 Feb 4 21:49 /usr/lib/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.1.2
lrwxrwxrwx 1 root root 34 Feb 4 21:49 /usr/lib/libtcmalloc_minimal_debug.so.4 -> libtcmalloc_minimal_debug.so.4.1.2
-rw-r--r-- 1 root root 155000 Feb 4 21:49 /usr/lib/libtcmalloc_minimal.so.4.1.2
-rw-r--r-- 1 root root 204184 Feb 4 21:49 /usr/lib/libtcmalloc_minimal_debug.so.4.1.2

$> file /usr/lib/libtcmalloc_minimal.so.4.1.2
/usr/lib/libtcmalloc_minimal.so.4.1.2: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=3f5d8c7638476e345c4f9c0345c4506179f186fe, stripped


but when I test whether Moses links it, with these commands (as suggested by Kenneth (see his mail of November 18, 2012 7:04:39 PM)

$> bash -c "g++ -ltcmalloc_minimal -x c++ - <<<'int main() {}' -o /home/ubuntu/Software/MOSES/mosesdecoder-dynamic-models/dummy >/dev/null 2>/dev/null && rm /home/ubuntu/Software/MOSES/mosesdecoder-dynamic-models/dummy 2>/dev/null"
$> echo$?

I gor the value "1", instead of 0, i.e. error


What am I wrong?

cheers,
Nicola





------------------------------

Message: 4
Date: Mon, 28 Jul 2014 07:50:57 -0400
From: Tom Hoar <tahoar@precisiontranslationtools.com>
Subject: Re: [Moses-support] Issue with tc mallo con Ubuntu-14.04
To: moses-support@MIT.EDU
Message-ID: <53D63921.5050804@precisiontranslationtools.com>
Content-Type: text/plain; charset=windows-1252; format=flowed

Hi Nicola,

We traced one tmalloc failure failure to link on Ubuntu 14.04 to a
change in the installed default dependencies. The solution was to
install libgoogle-perftools-dev.

It works around the problem we had by creating an unversioned softlink
for the current specific version of libtcmalloc.

Tom


On 07/28/2014 04:08 AM, Nicola Bertoldi wrote:
> hi guys
>
> I am compiling moses on Ubuntu-14.04, but I cannot link it with tcmalloc correctly
>
> I installed the tcmalloc lib with the command
> sudo apt install libtcmalloc-minimal4
> and it went well, because the dynamic library exists as shown here
>
> $> ls -ltr /usr/lib/libtcmalloc_minimal*
> lrwxrwxrwx 1 root root 28 Feb 4 21:49 /usr/lib/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.1.2
> lrwxrwxrwx 1 root root 34 Feb 4 21:49 /usr/lib/libtcmalloc_minimal_debug.so.4 -> libtcmalloc_minimal_debug.so.4.1.2
> -rw-r--r-- 1 root root 155000 Feb 4 21:49 /usr/lib/libtcmalloc_minimal.so.4.1.2
> -rw-r--r-- 1 root root 204184 Feb 4 21:49 /usr/lib/libtcmalloc_minimal_debug.so.4.1.2
>
> $> file /usr/lib/libtcmalloc_minimal.so.4.1.2
> /usr/lib/libtcmalloc_minimal.so.4.1.2: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=3f5d8c7638476e345c4f9c0345c4506179f186fe, stripped
>
>
> but when I test whether Moses links it, with these commands (as suggested by Kenneth (see his mail of November 18, 2012 7:04:39 PM)
>
> $> bash -c "g++ -ltcmalloc_minimal -x c++ - <<<'int main() {}' -o /home/ubuntu/Software/MOSES/mosesdecoder-dynamic-models/dummy >/dev/null 2>/dev/null && rm /home/ubuntu/Software/MOSES/mosesdecoder-dynamic-models/dummy 2>/dev/null"
> $> echo$?
>
> I gor the value "1", instead of 0, i.e. error
>
>
> What am I wrong?
>
> cheers,
> Nicola
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support



------------------------------

Message: 5
Date: Mon, 28 Jul 2014 13:53:55 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Predetermined translations in training
data
To: Roee Aharoni <roee.aharoni@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbioVKSEeDmt4sRRoSX6MRPb+T3iaZ838o+OXav7SX7Gug@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

the training doesn't know anything about the XML tags. You must remove them
during training


On 26 July 2014 10:05, Roee Aharoni <roee.aharoni@gmail.com> wrote:

> Hi all,
>
> We use the predetermined translations feature in our system, using the <n
> translation="..."> XML tags. My question is whether the training script
> train-moses.perl knows to handle or ignore those tags, or what are the
> consequences of using those tags in the training data.
>
> Thanks in advance,
> Roee
> Roee
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140728/39d825d0/attachment-0001.htm

------------------------------

Message: 6
Date: Mon, 28 Jul 2014 14:15:15 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Issue with tc mallo con Ubuntu-14.04
To: Nicola Bertoldi <bertoldi@fbk.eu>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbgX8gKaqNm_1G106mbM7aoK41ZQZnU6P6KOcDBXYoX14g@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Looking at the listing in your 1st email, I think you should do what Tom
Hoar advised:
install libgoogle-perftools-dev.
On my system:
# ls /usr/lib/libtcmalloc_minimal*
/usr/lib/libtcmalloc_minimal.a
/usr/lib/libtcmalloc_minimal_debug.so.4
/usr/lib/libtcmalloc_minimal.so.4
/usr/lib/libtcmalloc_minimal_debug.a
/usr/lib/libtcmalloc_minimal_debug.so.4.1.2
/usr/lib/libtcmalloc_minimal.so.4.1.2
/usr/lib/libtcmalloc_minimal_debug.so /usr/lib/libtcmalloc_minimal.so




On 28 July 2014 14:03, Nicola Bertoldi <bertoldi@fbk.eu> wrote:

> I only have the dynamic library for tcmalloc.
>
> Do I need the static library as well? How can I install them?
>
> But actually you remove them and
> I am bit confused...
>
>
> Nicola
>
> On Jul 28, 2014, at 2:49 PM, Hieu Hoang wrote:
>
> i think the bjam build does link to tcmalloc. When I do
> ./bjam ... -d2
> The link command is
> g++ ...-Bstatic ... -ltcmalloc_minimal
> This looks like it links statically to tcmalloc, so you won't be able to
> see it when you execute
> ldd bin/moses
>
> However, if i delete the static libraries
> sudo rm libtcmalloc*.a
> and recompile, then i can see it:
> #ldd bin/moses
> ...
> libtcmalloc_minimal.so.4 => /usr/lib/libtcmalloc_minimal.so.4
> (0x00007fd4a3441000)
> ...
>
>
>
> On 28 July 2014 13:18, Nicola Bertoldi <bertoldi@fbk.eu> wrote:
>
>> I've just found this
>> https://www.mail-archive.com/moses-support@mit.edu/msg10244.html
>>
>> it is still unsolved?
>> I mean, no way to have tcmalloc (hence speed-up) on ubuntu 14.04
>> where is the issue?
>>
>> cheers,
>> Nicola
>>
>>
>> Begin forwarded message:
>>
>> *From: *Nicola Bertoldi <bertoldi@fbk.eu>
>> *Date: *July 28, 2014 10:08:25 AM GMT+02:00
>> *To: *moses-support support <moses-support@mit.edu>
>> *Cc: *Nicola Bertoldi <bertoldi@fbk.eu>
>> *Subject: **Issue with tc mallo con Ubuntu-14.04*
>>
>> hi guys
>>
>> I am compiling moses on Ubuntu-14.04, but I cannot link it with tcmalloc
>> correctly
>>
>> I installed the tcmalloc lib with the command
>> sudo apt install libtcmalloc-minimal4
>> and it went well, because the dynamic library exists as shown here
>>
>> $> ls -ltr /usr/lib/libtcmalloc_minimal*
>> lrwxrwxrwx 1 root root 28 Feb 4 21:49
>> /usr/lib/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.1.2
>> lrwxrwxrwx 1 root root 34 Feb 4 21:49
>> /usr/lib/libtcmalloc_minimal_debug.so.4 ->
>> libtcmalloc_minimal_debug.so.4.1.2
>> -rw-r--r-- 1 root root 155000 Feb 4 21:49
>> /usr/lib/libtcmalloc_minimal.so.4.1.2
>> -rw-r--r-- 1 root root 204184 Feb 4 21:49
>> /usr/lib/libtcmalloc_minimal_debug.so.4.1.2
>>
>> $> file /usr/lib/libtcmalloc_minimal.so.4.1.2
>> /usr/lib/libtcmalloc_minimal.so.4.1.2: ELF 64-bit LSB shared object,
>> x86-64, version 1 (SYSV), dynamically linked,
>> BuildID[sha1]=3f5d8c7638476e345c4f9c0345c4506179f186fe, stripped
>>
>>
>> but when I test whether Moses links it, with these commands (as suggested
>> by Kenneth (see his mail of November 18, 2012 7:04:39 PM)
>>
>> $> bash -c "g++ -ltcmalloc_minimal -x c++ - <<<'int main() {}' -o
>> /home/ubuntu/Software/MOSES/mosesdecoder-dynamic-models/dummy >/dev/null
>> 2>/dev/null && rm
>> /home/ubuntu/Software/MOSES/mosesdecoder-dynamic-models/dummy 2>/dev/null"
>> $> echo$?
>>
>> I gor the value "1", instead of 0, i.e. error
>>
>>
>> What am I wrong?
>>
>> cheers,
>> Nicola
>>
>>
>>
>>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140728/f88ab5de/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 93, Issue 35
*********************************************

0 Response to "Moses-support Digest, Vol 93, Issue 35"

Post a Comment

Subscribe to: Post Comments (Atom)