Moses-support Digest, Vol 97, Issue 52

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: mgiza++ force alignment: segmentation fault when
reloading a big N table (Eleftherios Avramidis)
2. Re: mgiza++ force alignment: segmentation fault when
reloading a big N table (Hala Almaghout)


----------------------------------------------------------------------

Message: 1
Date: Wed, 19 Nov 2014 12:43:18 +0200
From: Eleftherios Avramidis <eleftherios.avramidis@dfki.de>
Subject: Re: [Moses-support] mgiza++ force alignment: segmentation
fault when reloading a big N table
To: Hala Maghout <hala.maghout@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <546C7446.7080709@dfki.de>
Content-Type: text/plain; charset="utf-8"

Hi Hala,

I never found a solution to this since I had a strict deadline so I
stopped using mgiza and went back to IBM-model1. Thanks for reminding,
cause I may need it in the future. In case it helps. I have opened this
issue here https://github.com/moses-smt/mgiza/issues/2

Please go there and add you feedback as well.

What is your operating system? I have some suspicion that this may be
due to some outdated Ubuntu libraries.

best
Lefteris

On 19.11.2014 12:34, Hala Maghout wrote:
> Hi,
>
> I'm facing a segmentation fault problem when loading big N tables
> during forced alignment using MGIZA, which was posted previously on
> moses list (thread below) but no solution was suggested. As Lefteris
> explained, it's due to big N table size. Any suggestion on how to
> solve it other than cutting our entries from the N table?
>
> Many thanks,
>
> Best,
>
> Hala
>
>
> On 11 August 2014 11:03, Hieu Hoang <Hieu.Hoang@ed.ac.uk
> <mailto:Hieu.Hoang@ed.ac.uk>> wrote:
>
> did you manage to solve this issue? I tried contacting Qin Gao but
> there's been no reply so far.
>
> From my experience with mgiza a while ago, force alignment works ok
>
>
> On 3 August 2014 23:34, Eleftherios Avramidis
> <Eleftherios.Avramidis@dfki.de
> <mailto:Eleftherios.Avramidis@dfki.de>> wrote:
>
> Hi,
>
> I am trying to produce word alignment for individual
> sentences. For this purpose I am using the "force align"
> functionality of mgiza++ Unfortunately when I am loading a big
> N table (fertility), mgiza crashes with a segmentation fault.
>
> In particular, I have initially run mgiza on the full training
> parallel corpus using the default settings of the Moses script:
>
> /project/qtleap/software/moses-2.1.1/bin/training-tools/mgiza -CoocurrenceFile /local/tmp/elav01/selection-mechanism/systems/de-en/training/giza.1/en-de.cooc -c /local/tmp/elav01/selection-mechanism/systems/de-en/training/prepared.1/en-de-int-train.snt -m1 5 -m2 0 -m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 -ncpus 24 -nodumps 0 -nsmooth 4 -o /local/tmp/elav01/selection-mechanism/systems/de-en/training/giza.1/en-de -onlyaldumps 0 -p0 0.999 -s /local/tmp/elav01/selection-mechanism/systems/de-en/training/prepared.1/de.vcb -t /local/tmp/elav01/selection-mechanism/systems/de-en/training/prepared.1/en.vcb
>
> Afterwards, by executing the mgiza force-align script, I run
> the following command
>
> /project/qtleap/software/moses-2.1.1/mgizapp-code/mgizapp//bin/mgiza giza.en-de/en-de.gizacfg -c /local/tmp/elav01/selection-mechanism/systems/de-en/falign/qtmp_SOVBrE/prepared./en-de.snt -o /local/tmp/elav01/selection-mechanism/systems/de-en/falign/qtmp_SOVBrE/giza./en-de -s /local/tmp/elav01/selection-mechanism/systems/de-en/falign/qtmp_SOVBrE/prepared./de.vcb -t /local/tmp/elav01/selection-mechanism/systems/de-en/falign/qtmp_SOVBrE/prepared./en.vcb -m1 0 -m2 0 -mh 0 -coocurrence /local/tmp/elav01/selection-mechanism/systems/de-en/falign/qtmp_SOVBrE/giza./en-de.cooc -restart 11 -previoust giza.en-de/en-de.t3.final -previousa giza.en-de/en-de.a3.final -previousd giza.en-de/en-de.d3.final -previousn giza.en-de/en-de.n3.final -previousd4 giza.en-de/en-de.d4.final -previousd42 giza.en-de/en-de.D4.final -m3 0 -m4 1
>
> This runs fine, until I get the following error:
>
> We are going to load previous N model from giza.en-de/en-de.n3.final
>
> Reading fertility table from giza.en-de/en-de.n3.final
>
> Segmentation fault (core dumped)
>
> The n-table that is failing has about 300k entries. For this
> reason, I thought I should try to see if the size is a
> problem. So I concatenated the table to 60k entries. And it
> works! But the alignments are not good.
>
> I am struggling to fix this, so any help would be appreciated.
> I am running a freshly installed mgiza, on Ubuntu 12.04
>
> cheers,
> Lefteris
>
> --
> MSc. Inf. Eleftherios Avramidis
> DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
> Tel.+49-30 238 95-1806 <tel:%2B49-30%20238%2095-1806>
>
> Fax.+49-30 238 95-1810 <tel:%2B49-30%20238%2095-1810>
>
> -------------------------------------------------------------------------------------------
> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>
> Geschaeftsfuehrung:
> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
> Dr. Walter Olthoff
>
> Vorsitzender des Aufsichtsrats:
> Prof. Dr. h.c. Hans A. Aukes
>
> Amtsgericht Kaiserslautern, HRB 2313
> -------------------------------------------------------------------------------------------
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141119/8831607d/attachment-0001.htm

------------------------------

Message: 2
Date: Wed, 19 Nov 2014 11:33:14 +0000
From: Hala Almaghout <halmaghout@computing.dcu.ie>
Subject: Re: [Moses-support] mgiza++ force alignment: segmentation
fault when reloading a big N table
To: Eleftherios Avramidis <eleftherios.avramidis@dfki.de>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAE9fu9VmbNaybOYvYVJOKndPaKTzYNcLCtO0se0k=+XBuoV7JA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Lefteris,

I commented in the issue you opened on github. The OS I'm running MGIZA on
is openSUSE 12.2

Best,

Hala

On 19 November 2014 10:43, Eleftherios Avramidis <
eleftherios.avramidis@dfki.de> wrote:

> Hi Hala,
>
> I never found a solution to this since I had a strict deadline so I
> stopped using mgiza and went back to IBM-model1. Thanks for reminding,
> cause I may need it in the future. In case it helps. I have opened this
> issue here https://github.com/moses-smt/mgiza/issues/2
>
> Please go there and add you feedback as well.
>
> What is your operating system? I have some suspicion that this may be due
> to some outdated Ubuntu libraries.
>
> best
> Lefteris
>
>
> On 19.11.2014 12:34, Hala Maghout wrote:
>
> Hi,
>
> I'm facing a segmentation fault problem when loading big N tables during
> forced alignment using MGIZA, which was posted previously on moses list
> (thread below) but no solution was suggested. As Lefteris explained, it's
> due to big N table size. Any suggestion on how to solve it other than
> cutting our entries from the N table?
>
> Many thanks,
>
> Best,
>
> Hala
>
>
> On 11 August 2014 11:03, Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:
>
>> did you manage to solve this issue? I tried contacting Qin Gao but
>> there's been no reply so far.
>>
>> From my experience with mgiza a while ago, force alignment works ok
>>
>>
>> On 3 August 2014 23:34, Eleftherios Avramidis <
>> Eleftherios.Avramidis@dfki.de> wrote:
>>
>>> Hi,
>>>
>>> I am trying to produce word alignment for individual sentences. For this
>>> purpose I am using the "force align" functionality of mgiza++ Unfortunately
>>> when I am loading a big N table (fertility), mgiza crashes with a
>>> segmentation fault.
>>>
>>> In particular, I have initially run mgiza on the full training parallel
>>> corpus using the default settings of the Moses script:
>>>
>>> /project/qtleap/software/moses-2.1.1/bin/training-tools/mgiza -CoocurrenceFile /local/tmp/elav01/selection-mechanism/systems/de-en/training/giza.1/en-de.cooc -c /local/tmp/elav01/selection-mechanism/systems/de-en/training/prepared.1/en-de-int-train.snt -m1 5 -m2 0 -m3 3 -m4 3 -model1dumpfrequency 1 -model4smoothfactor 0.4 -ncpus 24 -nodumps 0 -nsmooth 4 -o /local/tmp/elav01/selection-mechanism/systems/de-en/training/giza.1/en-de -onlyaldumps 0 -p0 0.999 -s /local/tmp/elav01/selection-mechanism/systems/de-en/training/prepared.1/de.vcb -t /local/tmp/elav01/selection-mechanism/systems/de-en/training/prepared.1/en.vcb
>>>
>>> Afterwards, by executing the mgiza force-align script, I run the
>>> following command
>>>
>>> /project/qtleap/software/moses-2.1.1/mgizapp-code/mgizapp//bin/mgiza giza.en-de/en-de.gizacfg -c /local/tmp/elav01/selection-mechanism/systems/de-en/falign/qtmp_SOVBrE/prepared./en-de.snt -o /local/tmp/elav01/selection-mechanism/systems/de-en/falign/qtmp_SOVBrE/giza./en-de -s /local/tmp/elav01/selection-mechanism/systems/de-en/falign/qtmp_SOVBrE/prepared./de.vcb -t /local/tmp/elav01/selection-mechanism/systems/de-en/falign/qtmp_SOVBrE/prepared./en.vcb -m1 0 -m2 0 -mh 0 -coocurrence /local/tmp/elav01/selection-mechanism/systems/de-en/falign/qtmp_SOVBrE/giza./en-de.cooc -restart 11 -previoust giza.en-de/en-de.t3.final -previousa giza.en-de/en-de.a3.final -previousd giza.en-de/en-de.d3.final -previousn giza.en-de/en-de.n3.final -previousd4 giza.en-de/en-de.d4.final -previousd42 giza.en-de/en-de.D4.final -m3 0 -m4 1
>>>
>>> This runs fine, until I get the following error:
>>>
>>> We are going to load previous N model from giza.en-de/en-de.n3.final
>>>
>>> Reading fertility table from giza.en-de/en-de.n3.final
>>>
>>> Segmentation fault (core dumped)
>>>
>>>
>>> The n-table that is failing has about 300k entries. For this reason, I
>>> thought I should try to see if the size is a problem. So I concatenated the
>>> table to 60k entries. And it works! But the alignments are not good.
>>>
>>> I am struggling to fix this, so any help would be appreciated. I am
>>> running a freshly installed mgiza, on Ubuntu 12.04
>>>
>>> cheers,
>>> Lefteris
>>>
>>> --
>>> MSc. Inf. Eleftherios Avramidis
>>> DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
>>> Tel. +49-30 238 95-1806
>>>
>>> Fax. +49-30 238 95-1810
>>>
>>> -------------------------------------------------------------------------------------------
>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>>
>>> Geschaeftsfuehrung:
>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>>> Dr. Walter Olthoff
>>>
>>> Vorsitzender des Aufsichtsrats:
>>> Prof. Dr. h.c. Hans A. Aukes
>>>
>>> Amtsgericht Kaiserslautern, HRB 2313
>>> -------------------------------------------------------------------------------------------
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>>
>> --
>> Hieu Hoang
>> Research Associate
>> University of Edinburgh
>> http://www.hoang.co.uk/hieu
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141119/23ec1e01/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 97, Issue 52
*********************************************

Tweet