Moses-support Digest, Vol 103, Issue 61

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Incremental Giza(Inc-giza) crashes with core dump when
doing Incremental training using MOSES EMS (Hieu Hoang)
2. ERROR while installinh Moses (Harshit Gupta)
3. How to use EMS to test pre build model (pankaj singh)


----------------------------------------------------------------------

Message: 1
Date: Mon, 25 May 2015 08:53:21 +0400
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Incremental Giza(Inc-giza) crashes with
core dump when doing Incremental training using MOSES EMS
To: "Hegde, Sujay" <Sujay.Hegde@xerox.com>, "moses-support@mit.edu"
<moses-support@mit.edu>
Cc: "Venkatapathy, Sriram \(Calendar\)"
<sriram.venkatapathy@xrce.xerox.com>, "MudaliarMudaliar, Preeti J"
<preeti.mudaliarmudaliar@xerox.com>
Message-ID: <5562AAC1.6090305@gmail.com>
Content-Type: text/plain; charset="windows-1252"

seems like no-one knows. If you managed to figure it out, please tell us

On 20/05/2015 14:32, Hegde, Sujay wrote:
>
> Hi Admin,
>
> Inc-giza taken from : https://code.google.com/p/inc-giza-pp/
> <https://code.google.com/p/inc-giza-pp/>
>
> And manually compiled.
>
> The command invoked by Moses EMS(actually train-model.perl ) is :
>
> /mnt/hd1/git/mosesdecoder/training-tools/GIZA++ -CoocurrenceFile
> /mnt/hd1/training/working-dir-en-es/training/giza.5/en-es.cooc -c
> /mnt/hd1/training/working-dir-en-es/training/prepared.5/en-es-int-train.snt
> -hmmdumpfrequency 5 -hmmiterations 5 -m1 5 -m2 0 -m3 0 -m4 0 -m5 0
> -model1dumpfrequency 0 -model2dumpfrequency 0 -model345dumpfrequency 0
> -model4smoothfactor 0.4 -nodumps 0 -nsmooth 4 -o
> /mnt/hd1/training/working-dir-en-es/training/giza.5/en-es -oldAlPrbs
> /mnt/hd1/training/working-dir-en-es/training/giza.1/en-es.hhmm.5
> -oldTrPrbs
> /mnt/hd1/training/working-dir-en-es/training/giza.1/en-es.thmm.5
> -onlyaldumps 1 -p0 0.999 -s
> /mnt/hd1/training/working-dir-en-es/training/prepared.5/es.vcb -step_k
> 1 -t /mnt/hd1/training/working-dir-en-es/training/prepared.5/en.vcb
>
> The baseline model used is 1.
>
> It crashes with below output on console.It is a HEAP CORRPUTION.
>
> -----------
>
> Model1: Iteration 1
>
> number of French (target) words = 1255
>
> initial unifrom prob = 0.000796813
>
> Model1: (1) TRAIN CROSS-ENTROPY 16.2367 PERPLEXITY 77219
>
> Model1: (1) VITERBI TRAIN CROSS-ENTROPY 20.8406 PERPLEXITY 1.87773e+06
>
> Model 1 Iteration: 1 took: 0 seconds
>
> -----------
>
> Model1: Iteration 2
>
> number of French (target) words = 1255
>
> initial unifrom prob = 0.000796813
>
> Model1: (2) TRAIN CROSS-ENTROPY 5.75346 PERPLEXITY 53.9466
>
> Model1: (2) VITERBI TRAIN CROSS-ENTROPY 9.11781 PERPLEXITY 555.562
>
> Model 1 Iteration: 2 took: 0 seconds
>
> -----------
>
> Model1: Iteration 3
>
> number of French (target) words = 1255
>
> initial unifrom prob = 0.000796813
>
> Model1: (3) TRAIN CROSS-ENTROPY 5.61082 PERPLEXITY 48.868
>
> Model1: (3) VITERBI TRAIN CROSS-ENTROPY 8.86927 PERPLEXITY 467.646
>
> Model 1 Iteration: 3 took: 0 seconds
>
> -----------
>
> Model1: Iteration 4
>
> number of French (target) words = 1255
>
> initial unifrom prob = 0.000796813
>
> Model1: (4) TRAIN CROSS-ENTROPY 5.51417 PERPLEXITY 45.7014
>
> Model1: (4) VITERBI TRAIN CROSS-ENTROPY 8.64774 PERPLEXITY 401.078
>
> Model 1 Iteration: 4 took: 0 seconds
>
> -----------
>
> Model1: Iteration 5
>
> number of French (target) words = 1255
>
> initial unifrom prob = 0.000796813
>
> Model1: (5) TRAIN CROSS-ENTROPY 5.44065 PERPLEXITY 43.4309
>
> Model1: (5) VITERBI TRAIN CROSS-ENTROPY 8.44812 PERPLEXITY 349.25
>
> Model 1 Iteration: 5 took: 0 seconds
>
> Entire Model1 Training took: 0 seconds
>
> NOTE: I am doing iterations with the HMM model!
>
> Loading HMM alignments from file.
>
> **** glibc detected ***
> /mnt/hd1/git/mosesdecoder/training-tools/GIZA++: malloc(): memory
> corruption: 0x00000000011c4470 ****
>
> ======= Backtrace: =========
>
> [0x524d22]
>
> [0x526f4d]
>
> [0x527b7b]
>
> [0x4f0fcd]
>
> [0x4d2829]
>
> [0x4d329a]
>
> [0x4d340b]
>
> [0x495c39]
>
> [0x48bdf2]
>
> [0x42822f]
>
> [0x4294ba]
>
> [0x505e66]
>
> [0x4001e9]
>
> ======= Memory map: ========
>
> 00400000-005d9000 r-xp 00000000 08:10 6030184
> /mnt/hd1/git/mosesdecoder/training-tools/GIZA++
>
> 007d8000-007db000 rw-p 001d8000 08:10 6030184
> /mnt/hd1/git/mosesdecoder/training-tools/GIZA++
>
> 007db000-007f2000 rw-p 00000000 00:00 0
>
> 00fc4000-01438000 rw-p 00000000 00:00
> 0 [heap]
>
> 7f3054000000-7f3054028000 rw-p 00000000 00:00 0
>
> 7f3054028000-7f3058000000 ---p 00000000 00:00 0
>
> 7f3059db4000-7f3059db5000 rw-p 00000000 00:00 0
>
> 7fff689b0000-7fff689c5000 rw-p 00000000 00:00
> 0 [stack]
>
> 7fff689ff000-7fff68a00000 r-xp 00000000 00:00
> 0 [vdso]
>
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00
> 0 [vsyscall]
>
> Aborted (core dumped)
>
> Also core analysis using GDB yields:
>
> Core was generated by `/mnt/hd1/git/mosesdecoder/training-tools/GIZA++
> -CoocurrenceFile /mnt/hd1/train'.
>
> *Program terminated with signal 6, Aborted.*
>
> #0 0x0000000000547ee5 in raise ()
>
> (gdb) where
>
> #0 0x0000000000547ee5 in raise ()
>
> #1 0x000000000050e635 in abort ()
>
> #2 0x000000000051f365 in __libc_message ()
>
> #3 0x0000000000524d22 in malloc_printerr ()
>
> #4 0x0000000000526f4d in _int_malloc ()
>
> #5 0x0000000000527b7b in malloc ()
>
> #6 0x00000000004f0fcd in operator new(unsigned long) ()
>
> #7 0x00000000004d2829 in std::basic_string<char,
> std::char_traits<char>, std::allocator<char>
> >::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char>
> const&) ()
>
> #8 0x00000000004d329a in char* std::basic_string<char,
> std::char_traits<char>, std::allocator<char> >::_S_construct<char
> const*>(char const*, char const*, std::allocator<char> const&,
> std::forward_iterator_tag) ()
>
> #9 0x00000000004d340b in std::basic_string<char,
> std::char_traits<char>, std::allocator<char> >::basic_string(char
> const*, unsigned long, std::allocator<char> const&) ()
>
> #10 0x0000000000495c39 in HMMTables<int,
> WordClasses>::readJumps(std::basic_istream<char,
> std::char_traits<char> >&) ()
>
> #11 0x000000000048bdf2 in hmm::load_table(char const*) ()
>
> #12 0x000000000042822f in StartTraining(int&) ()
>
> #13 0x00000000004294ba in main ()
>
> Is it a problem with inc-Giza or could it be a problem with the libs
> being used.
>
> Regards,
>
> Sujay
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

--
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150525/acb093b6/attachment-0001.htm

------------------------------

Message: 2
Date: Mon, 25 May 2015 13:14:11 +0530
From: Harshit Gupta <harshitgupta165@gmail.com>
Subject: [Moses-support] ERROR while installinh Moses
To: moses-support@mit.edu
Message-ID:
<CAHgj_vv+5HhmC-vsMiymRL0dAJDgTzv-P_U9sT4naCyhGyYHkQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

PFA build.log.gz

The command which I run was
*./bjam -j8*

--
Harshit Gupta
Third Year Undergraduate
Electrical Engineering
IIT Madras
P* Please don't print this e-mail unless you really need to*PFA
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150525/4f6fbdfb/attachment-0001.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log.gz
Type: application/x-gzip
Size: 662 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20150525/4f6fbdfb/attachment-0001.bin

------------------------------

Message: 3
Date: Mon, 25 May 2015 15:46:26 +0530
From: pankaj singh <pr.pankajsingh@gmail.com>
Subject: [Moses-support] How to use EMS to test pre build model
To: moses-support@mit.edu
Cc: Himanshu Ojha <himanshuojha.lko@gmail.com>, Ratish Puduppully
<ratish123@gmail.com>
Message-ID:
<CAEsBN1D5hK4AQcbotF+zzintJPNcWU_ONfbMsjF_DeggF3fLBw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hello,

I want to use pre build model (LINK:
http://www.statmt.org/moses/RELEASE-3.0/models/en-fr/ ).
I followed instruction mentioned in release note (page 5 of
http://www.statmt.org/mosescore/uploads/Internal/D1.4_Moses_v3_Release_Notes.pdf
)

Now I have following files ready with me:

- europarl.binlm.1
- moses.tuned.ini
- phrase-table.minphr
- reordering-table.minlexr

I want to use this binarised model to test it on separate parallel test
corpus.

*I want to use EMS for this. How to change the config file for this so that
it can use existing binariesed model and it should not retrain/
tune/prepare corpus etc. It should only Evaluate, analyze and report.*


*Thanks and regards,*

*Pankaj Singh*
*CSE Department of IIT Bombay*
*Thanks and regards*
*Pankaj Singh*
*+918454992911*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150525/592b96bb/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 103, Issue 61
**********************************************

Tweet