Moses-support Digest, Vol 126, Issue 10

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: inconsistent number of probabilities (Alexander Erdmann)


----------------------------------------------------------------------

Message: 1
Date: Fri, 7 Apr 2017 17:58:37 +0400
From: Alexander Erdmann <ae1541@nyu.edu>
Subject: Re: [Moses-support] inconsistent number of probabilities
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAGTOGBfLmz9+yWJ0bXrMPjWXOPvSLsL9OvrG_7q3_ZBY2+5UZQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Just for reference, The issue was that I didn't filter the phrase table
beforehand. based on how the model was trained, I knew this wouldn't reduce
the size of the phrase table enough to make it worthwhile (or so I
thought), but as soon as I filtered the phrase table it stopped segFaulting.

On Fri, Apr 7, 2017 at 4:11 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:

> The moses.ini file looks ok.
>
> The segfault is difficult to debug by inspection. It's best sprinkle the
> decoder code with debugging message and run 1 instance of the decoder to
> see exactly where it goes wrong. And you should binarise the pt and
> reordering models to save yourself time
>
> * Looking for MT/NLP opportunities *
> Hieu Hoang
> http://moses-smt.org/
>
>
> On 6 April 2017 at 11:30, Alexander Erdmann <ae1541@nyu.edu> wrote:
>
>> For reference, here is the output from the segFault:
>>
>> Defined parameters (per moses.ini or switch):
>> config: /scratch/ae1541/EMNLP/seed/150000/iteration1/mosesTryingGZ.ini
>> distortion-limit: 6
>> feature: UnknownWordPenalty WordPenalty PhrasePenalty
>> PhraseDictionaryMemory name=TranslationModel0 num-features=4
>> path=/scratch/ae1541/EMNLP/seed/150000/iteration0/phrase-table.gz
>> LexicalReordering name=LexicalReordering0 num-features=6
>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>> path=/scratch/ae1541/EMNLP/seed/150000/iteration0/reordering-table.gz
>> Distortion KENLM name=LM0 factor=0 path=/scratch/ae1541/EMNLP/seed/LM/LM.lev.lm
>> order=5
>> input-factors: 0
>> input-file: seed/150000/iteration1/del00.egy
>> mapping: 0 T 0
>> threads: 12
>> weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
>> TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3
>> 0.3 Distortion0= 0.3 LM0= 0.5
>> line=UnknownWordPenalty
>> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
>> line=WordPenalty
>> FeatureFunction: WordPenalty0 start: 1 end: 1
>> line=PhrasePenalty
>> FeatureFunction: PhrasePenalty0 start: 2 end: 2
>> line=PhraseDictionaryMemory name=TranslationModel0 num-features=4
>> path=/scratch/ae1541/EMNLP/seed/150000/iteration0/phrase-table.gz
>> FeatureFunction: TranslationModel0 start: 3 end: 6
>> line=LexicalReordering name=LexicalReordering0 num-features=6
>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>> path=/scratch/ae1541/EMNLP/seed/150000/iteration0/reordering-table.gz
>> FeatureFunction: LexicalReordering0 start: 7 end: 12
>> Initializing LexicalReordering..
>> line=Distortion
>> FeatureFunction: Distortion0 start: 13 end: 13
>> line=KENLM name=LM0 factor=0 path=/scratch/ae1541/EMNLP/seed/LM/LM.lev.lm
>> order=5
>> FeatureFunction: LM0 start: 14 end: 14
>> Loading the LM will be faster if you build a binary file.
>> Reading /scratch/ae1541/EMNLP/seed/LM/LM.lev.lm
>> ----5---10---15---20---25---30---35---40---45---50---55---60
>> ---65---70---75---80---85---90---95--100
>> ************************************************************
>> ****************************************
>> Loading UnknownWordPenalty0
>> Loading WordPenalty0
>> Loading PhrasePenalty0
>> Loading LexicalReordering0
>> Loading table into memory...done.
>> Loading Distortion0
>> Loading LM0
>> Loading TranslationModel0
>> Start loading text phrase table. Moses format : [1574.737] seconds
>> Reading /scratch/ae1541/EMNLP/seed/150000/iteration0/phrase-table.gz
>> ----5---10---15---20---25---30---35---40---45---50---55---60
>> ---65---70---75---80---85---90---95--100
>> ************************************************************
>> ****************************************
>> Created input-output object : [1608.439] seconds
>> Translating: Translating: A y Translating: jAC Translating: Translating:
>> AbwTranslating: RRR w hA Al mAd mrwAn
>> sAdRRR jrydp A m trtr . . jrydp ywmy p . . bTyx yp . . mhlb yp bAl
>> tEycTranslating: w nA Al byAn Al tAly p
>> Line 1: Initialize search took Line 8: Initialize search took tqly p
>> 0.006 seconds total
>> 0.005 seconds total
>> Translating: t h y Translating: qwqAl A b mSr RRR rytc bAl Al dAxly p y
>> AxbAr ?
>> Line 9: Initialize search took 0.007 seconds total
>> w b h mfyc RRR
>> AxwAn hm Hd m Al mslmyn AnDrb
>> Line 11: Initialize search took Line 0: Initialize search took Al0.010
>> seconds total
>> Hrp 0.009 seconds total
>> Al Translating: AfAd Al mhlb jrydpmtHdv Al rsmy l wzArp Al nSAry bEd
>> Translating: yp Al m m A y h Avl Al dAxly w
>> An p AfAdt Line 7: Initialize search took qyAd ( Line xyr p ll TbE
>> Al mSlHp 8: Collecting options took Line 2: Initialize search took lhm Al
>> 0.005 seconds at moses/Manager.cpp:AjElh bHv 0.015 seconds total
>> 0.013 seconds total
>> wrAC Amn xyr mvyr Al Al y dwlp tlyfzywn Al sAbq Al mSry A ) Almbjl ytHdv
>> b Al An lsAn jnAQy h jrydp An bEd Al ftn A tHry w m w Al trtr Al bHv w
>> mtTrf f yTlb yn y mn Al AHdAv Al mslmyn
>> hm Line 6: Initialize search took 117
>> nA 0.020 seconds total
>> s *** bSegmentation fault
>> Register dump:
>>
>> RAX: 0000000000000000 RBX: 00000000012e28e0 RCX: 0000000000000000
>> RDX: 00007f91b7413010 RSI: 0000000000000000 RDI: 00007f917400b9c0
>> RBP: 0000000000000000 R8 : 0000000000000002 R9 : 0000000000000038
>> R10: 00007f91aa8f3310 R11: 00007f91b64cfbf0 R12: 00007f917400b028
>> R13: 00007f9174004f60 R14: 00007f917400b9c0 R15: 0000000000000002
>> RSP: 00007f91aa8f3730
>>
>> RIP: 00007f91b66e8833 EFLAGS: 00010246
>>
>> CS: 0033 FS: 0000 GS: 0000
>>
>> Trap: 0000000e Error: 00000004 OldMask: 00000000 CR2: 00000010
>>
>> FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000
>> RIP: 00000000 RDP: 00000000
>>
>> ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000
>> ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000
>> ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000
>> ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000
>> mxcsr: 1fa4
>> XMM0: 00000000000000000000000000000000 XMM1:
>> 00000000000000000000000000000000
>> XMM2: 00000000000000000000000000000000 XMM3:
>> 00000000000000000000000000000000
>> XMM4: 00000000000000000000000000000000 XMM5:
>> 00000000000000000000000000000000
>> XMM6: 00000000000000000000000000000000 XMM7:
>> 00000000000000000000000000000000
>> XMM8: 00000000000000000000000000000000 XMM9:
>> 00000000000000000000000000000000
>> XMM10: 00000000000000000000000000000000 XMM11:
>> 00000000000000000000000000000000
>> XMM12: 00000000000000000000000000000000 XMM13:
>> 00000000000000000000000000000000
>> XMM14: 00000000000000000000000000000000 XMM15:
>> 00000000000000000000000000000000
>>
>> Backtrace:
>> y Sdd lA tjmyE hyj rAs wA AHd/share/apps/NYUAD/mosesdeco
>> der/3.0/lib/libmoses.so(_ZNK5Moses16LanguageModelKenIN2lm5ng
>> ram12ProbingModelEE19EvaluateWhenAppliedERKNS_10HypothesisEP
>> KNS_7FFStateEPNS_24ScoreComponentCollectionE+0xa3)[0x7f91b66e8833]
>> Ely/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5M
>> oses10Hypothesis19EvaluateWhenAppliedERKNS_12SquareMatrixE+
>> 0xf0)[0x7f91b64cfab0]
>> Al /share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses
>> 12SearchNormal16ExpandHypothesisERKNS_10HypothesisERKNS_17Tr
>> anslationOptionEf+0x165)[0x7f91b65484a5]
>> bED /share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses
>> 12SearchNormal19ExpandAllHypothesesERKNS_10HypothesisEmm+
>> 0x79)[0x7f91b6548829]
>>
>> mALine /share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses
>> 12SearchNormal20ProcessOneHypothesisERKNS_10HypothesisE+
>> 0x34b)[0x7f91b6548beb]
>> 10: Initialize search took /share/apps/NYUAD/mosesdecoder
>> /3.0/lib/libmoses.so(_ZN5Moses12SearchNormal6DecodeEv+0x11c)
>> [0x7f91b6548ffc]
>> DHAyA/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_
>> ZN5Moses7Manager6DecodeEv+0xe3)[0x7f91b64f9173]
>> sbyrw /share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses
>> 15TranslationTask3RunEv+0x1a5)[0x7f91b656f505]
>> f /share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses
>> 10ThreadPool7ExecuteEv+0x89)[0x7f91b6724b99]
>> ./share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57
>> .0/lib/libboost_thread.so.1.57.0(+0x1211a)[0x7f91b54ee11a]
>> /usr/lib64/libpthread.so.0(+0x7dc5)[0x7f91b489cdc5]
>> y /usr/lib64/libc.so.6(clone+0x6d)[0x7f91b45ca28d]
>> .
>> Memory map:
>>
>> tlk 00400000-0040b000 r-xp 00000000 00:2b 1254532671
>> /share/apps/NYUAD3/mosesdecoder/3.0/bin/moses
>> 0060b000-0060c000 rw-p 0000b000 00:2b 1254532671
>> /share/apps/NYUAD3/mosesdecoder/3.0/bin/moses
>> 012ce000-52974a000qd rw-p 00000000 00:00 0
>> [heap]
>> 7f9174000000-7f9174021000 rw-p 00000000 00:00 0
>> 7f9174021000-7f9178000000 ---p 00000000 00:00 0
>> 7f9178000000-7f917807b000 rw-p 00000000 00:00 0
>> 7f917807b000-7f917c000000 ---p 00000000 00:00 0
>> 7f917c000000-7f917c034000 rw-p 00000000 00:00 0
>> 7f917c034000-7f9180000000 ---p 00000000 00:00 0
>> 7f9184000000-7f918409e000 rw-p 00000000 00:00 0
>> 7f918409e000-7f9188000000 ---p 00000000 00:00 0
>> 7f9188000000-7f9188047000 rw-p 00000000 00:00 0
>> 7f9188047Al000-7f918c000000 ---p 00000000 00:00 0
>> 7f918c000000-7f918c23d000 rw-p 00000000 00:00 0
>> 7f918c23d000-7f9190000000 ---p 00000000 00:00 0
>> 7f9190000000-7f9190106000 rw-p 00000000 00:00 0
>> 7f9190106000-7f9194000000 ---p 00000000 00:00 0
>> 7f9194000000-7f91940 c7000 rw-p 00000000 00:00 0
>> 7f91940c9000-7f9198000000 ---p 00000000 00:00 0
>> 7f9198000000-7f9198106000 rw-p 00000000 00:00 0
>> 7f9198106000-7f919c000000 ---p 00000000 00:00 0
>> 7f919c000000-7f919c165000 rw-p 00000000 00:00 0
>> 7f919c165000-7f91a0000000 ---p tbyn00000000 00:00 0
>> 7f919c167000-7f91a0000000 ---p 00000000 00:00 0
>> 7f91a0000000-7f91a009f000 rw-p 00000000 00:00 0
>> 7f91a009f000-7f91a4000000 ---p 00000000 00:00 0
>> 7f91a4000000-7f91a4047000 rw-p 00000000 00:00 0
>> 7f91a4047000-7f91a8000000 ---p 00000000 00 :00 0
>> 7f91a80f0000-7f91a80f1000 ---p 00000000 00:00 0
>> 7f91a80f1000-7f91a88f1000 rw-p 00000000 00:00 0
>> [stack:10106]
>> 7f91a88f1000-7f91a88f2000 ---p 00000000 00:00 0
>> 7f91a88f2000-7f91a90f2000 rw-p 00000000 00:00 0 AHdAv
>> [stack:10105]
>> 7f91a90f2000-7f91a90f3000 ---p 00000000 00:00 0
>> 7f91a90f3000-7f91a98f3000 rw-p 00000000 00:00 0
>> [stack:10104]
>> 7f91a98f3000-7f91a98f4000 ---p 00000000 00:00 0
>> 7f91a98f4000-7f91aa0f4000 rw-p 00000000 00:00 0
>> [stack:10103]
>> 7f91aa0f4000-7f91aa0f5000 ---p 00000000 00:00 0
>> 7f91aa0f5000-7f91aa8f5000 rw-p 00000000 00:00 0
>> [stack:10102]
>> 7f91aa8f5000-7f91aa8f6000 ---p 00000000 00:00 0
>> 7f91aa8f6000-7f91ab0f6000 rw-p 0lnA0000000 00:00 0
>> [stack:10101]
>> 7f91ab0f6000-7f91ab0f7000 ---p 00000000 00:00 0
>> 7f91ab0f7000-7f91ab8f7000 rw-p 00000000 00:00 0
>> [stack:10100]
>> 7f91ab8f7000-7f91ab8f8000 ---p 00000000 00:00 0
>> 7f91ab8f8000-7f9 1ac0f8000 rw-p 00000000 00:00 0
>> [stack:10099]
>> 7f91ac0f8000-7f91ac0f9000 ---p 00000000 00:00 0
>> 7f91ac0f9000-7f91ae8fa000 rw-p 00000000 00:00 0
>> [stack:10098]
>> 7f91aef80000-7f91aef81000 ---p 00000000 00:00 0
>> w7f91aef81000-7f91af781000 rw-p 00000000 00:00 0
>> [stack:10097]
>> 7f91af781000-7f91af782000 ---p 00000000 00:00 0
>> 7f91af782000-7f91aff82000 rw-p 00000000 00:00 0
>> [stack:10096]
>> 7f91aff82000-7f91aff83000 ---p 00 000000 00:00 0
>> 7f91aff83000-7f91b1784000 rw-p 00000000 00:00 0
>> [stack:10095]
>> 7f91b1886000-7f91b18c7000 rw-p 00000000 00:00 0
>> 7f91b18c7000-7f91b409f000 rw-p 00000000 00:00 0
>> 7f91b409f000-7f91b40ae000 r-xp 00000000 00:12 54468 An
>> /usr/lib64/libbz2.so.1.0.6
>> 7f91b40ae000-7f91b42ad000 ---p 0000f000 00:12 54468
>> /usr/lib64/libbz2.so.1.0.6
>> 7f91b42ad000-7f91b42ae000 r--p 0000e000 00:12 54468
>> /usr/lib64/libbz2.so.1.0.6
>> 7f91b42ae00 0-7f91b42af000 rw-p 0000f000 00:12 54468
>> /usr/lib64/libbz2.so.1.0.6
>> 7f91b42af000-7f91b42d3000 r-xp 00000000 00:12 54446
>> /usr/lib64/liblzma.so.5.0.99
>> 7f91b42d3000-7f91b44d2000 ---p 00024000 00:12 54446 AlVy
>> /usr/lib64/liblzma.so.5.0.99
>> 7f91b44d2000-7f91b44d3000 r--p 00023000 00:12 54446
>> /usr/lib64/liblzma.so.5.0.99
>> 7f91b44d3000-7f91b44d4000 rw-p 00024000 00:12 54446
>> /usr/lib64/liblzma.so.5.0.99
>> 7f91b44d4000-7f 91b468a000 r-xp 00000000 00:12 54398
>> /usr/lib64/libc-2.17.so
>> 7f91b468a000-7f91b488a000 ---p 001b6000 00:12 54398
>> /usr/lib64/libc-2.17.so
>> 7f91b488a000-7f91b488e000 r--p 001b6000 00:12 54398
>> /usrAl/lib64/libc-2.17.so
>> 7f91b488e000-7f91b4890000 rw-p 001ba000 00:12 54398
>> /usr/lib64/libc-2.17.so
>> 7f91b4890000-7f91b4895000 rw-p 00000000 00:00 0
>> 7f91b4895000-7f91b48ab000 r-xp 00000000 00:12 54451
>> /usr/lib64/libpth read-2.17.so
>> 7f91b48ab000-7f91b4aab000 ---p 00016000 00:12 54451
>> /usr/lib64/libpthread-2.17.so
>> 7f91b4aab000-7f91b4aac000 r--p 00016000 00:12 54451
>> /usr/lib64/libpthread-2.17.so
>> 7f91b4aac000-7f91b4aad000 rw-p 000170t00 00:12 54451
>> /usr/lib64/libpthread-2.17.so
>> 7f91b4aad000-7f91b4ab1000 rw-p 00000000 00:00 0
>> 7f91b4ab1000-7f91b4ac7000 r-xp 00000000 00:2b 390181951
>> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown-linux-gn
>> u/lib64/libgcc_s.so.1
>> 7f91b4ac7000-7f91b4cc6000 ---p 00016000 00:2b 390181951
>> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown
>> -linux-gnu/lib64/libgcc_s.so.1
>> 7f91b4cc6000-7f91b4cc7000 r--p 00015000 00:2b 390181951
>> slfyyn /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown
>> -linux-gnu/lib64/libgcc_s.so.1
>> 7f91b4cc7000-7f91b4cc8000 rw-p 00016000 00:2b 390181951
>> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown
>> -linux-gnu/lib64/libgcc_s.so.1
>> 7 f91b4cc8000-7f91b4dc9000 r-xp 00000000 00:12 54414
>> /usr/lib64/libm-2.17.so
>> 7f91b4dc9000-7f91b4fc8000 ---p 00101000 00:12 54414
>> /usr/lib64/libm-2.17.so
>> 7f91b4fc8000-7f91b4fc9000 r--p 00100000 00:12 54414 bdw
>> /usr/lib64/libm-2.17.so
>> 7f91b4fc9000-7f91b4fca000 rw-p 00101000 00:12 54414
>> /usr/lib64/libm-2.17.so
>> 7f91b4fca000-7f91b50b5000 r-xp 00000000 00:2b 3536542694
>> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-
>> unknown-linux-gnu/4.9.3/libstdc++.so.6.0.20
>> 7f91b50b5000-7f91b52b5000 ---p 000eb000 00:2b 3536542694
>> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown-
>> linux-gnu/4.9.3/libstdc++.so.6.0.20
>> 7f91b52b5000-7f91b52bd000 r--p 000eb000 00:hm 2b 3536542694
>> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown-
>> linux-gnu/4.9.3/libstdc++.so.6.0.20
>> 7f91b52bd000-7f91b52bf000 rw-p 000f3000 00:2b 3536542694
>> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknomlA
>> mHwn-linux-gnu/4.9.3/libstdc++.so.6.0.20
>> 7f91b52bf000-7f91b52d4000 rw-p 00000000 00:00 0
>> 7f91b52d4000-7f91b52db000 r-xp 00000000 00:12 54459
>> /usr/lib64/librt-2.17.so
>> 7f91b52db000-7f91b54da000 ---p 00007000 00:12 54459
>> /usr/lib64/librt-2.17.so
>> 7f91b54da000-7f91b54db000 r--p 00006000 00:12 54459
>> /usr/lib64/librt-2.17.so
>> 7f91b54db000-7f91b54dc000 rw-p 00007000 00:12 54459
>> /usr/lib64/librt-2.17.so
>> 7f91b54dc000-7f91b54fd000 r-xp 0mn0000000 00:2b 189775209
>> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57
>> .0/lib/libboost_thread.so.1.57.0
>> 7f91b54fd000-7f91b56fc000 ---p 00021000 00:2b 189775209
>> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/
>> avx2/1.57.0/lib/libboost_thread.so.1.57.0
>> 7f91b56fc000-7f91b56ff000 rw-p 00020000 00:2b 189775209
>> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57
>> .0/lib/libboost_thread.so.1.57.0
>> 7f91b56ff000-7f91b5714000 r-xp 00000000 00:12 544h64
>> /usr/lib64/libz.so.1.2.7
>> 7f91b5714000-7f91b5913000 ---p 00015000 00:12 54464
>> /usr/lib64/libz.so.1.2.7
>> 7f91b5913000-7f91b5914000 r--p 00014000 00:12 54464
>> /usr/lib64/libz.so.1.2.7
>> 7f91b591400 0-7f91b5915000 rw-p 00015000 00:12 54464
>> /usr/lib64/libz.so.1.2.7
>> 7f91b5915000-7f91b5918000 r-xp 00000000 00:2b 189775207
>> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57
>> .0/lib/libboost_system.so.1.57.0
>> 7f91AcElb5918000-7f91b5b17000 ---p 00003000 00:2b 189775207
>> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57
>> .0/lib/libboost_system.so.1.57.0
>> 7f91b5b17000-7f91b5b18000 rw-p 00002000 00:2b 189775207
>> /share/apps/NYUAD3/boo st/gcc_4.9.3/openmpi_1.10.2/av
>> x2/1.57.0/lib/libboost_system.so.1.57.0
>> 7f91b5b18000-7f91b5b2e000 r-xp 00000000 00:2b 189756212
>> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57
>> .0/lib/libboost_iostreams.so.1.57.0
>> 7f91b5b2e000-7f91bwADH5d2e000 ---p 00016000 00:2b 189756212
>> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57
>> .0/lib/libboost_iostreams.so.1.57.0
>> 7f91b5d2e000-7f91b5d30000 rw-p 00016000 00:2b 189756212
>> /share/apps/NYUAD3/boost/gcc_4.9. 3/openmpi_1.10.2/avx2/1.57.0/l
>> ib/libboost_iostreams.so.1.57.0
>> 7f91b5d30000-7f91b5d34000 r-xp 00000000 00:12 54392
>> /usr/lib64/libSegFault.so
>> 7f91b5d34000-7f91b5f33000 ---p 00004000 00:12 54392
>> /usr/lib64/libSegFaultwA.so
>> 7f91b5f33000-7f91b5f34000 r--p 00003000 00:12 54392
>> /usr/lib64/libSegFault.so
>> 7f91b5f34000-7f91b5f35000 rw-p 00004000 00:12 54392
>> /usr/lib64/libSegFault.so
>> 7f91b5f35000-7f91b5f38000 r-xp 00000000 00:12 54410
>> /usr/lib64/libdl-2.17.so
>> 7f91b5f38000-7f91b6137000 ---p 00003000 00:12 54410
>> /usr/lib64/libdl-2.17.so
>> 7f91b6137000-7f91b6138000 r--p 00002000 00:12 54410
>> /usr/lib64/libdl-2.17.so
>> 7f91b6138000-7f90.0221b6139000 rw-p 00003000 00:12 54410
>> /usr/lib64/libdl-2.17.so
>> 7f91b6139000-7f91b6150000 r-xp 00000000 00:2b 189756209
>> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57
>> .0/lib/libboost_filesystem.so.1.57.0
>> 7f91b seconds total6150000-7f91b634f000 ---p 00017000 00:2b 189756209
>> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57
>> .0/lib/libboost_filesystem.so.1.57.0
>> 7f91b634f000-7f91b6350000 rw-p 00016000 00:2b 189756209
>> /share/apps/NYUAD3/
>> boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_file
>> system.so.1.57.0
>> 7f91b6350000-7f91b6868000 r-xp 00000000 00:2b 2421011537
>> /share/apps/NYUAD3/mosesdecoder/3.0/lib/libmoses.so
>> 7f91b6868000-7f91b6a67000 ---p 00518000 00:2b 24210115ftyl37
>> /share/apps/NYUAD3/mosesdecoder/3.0/lib/libmoses.so
>> 7f91b6a67000-7f91b6a82000 rw-p 00517000 00:2b 2421011537
>> /share/apps/NYUAD3/mosesdecoder/3.0/lib/libmoses.so
>> 7f91b6a82000-7f91b73b2000 rw-p 00000000 00:00 0
>> 7f91b73b200 0-7f91b73d3000 r-xp 00000000 00:12 54384
>> /usr/lib64/ld-2.17.so
>> 7f91b7413000-7f91b743b000 rw-p 00000000 00:00 0
>> 7f91b745c000-7f91b74c6000 rw-p 00000000 00:00 0
>> 7f91b75c7000-7f91b75d3000 rw-p 00000000 00:00 0
>> 7f91b75d3000-7f91b75d4000Line r--p 00021000 00:12 54384
>> /usr/lib64/ld-2.17.so
>> 7f91b75d4000-7f91b75d5000 rw-p 00022000 00:12 54384
>> /usr/lib64/ld-2.17.so
>> 7f91b75d5000-7f91b75d6000 rw-p 00000000 00:00 0
>> 7ffcb1849000-7ffcb186c000 rw-p 00000000 00p:00 0
>> [stack]
>> 7ffcb191f000-7ffcb1921000 r-xp 00000000 00:00 0
>> [vdso]
>> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
>> [vsyscall]
>> 9: Collecting options took
>>
>> On Thu, Apr 6, 2017 at 1:58 AM, Alexander Erdmann <ae1541@nyu.edu> wrote:
>>
>>> Hi Moses Team,
>>>
>>> I'm running an MT system with 2 phrase tables and 2 reordering tables in
>>> conjunction using the default 4 probabilities in the former and 6
>>> probabilities in the latter.
>>>
>>> When I call it with this (and I'm running lots of jobs in parallel so I
>>> may run this command 20 times at once just varying the suffix on the input
>>> file 'del')
>>>
>>> $MOSESDECODER_ROOT/bin/moses -threads 4 -config
>>> $PWD/seed/$seedSize/iteration$iterations/moses.ini -input-file
>>> data/del$var 1> seed/$seedSize/iteration$iterations/del$var.lev 2>
>>> seed/$seedSize/iteration$iterations/del$var.log
>>>
>>> it never translates a single line, but it also doesn't die in the same
>>> place every time. Sometimes the error message is:
>>>
>>> Loading UnknownWordPenalty0
>>>
>>> Loading WordPenalty0
>>>
>>> Loading PhrasePenalty0
>>>
>>> Loading LexicalReordering0
>>>
>>> Loading table into memory...found inconsistent number of
>>> probabilities... found 4 expected 6
>>>
>>> and sometimes it loads the lexical reordering table just fine but
>>> segFaults while trying to translate the first line.
>>>
>>> I am positive that there are in fact 6 probabilities in every line of
>>> the reordering table so I'm especially confused about that error... any
>>> ideas?
>>>
>>> Here is my config file:
>>>
>>> # input factors
>>> [input-factors]
>>> 0
>>>
>>> # mapping steps
>>> [mapping]
>>> 0 T 0
>>> 1 T 1
>>>
>>> [distortion-limit]
>>> 6
>>>
>>> # feature functions
>>> [feature]
>>> UnknownWordPenalty
>>> WordPenalty
>>> PhrasePenalty
>>> PhraseDictionaryMemory name=TranslationModel0 num-features=4
>>> path=/scratch/ae1541/EMNLP/seed/$$$/iteration0/phrase-table
>>> PhraseDictionaryMemory name=TranslationModel1 num-features=4
>>> path=/scratch/ae1541/EMNLP/seed/$$$/iteration***/phrase-table
>>> input-factor=0 output-factor=0
>>> LexicalReordering name=LexicalReordering0 num-features=6
>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>>> path=/scratch/ae1541/EMNLP/seed/$$$/iteration0/reordering-table
>>> LexicalReordering name=LexicalReordering1 num-features=6
>>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>>> path=/scratch/ae1541/EMNLP/seed/$$$/iteration***/reordering-table
>>> Distortion
>>> KENLM name=LM0 factor=0 path=/scratch/ae1541/EMNLP/seed/LM/LM.lev.lm
>>> order=5
>>>
>>> # dense weights for feature functions
>>> [weight]
>>> UnknownWordPenalty0= 1
>>> WordPenalty0= -1
>>> PhrasePenalty0= 0.2
>>> TranslationModel0= 0.2 0.2 0.2 0.2
>>> TranslationModel1= 0.2 0.2 0.2 0.2
>>> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
>>> LexicalReordering1= 0.3 0.3 0.3 0.3 0.3 0.3
>>> Distortion0= 0.3
>>> LM0= 0.5
>>>
>>> Thanks!
>>>
>>> --
>>> Alex Erdmann
>>> PhD Student in Linguistics at The Ohio State University
>>> Visiting Scholar at NYU Abu Dhabi
>>>
>>
>>
>>
>> --
>> Alex Erdmann
>> PhD Student in Linguistics at The Ohio State University
>> Visiting Scholar at NYU Abu Dhabi
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>


--
Alex Erdmann
PhD Student in Linguistics at The Ohio State University
Visiting Scholar at NYU Abu Dhabi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170407/baae70ae/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 126, Issue 10
**********************************************

Tweet