Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Adding new aligned phrases to the existing phrase table (sriram)
2. Re: inconsistent number of probabilities (Hieu Hoang)
----------------------------------------------------------------------
Message: 1
Date: Thu, 6 Apr 2017 23:44:07 +0530
From: sriram <sriramchaudhury@gmail.com>
Subject: [Moses-support] Adding new aligned phrases to the existing
phrase table
To: moses-support@mit.edu
Message-ID:
<CAA-zSKbqXQzUFtPQmU-QiTGF-Mg599bsoH3K=8igcEXDtR+FXg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
I have some good aligned phrases collection and I want to add to the
existing phrase table. Is there any existing tool to add the same in Moses.
Thanks,
Sriram
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170406/14b7bb8b/attachment-0001.html
------------------------------
Message: 2
Date: Fri, 7 Apr 2017 13:11:13 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] inconsistent number of probabilities
To: Alexander Erdmann <ae1541@nyu.edu>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbjD9p0+ejEgvb2Z5P7OZ+EZ+ncNdLUdc2L97e2cPQ1JBA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
The moses.ini file looks ok.
The segfault is difficult to debug by inspection. It's best sprinkle the
decoder code with debugging message and run 1 instance of the decoder to
see exactly where it goes wrong. And you should binarise the pt and
reordering models to save yourself time
* Looking for MT/NLP opportunities *
Hieu Hoang
http://moses-smt.org/
On 6 April 2017 at 11:30, Alexander Erdmann <ae1541@nyu.edu> wrote:
> For reference, here is the output from the segFault:
>
> Defined parameters (per moses.ini or switch):
> config: /scratch/ae1541/EMNLP/seed/150000/iteration1/mosesTryingGZ.ini
> distortion-limit: 6
> feature: UnknownWordPenalty WordPenalty PhrasePenalty
> PhraseDictionaryMemory name=TranslationModel0 num-features=4
> path=/scratch/ae1541/EMNLP/seed/150000/iteration0/phrase-table.gz
> LexicalReordering name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/scratch/ae1541/EMNLP/seed/150000/iteration0/reordering-table.gz
> Distortion KENLM name=LM0 factor=0 path=/scratch/ae1541/EMNLP/seed/LM/LM.lev.lm
> order=5
> input-factors: 0
> input-file: seed/150000/iteration1/del00.egy
> mapping: 0 T 0
> threads: 12
> weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
> TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3
> 0.3 Distortion0= 0.3 LM0= 0.5
> line=UnknownWordPenalty
> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
> line=WordPenalty
> FeatureFunction: WordPenalty0 start: 1 end: 1
> line=PhrasePenalty
> FeatureFunction: PhrasePenalty0 start: 2 end: 2
> line=PhraseDictionaryMemory name=TranslationModel0 num-features=4
> path=/scratch/ae1541/EMNLP/seed/150000/iteration0/phrase-table.gz
> FeatureFunction: TranslationModel0 start: 3 end: 6
> line=LexicalReordering name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/scratch/ae1541/EMNLP/seed/150000/iteration0/reordering-table.gz
> FeatureFunction: LexicalReordering0 start: 7 end: 12
> Initializing LexicalReordering..
> line=Distortion
> FeatureFunction: Distortion0 start: 13 end: 13
> line=KENLM name=LM0 factor=0 path=/scratch/ae1541/EMNLP/seed/LM/LM.lev.lm
> order=5
> FeatureFunction: LM0 start: 14 end: 14
> Loading the LM will be faster if you build a binary file.
> Reading /scratch/ae1541/EMNLP/seed/LM/LM.lev.lm
> ----5---10---15---20---25---30---35---40---45---50---55---
> 60---65---70---75---80---85---90---95--100
> ************************************************************
> ****************************************
> Loading UnknownWordPenalty0
> Loading WordPenalty0
> Loading PhrasePenalty0
> Loading LexicalReordering0
> Loading table into memory...done.
> Loading Distortion0
> Loading LM0
> Loading TranslationModel0
> Start loading text phrase table. Moses format : [1574.737] seconds
> Reading /scratch/ae1541/EMNLP/seed/150000/iteration0/phrase-table.gz
> ----5---10---15---20---25---30---35---40---45---50---55---
> 60---65---70---75---80---85---90---95--100
> ************************************************************
> ****************************************
> Created input-output object : [1608.439] seconds
> Translating: Translating: A y Translating: jAC Translating: Translating:
> AbwTranslating: RRR w hA Al mAd mrwAn
> sAdRRR jrydp A m trtr . . jrydp ywmy p . . bTyx yp . . mhlb yp bAl
> tEycTranslating: w nA Al byAn Al tAly p
> Line 1: Initialize search took Line 8: Initialize search took tqly p 0.006
> seconds total
> 0.005 seconds total
> Translating: t h y Translating: qwqAl A b mSr RRR rytc bAl Al dAxly p y
> AxbAr ?
> Line 9: Initialize search took 0.007 seconds total
> w b h mfyc RRR
> AxwAn hm Hd m Al mslmyn AnDrb
> Line 11: Initialize search took Line 0: Initialize search took Al0.010
> seconds total
> Hrp 0.009 seconds total
> Al Translating: AfAd Al mhlb jrydpmtHdv Al rsmy l wzArp Al nSAry bEd
> Translating: yp Al m m A y h Avl Al dAxly w
> An p AfAdt Line 7: Initialize search took qyAd ( Line xyr p ll TbE
> Al mSlHp 8: Collecting options took Line 2: Initialize search took lhm Al
> 0.005 seconds at moses/Manager.cpp:AjElh bHv 0.015 seconds total
> 0.013 seconds total
> wrAC Amn xyr mvyr Al Al y dwlp tlyfzywn Al sAbq Al mSry A ) Almbjl ytHdv b
> Al An lsAn jnAQy h jrydp An bEd Al ftn A tHry w m w Al trtr Al bHv w mtTrf
> f yTlb yn y mn Al AHdAv Al mslmyn
> hm Line 6: Initialize search took 117
> nA 0.020 seconds total
> s *** bSegmentation fault
> Register dump:
>
> RAX: 0000000000000000 RBX: 00000000012e28e0 RCX: 0000000000000000
> RDX: 00007f91b7413010 RSI: 0000000000000000 RDI: 00007f917400b9c0
> RBP: 0000000000000000 R8 : 0000000000000002 R9 : 0000000000000038
> R10: 00007f91aa8f3310 R11: 00007f91b64cfbf0 R12: 00007f917400b028
> R13: 00007f9174004f60 R14: 00007f917400b9c0 R15: 0000000000000002
> RSP: 00007f91aa8f3730
>
> RIP: 00007f91b66e8833 EFLAGS: 00010246
>
> CS: 0033 FS: 0000 GS: 0000
>
> Trap: 0000000e Error: 00000004 OldMask: 00000000 CR2: 00000010
>
> FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000
> RIP: 00000000 RDP: 00000000
>
> ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000
> ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000
> ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000
> ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000
> mxcsr: 1fa4
> XMM0: 00000000000000000000000000000000 XMM1:
> 00000000000000000000000000000000
> XMM2: 00000000000000000000000000000000 XMM3:
> 00000000000000000000000000000000
> XMM4: 00000000000000000000000000000000 XMM5:
> 00000000000000000000000000000000
> XMM6: 00000000000000000000000000000000 XMM7:
> 00000000000000000000000000000000
> XMM8: 00000000000000000000000000000000 XMM9:
> 00000000000000000000000000000000
> XMM10: 00000000000000000000000000000000 XMM11:
> 00000000000000000000000000000000
> XMM12: 00000000000000000000000000000000 XMM13:
> 00000000000000000000000000000000
> XMM14: 00000000000000000000000000000000 XMM15:
> 00000000000000000000000000000000
>
> Backtrace:
> y Sdd lA tjmyE hyj rAs wA AHd/share/apps/NYUAD/
> mosesdecoder/3.0/lib/libmoses.so(_ZNK5Moses16LanguageModelKenIN2
> lm5ngram12ProbingModelEE19EvaluateWhenAppliedERKNS_10HypothesisEPKNS_
> 7FFStateEPNS_24ScoreComponentCollectionE+0xa3)[0x7f91b66e8833]
> Ely/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_
> ZN5Moses10Hypothesis19EvaluateWhenAppliedERKNS_12SquareMatrixE+0xf0)[
> 0x7f91b64cfab0]
> Al /share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_
> ZN5Moses12SearchNormal16ExpandHypothesisERKNS_10HypothesisERKNS_
> 17TranslationOptionEf+0x165)[0x7f91b65484a5]
> bED /share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_
> ZN5Moses12SearchNormal19ExpandAllHypothesesERKNS_10HypothesisEmm+0x79)[
> 0x7f91b6548829]
>
> mALine /share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_
> ZN5Moses12SearchNormal20ProcessOneHypothesisERKNS_10HypothesisE+0x34b)[
> 0x7f91b6548beb]
> 10: Initialize search took /share/apps/NYUAD/
> mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses12SearchNormal6DecodeE
> v+0x11c)[0x7f91b6548ffc]
> DHAyA/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.
> so(_ZN5Moses7Manager6DecodeEv+0xe3)[0x7f91b64f9173]
> sbyrw /share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_
> ZN5Moses15TranslationTask3RunEv+0x1a5)[0x7f91b656f505]
> f /share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_
> ZN5Moses10ThreadPool7ExecuteEv+0x89)[0x7f91b6724b99]
> ./share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.
> 57.0/lib/libboost_thread.so.1.57.0(+0x1211a)[0x7f91b54ee11a]
> /usr/lib64/libpthread.so.0(+0x7dc5)[0x7f91b489cdc5]
> y /usr/lib64/libc.so.6(clone+0x6d)[0x7f91b45ca28d]
> .
> Memory map:
>
> tlk 00400000-0040b000 r-xp 00000000 00:2b 1254532671
> /share/apps/NYUAD3/mosesdecoder/3.0/bin/moses
> 0060b000-0060c000 rw-p 0000b000 00:2b 1254532671
> /share/apps/NYUAD3/mosesdecoder/3.0/bin/moses
> 012ce000-52974a000qd rw-p 00000000 00:00 0
> [heap]
> 7f9174000000-7f9174021000 rw-p 00000000 00:00 0
> 7f9174021000-7f9178000000 ---p 00000000 00:00 0
> 7f9178000000-7f917807b000 rw-p 00000000 00:00 0
> 7f917807b000-7f917c000000 ---p 00000000 00:00 0
> 7f917c000000-7f917c034000 rw-p 00000000 00:00 0
> 7f917c034000-7f9180000000 ---p 00000000 00:00 0
> 7f9184000000-7f918409e000 rw-p 00000000 00:00 0
> 7f918409e000-7f9188000000 ---p 00000000 00:00 0
> 7f9188000000-7f9188047000 rw-p 00000000 00:00 0
> 7f9188047Al000-7f918c000000 ---p 00000000 00:00 0
> 7f918c000000-7f918c23d000 rw-p 00000000 00:00 0
> 7f918c23d000-7f9190000000 ---p 00000000 00:00 0
> 7f9190000000-7f9190106000 rw-p 00000000 00:00 0
> 7f9190106000-7f9194000000 ---p 00000000 00:00 0
> 7f9194000000-7f91940 c7000 rw-p 00000000 00:00 0
> 7f91940c9000-7f9198000000 ---p 00000000 00:00 0
> 7f9198000000-7f9198106000 rw-p 00000000 00:00 0
> 7f9198106000-7f919c000000 ---p 00000000 00:00 0
> 7f919c000000-7f919c165000 rw-p 00000000 00:00 0
> 7f919c165000-7f91a0000000 ---p tbyn00000000 00:00 0
> 7f919c167000-7f91a0000000 ---p 00000000 00:00 0
> 7f91a0000000-7f91a009f000 rw-p 00000000 00:00 0
> 7f91a009f000-7f91a4000000 ---p 00000000 00:00 0
> 7f91a4000000-7f91a4047000 rw-p 00000000 00:00 0
> 7f91a4047000-7f91a8000000 ---p 00000000 00 :00 0
> 7f91a80f0000-7f91a80f1000 ---p 00000000 00:00 0
> 7f91a80f1000-7f91a88f1000 rw-p 00000000 00:00 0
> [stack:10106]
> 7f91a88f1000-7f91a88f2000 ---p 00000000 00:00 0
> 7f91a88f2000-7f91a90f2000 rw-p 00000000 00:00 0 AHdAv
> [stack:10105]
> 7f91a90f2000-7f91a90f3000 ---p 00000000 00:00 0
> 7f91a90f3000-7f91a98f3000 rw-p 00000000 00:00 0
> [stack:10104]
> 7f91a98f3000-7f91a98f4000 ---p 00000000 00:00 0
> 7f91a98f4000-7f91aa0f4000 rw-p 00000000 00:00 0
> [stack:10103]
> 7f91aa0f4000-7f91aa0f5000 ---p 00000000 00:00 0
> 7f91aa0f5000-7f91aa8f5000 rw-p 00000000 00:00 0
> [stack:10102]
> 7f91aa8f5000-7f91aa8f6000 ---p 00000000 00:00 0
> 7f91aa8f6000-7f91ab0f6000 rw-p 0lnA0000000 00:00 0
> [stack:10101]
> 7f91ab0f6000-7f91ab0f7000 ---p 00000000 00:00 0
> 7f91ab0f7000-7f91ab8f7000 rw-p 00000000 00:00 0
> [stack:10100]
> 7f91ab8f7000-7f91ab8f8000 ---p 00000000 00:00 0
> 7f91ab8f8000-7f9 1ac0f8000 rw-p 00000000 00:00 0
> [stack:10099]
> 7f91ac0f8000-7f91ac0f9000 ---p 00000000 00:00 0
> 7f91ac0f9000-7f91ae8fa000 rw-p 00000000 00:00 0
> [stack:10098]
> 7f91aef80000-7f91aef81000 ---p 00000000 00:00 0
> w7f91aef81000-7f91af781000 rw-p 00000000 00:00 0
> [stack:10097]
> 7f91af781000-7f91af782000 ---p 00000000 00:00 0
> 7f91af782000-7f91aff82000 rw-p 00000000 00:00 0
> [stack:10096]
> 7f91aff82000-7f91aff83000 ---p 00 000000 00:00 0
> 7f91aff83000-7f91b1784000 rw-p 00000000 00:00 0
> [stack:10095]
> 7f91b1886000-7f91b18c7000 rw-p 00000000 00:00 0
> 7f91b18c7000-7f91b409f000 rw-p 00000000 00:00 0
> 7f91b409f000-7f91b40ae000 r-xp 00000000 00:12 54468 An
> /usr/lib64/libbz2.so.1.0.6
> 7f91b40ae000-7f91b42ad000 ---p 0000f000 00:12 54468
> /usr/lib64/libbz2.so.1.0.6
> 7f91b42ad000-7f91b42ae000 r--p 0000e000 00:12 54468
> /usr/lib64/libbz2.so.1.0.6
> 7f91b42ae00 0-7f91b42af000 rw-p 0000f000 00:12 54468
> /usr/lib64/libbz2.so.1.0.6
> 7f91b42af000-7f91b42d3000 r-xp 00000000 00:12 54446
> /usr/lib64/liblzma.so.5.0.99
> 7f91b42d3000-7f91b44d2000 ---p 00024000 00:12 54446 AlVy
> /usr/lib64/liblzma.so.5.0.99
> 7f91b44d2000-7f91b44d3000 r--p 00023000 00:12 54446
> /usr/lib64/liblzma.so.5.0.99
> 7f91b44d3000-7f91b44d4000 rw-p 00024000 00:12 54446
> /usr/lib64/liblzma.so.5.0.99
> 7f91b44d4000-7f 91b468a000 r-xp 00000000 00:12 54398
> /usr/lib64/libc-2.17.so
> 7f91b468a000-7f91b488a000 ---p 001b6000 00:12 54398
> /usr/lib64/libc-2.17.so
> 7f91b488a000-7f91b488e000 r--p 001b6000 00:12 54398
> /usrAl/lib64/libc-2.17.so
> 7f91b488e000-7f91b4890000 rw-p 001ba000 00:12 54398
> /usr/lib64/libc-2.17.so
> 7f91b4890000-7f91b4895000 rw-p 00000000 00:00 0
> 7f91b4895000-7f91b48ab000 r-xp 00000000 00:12 54451
> /usr/lib64/libpth read-2.17.so
> 7f91b48ab000-7f91b4aab000 ---p 00016000 00:12 54451
> /usr/lib64/libpthread-2.17.so
> 7f91b4aab000-7f91b4aac000 r--p 00016000 00:12 54451
> /usr/lib64/libpthread-2.17.so
> 7f91b4aac000-7f91b4aad000 rw-p 000170t00 00:12 54451
> /usr/lib64/libpthread-2.17.so
> 7f91b4aad000-7f91b4ab1000 rw-p 00000000 00:00 0
> 7f91b4ab1000-7f91b4ac7000 r-xp 00000000 00:2b 390181951
> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown-linux-gn
> u/lib64/libgcc_s.so.1
> 7f91b4ac7000-7f91b4cc6000 ---p 00016000 00:2b 390181951
> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-
> unknown-linux-gnu/lib64/libgcc_s.so.1
> 7f91b4cc6000-7f91b4cc7000 r--p 00015000 00:2b 390181951
> slfyyn /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-
> unknown-linux-gnu/lib64/libgcc_s.so.1
> 7f91b4cc7000-7f91b4cc8000 rw-p 00016000 00:2b 390181951
> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-
> unknown-linux-gnu/lib64/libgcc_s.so.1
> 7 f91b4cc8000-7f91b4dc9000 r-xp 00000000 00:12 54414
> /usr/lib64/libm-2.17.so
> 7f91b4dc9000-7f91b4fc8000 ---p 00101000 00:12 54414
> /usr/lib64/libm-2.17.so
> 7f91b4fc8000-7f91b4fc9000 r--p 00100000 00:12 54414 bdw
> /usr/lib64/libm-2.17.so
> 7f91b4fc9000-7f91b4fca000 rw-p 00101000 00:12 54414
> /usr/lib64/libm-2.17.so
> 7f91b4fca000-7f91b50b5000 r-xp 00000000 00:2b 3536542694
> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-
> unknown-linux-gnu/4.9.3/libstdc++.so.6.0.20
> 7f91b50b5000-7f91b52b5000 ---p 000eb000 00:2b 3536542694
> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-
> unknown-linux-gnu/4.9.3/libstdc++.so.6.0.20
> 7f91b52b5000-7f91b52bd000 r--p 000eb000 00:hm 2b 3536542694
> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-
> unknown-linux-gnu/4.9.3/libstdc++.so.6.0.20
> 7f91b52bd000-7f91b52bf000 rw-p 000f3000 00:2b 3536542694
> /share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-
> unknomlAmHwn-linux-gnu/4.9.3/libstdc++.so.6.0.20
> 7f91b52bf000-7f91b52d4000 rw-p 00000000 00:00 0
> 7f91b52d4000-7f91b52db000 r-xp 00000000 00:12 54459
> /usr/lib64/librt-2.17.so
> 7f91b52db000-7f91b54da000 ---p 00007000 00:12 54459
> /usr/lib64/librt-2.17.so
> 7f91b54da000-7f91b54db000 r--p 00006000 00:12 54459
> /usr/lib64/librt-2.17.so
> 7f91b54db000-7f91b54dc000 rw-p 00007000 00:12 54459
> /usr/lib64/librt-2.17.so
> 7f91b54dc000-7f91b54fd000 r-xp 0mn0000000 00:2b 189775209
> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.
> 57.0/lib/libboost_thread.so.1.57.0
> 7f91b54fd000-7f91b56fc000 ---p 00021000 00:2b 189775209
> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/
> avx2/1.57.0/lib/libboost_thread.so.1.57.0
> 7f91b56fc000-7f91b56ff000 rw-p 00020000 00:2b 189775209
> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.
> 57.0/lib/libboost_thread.so.1.57.0
> 7f91b56ff000-7f91b5714000 r-xp 00000000 00:12 544h64
> /usr/lib64/libz.so.1.2.7
> 7f91b5714000-7f91b5913000 ---p 00015000 00:12 54464
> /usr/lib64/libz.so.1.2.7
> 7f91b5913000-7f91b5914000 r--p 00014000 00:12 54464
> /usr/lib64/libz.so.1.2.7
> 7f91b591400 0-7f91b5915000 rw-p 00015000 00:12 54464
> /usr/lib64/libz.so.1.2.7
> 7f91b5915000-7f91b5918000 r-xp 00000000 00:2b 189775207
> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.
> 57.0/lib/libboost_system.so.1.57.0
> 7f91AcElb5918000-7f91b5b17000 ---p 00003000 00:2b 189775207
> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.
> 57.0/lib/libboost_system.so.1.57.0
> 7f91b5b17000-7f91b5b18000 rw-p 00002000 00:2b 189775207
> /share/apps/NYUAD3/boo st/gcc_4.9.3/openmpi_1.10.2/
> avx2/1.57.0/lib/libboost_system.so.1.57.0
> 7f91b5b18000-7f91b5b2e000 r-xp 00000000 00:2b 189756212
> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.
> 57.0/lib/libboost_iostreams.so.1.57.0
> 7f91b5b2e000-7f91bwADH5d2e000 ---p 00016000 00:2b 189756212
> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.
> 57.0/lib/libboost_iostreams.so.1.57.0
> 7f91b5d2e000-7f91b5d30000 rw-p 00016000 00:2b 189756212
> /share/apps/NYUAD3/boost/gcc_4.9. 3/openmpi_1.10.2/avx2/1.57.0/
> lib/libboost_iostreams.so.1.57.0
> 7f91b5d30000-7f91b5d34000 r-xp 00000000 00:12 54392
> /usr/lib64/libSegFault.so
> 7f91b5d34000-7f91b5f33000 ---p 00004000 00:12 54392
> /usr/lib64/libSegFaultwA.so
> 7f91b5f33000-7f91b5f34000 r--p 00003000 00:12 54392
> /usr/lib64/libSegFault.so
> 7f91b5f34000-7f91b5f35000 rw-p 00004000 00:12 54392
> /usr/lib64/libSegFault.so
> 7f91b5f35000-7f91b5f38000 r-xp 00000000 00:12 54410
> /usr/lib64/libdl-2.17.so
> 7f91b5f38000-7f91b6137000 ---p 00003000 00:12 54410
> /usr/lib64/libdl-2.17.so
> 7f91b6137000-7f91b6138000 r--p 00002000 00:12 54410
> /usr/lib64/libdl-2.17.so
> 7f91b6138000-7f90.0221b6139000 rw-p 00003000 00:12 54410
> /usr/lib64/libdl-2.17.so
> 7f91b6139000-7f91b6150000 r-xp 00000000 00:2b 189756209
> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.
> 57.0/lib/libboost_filesystem.so.1.57.0
> 7f91b seconds total6150000-7f91b634f000 ---p 00017000 00:2b 189756209
> /share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.
> 57.0/lib/libboost_filesystem.so.1.57.0
> 7f91b634f000-7f91b6350000 rw-p 00016000 00:2b 189756209
> /share/apps/NYUAD3/
> boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_
> filesystem.so.1.57.0
> 7f91b6350000-7f91b6868000 r-xp 00000000 00:2b 2421011537
> /share/apps/NYUAD3/mosesdecoder/3.0/lib/libmoses.so
> 7f91b6868000-7f91b6a67000 ---p 00518000 00:2b 24210115ftyl37
> /share/apps/NYUAD3/mosesdecoder/3.0/lib/libmoses.so
> 7f91b6a67000-7f91b6a82000 rw-p 00517000 00:2b 2421011537
> /share/apps/NYUAD3/mosesdecoder/3.0/lib/libmoses.so
> 7f91b6a82000-7f91b73b2000 rw-p 00000000 00:00 0
> 7f91b73b200 0-7f91b73d3000 r-xp 00000000 00:12 54384
> /usr/lib64/ld-2.17.so
> 7f91b7413000-7f91b743b000 rw-p 00000000 00:00 0
> 7f91b745c000-7f91b74c6000 rw-p 00000000 00:00 0
> 7f91b75c7000-7f91b75d3000 rw-p 00000000 00:00 0
> 7f91b75d3000-7f91b75d4000Line r--p 00021000 00:12 54384
> /usr/lib64/ld-2.17.so
> 7f91b75d4000-7f91b75d5000 rw-p 00022000 00:12 54384
> /usr/lib64/ld-2.17.so
> 7f91b75d5000-7f91b75d6000 rw-p 00000000 00:00 0
> 7ffcb1849000-7ffcb186c000 rw-p 00000000 00p:00 0
> [stack]
> 7ffcb191f000-7ffcb1921000 r-xp 00000000 00:00 0
> [vdso]
> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
> [vsyscall]
> 9: Collecting options took
>
> On Thu, Apr 6, 2017 at 1:58 AM, Alexander Erdmann <ae1541@nyu.edu> wrote:
>
>> Hi Moses Team,
>>
>> I'm running an MT system with 2 phrase tables and 2 reordering tables in
>> conjunction using the default 4 probabilities in the former and 6
>> probabilities in the latter.
>>
>> When I call it with this (and I'm running lots of jobs in parallel so I
>> may run this command 20 times at once just varying the suffix on the input
>> file 'del')
>>
>> $MOSESDECODER_ROOT/bin/moses -threads 4 -config
>> $PWD/seed/$seedSize/iteration$iterations/moses.ini -input-file
>> data/del$var 1> seed/$seedSize/iteration$iterations/del$var.lev 2>
>> seed/$seedSize/iteration$iterations/del$var.log
>>
>> it never translates a single line, but it also doesn't die in the same
>> place every time. Sometimes the error message is:
>>
>> Loading UnknownWordPenalty0
>>
>> Loading WordPenalty0
>>
>> Loading PhrasePenalty0
>>
>> Loading LexicalReordering0
>>
>> Loading table into memory...found inconsistent number of probabilities...
>> found 4 expected 6
>>
>> and sometimes it loads the lexical reordering table just fine but
>> segFaults while trying to translate the first line.
>>
>> I am positive that there are in fact 6 probabilities in every line of the
>> reordering table so I'm especially confused about that error... any ideas?
>>
>> Here is my config file:
>>
>> # input factors
>> [input-factors]
>> 0
>>
>> # mapping steps
>> [mapping]
>> 0 T 0
>> 1 T 1
>>
>> [distortion-limit]
>> 6
>>
>> # feature functions
>> [feature]
>> UnknownWordPenalty
>> WordPenalty
>> PhrasePenalty
>> PhraseDictionaryMemory name=TranslationModel0 num-features=4
>> path=/scratch/ae1541/EMNLP/seed/$$$/iteration0/phrase-table
>> PhraseDictionaryMemory name=TranslationModel1 num-features=4
>> path=/scratch/ae1541/EMNLP/seed/$$$/iteration***/phrase-table
>> input-factor=0 output-factor=0
>> LexicalReordering name=LexicalReordering0 num-features=6
>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>> path=/scratch/ae1541/EMNLP/seed/$$$/iteration0/reordering-table
>> LexicalReordering name=LexicalReordering1 num-features=6
>> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>> path=/scratch/ae1541/EMNLP/seed/$$$/iteration***/reordering-table
>> Distortion
>> KENLM name=LM0 factor=0 path=/scratch/ae1541/EMNLP/seed/LM/LM.lev.lm
>> order=5
>>
>> # dense weights for feature functions
>> [weight]
>> UnknownWordPenalty0= 1
>> WordPenalty0= -1
>> PhrasePenalty0= 0.2
>> TranslationModel0= 0.2 0.2 0.2 0.2
>> TranslationModel1= 0.2 0.2 0.2 0.2
>> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
>> LexicalReordering1= 0.3 0.3 0.3 0.3 0.3 0.3
>> Distortion0= 0.3
>> LM0= 0.5
>>
>> Thanks!
>>
>> --
>> Alex Erdmann
>> PhD Student in Linguistics at The Ohio State University
>> Visiting Scholar at NYU Abu Dhabi
>>
>
>
>
> --
> Alex Erdmann
> PhD Student in Linguistics at The Ohio State University
> Visiting Scholar at NYU Abu Dhabi
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170407/202765d8/attachment.html
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 126, Issue 8
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 126, Issue 8"
Post a Comment