Home » Uncategories » Moses-support Digest, Vol 126, Issue 6

Moses-support Digest, Vol 126, Issue 6

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. inconsistent number of probabilities (Alexander Erdmann)
2. mkcls - usable for bilingual class creation? (Allen Smith)
3. Re: Eliminating load times for MOSES phrase table, language
model and reordering model (Mathias M?ller)
4. Re: inconsistent number of probabilities (Alexander Erdmann)


----------------------------------------------------------------------

Message: 1
Date: Thu, 6 Apr 2017 01:58:33 +0400
From: Alexander Erdmann <ae1541@nyu.edu>
Subject: [Moses-support] inconsistent number of probabilities
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAGTOGBcA3y5qgSbAHRU8-gYrUMjdo-34XehosGO_Bt2YO6YmUg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Moses Team,

I'm running an MT system with 2 phrase tables and 2 reordering tables in
conjunction using the default 4 probabilities in the former and 6
probabilities in the latter.

When I call it with this (and I'm running lots of jobs in parallel so I may
run this command 20 times at once just varying the suffix on the input file
'del')

$MOSESDECODER_ROOT/bin/moses -threads 4 -config
$PWD/seed/$seedSize/iteration$iterations/moses.ini -input-file data/del$var
1> seed/$seedSize/iteration$iterations/del$var.lev 2>
seed/$seedSize/iteration$iterations/del$var.log

it never translates a single line, but it also doesn't die in the same
place every time. Sometimes the error message is:

Loading UnknownWordPenalty0

Loading WordPenalty0

Loading PhrasePenalty0

Loading LexicalReordering0

Loading table into memory...found inconsistent number of probabilities...
found 4 expected 6

and sometimes it loads the lexical reordering table just fine but segFaults
while trying to translate the first line.

I am positive that there are in fact 6 probabilities in every line of the
reordering table so I'm especially confused about that error... any ideas?

Here is my config file:

# input factors
[input-factors]
0

# mapping steps
[mapping]
0 T 0
1 T 1

[distortion-limit]
6

# feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryMemory name=TranslationModel0 num-features=4
path=/scratch/ae1541/EMNLP/seed/$$$/iteration0/phrase-table
PhraseDictionaryMemory name=TranslationModel1 num-features=4
path=/scratch/ae1541/EMNLP/seed/$$$/iteration***/phrase-table
input-factor=0 output-factor=0
LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=/scratch/ae1541/EMNLP/seed/$$$/iteration0/reordering-table
LexicalReordering name=LexicalReordering1 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=/scratch/ae1541/EMNLP/seed/$$$/iteration***/reordering-table
Distortion
KENLM name=LM0 factor=0 path=/scratch/ae1541/EMNLP/seed/LM/LM.lev.lm order=5

# dense weights for feature functions
[weight]
UnknownWordPenalty0= 1
WordPenalty0= -1
PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2
TranslationModel1= 0.2 0.2 0.2 0.2
LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
LexicalReordering1= 0.3 0.3 0.3 0.3 0.3 0.3
Distortion0= 0.3
LM0= 0.5

Thanks!

--
Alex Erdmann
PhD Student in Linguistics at The Ohio State University
Visiting Scholar at NYU Abu Dhabi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170405/4dabcf8e/attachment-0001.html

------------------------------

Message: 2
Date: Wed, 5 Apr 2017 17:49:15 -0500
From: Allen Smith <allen.w.smith1@gmail.com>
Subject: [Moses-support] mkcls - usable for bilingual class creation?
To: moses-support@mit.edu
Message-ID:
<CALiVeRHqek0LJKzwb9R9RBsgWqHstuJy_kQjYTvF85RfDnQckQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi. The original Och paper (at least the one not in, IIRC, German) that's
referenced for mkcls (at http://www.fjoch.com/mkcls.html) includes a
description of using a starting alignment to improve the word
classification. Is this indeed doable with mkcls, and if so, how?

Thanks,

-Allen (Allen W. Smith, Ph.D.)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170405/431b8fd6/attachment-0001.html

------------------------------

Message: 3
Date: Thu, 6 Apr 2017 10:53:39 +0200
From: Mathias M?ller <mmueller@cl.uzh.ch>
Subject: Re: [Moses-support] Eliminating load times for MOSES phrase
table, language model and reordering model
To: RR <rmijic@gmail.com>, moses-support@mit.edu
Message-ID: <9916dc1b-10cf-fb90-3ada-40b898a4c0cb@cl.uzh.ch>
Content-Type: text/plain; charset="windows-1252"

Hi Roko

Apart from compact phrase tables, which you should try:

You could start several Moses servers on different ports, with different
decoding parameters. Would that not be feasible for you?

Regards

Mathias


On 05.04.17 13:33, RR wrote:
> Hi,
>
> I am currently working on a system to explore MOSES decoding parameter
> space - running MOSES in decoding mode with one parameter set, getting
> a BLEU score and rerunning with different parameters.
>
> More than 50% of my time is being spent repeatedly "loading" the
> phrase table, language model and reordering model. For example,
>
> $ ./example.fr <http://example.fr> | .../moses -f ./run11.moses.ini
> -threads 31 > ./trans_baseline
>
> gives: "Start loading text phrase table. Moses format : [37.296]
> seconds."
>
> 37 seconds just to load a 590MB phrase table, even if it is on
> RAMDISK. htop indicates that for those 37 seconds, a single core is
> 100% utilized.
>
> My understanding is that some kind of optimized data structure (a hash
> table of some kind?) is being created in those 37 seconds, which is
> then lost and recreated when I re-run MOSES with a different value of
> a decoding parameter.
>
> If I want to eliminate this loading time, what is the best way
> forward? Is there a phrase table format that avoids a significant
> loading time (i.e. a 500MB phrase table loads in < 1 sec)? Should I
> try to find a way to run a single MOSES server but with different
> decoding parameters? I have a system with 130GB RAM and I am using a
> small phrase table that has been filtered for my decoding set.
>
> Thanks in advance,
>
> Roko
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170406/cb0249f8/attachment-0001.html

------------------------------

Message: 4
Date: Thu, 6 Apr 2017 14:30:35 +0400
From: Alexander Erdmann <ae1541@nyu.edu>
Subject: Re: [Moses-support] inconsistent number of probabilities
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAGTOGBdrQhhnR9KQxC2qfPsNSm9iSzg-G=RJN-e_hysyKigf-A@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

For reference, here is the output from the segFault:

Defined parameters (per moses.ini or switch):
config: /scratch/ae1541/EMNLP/seed/150000/iteration1/mosesTryingGZ.ini
distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty
PhraseDictionaryMemory name=TranslationModel0 num-features=4
path=/scratch/ae1541/EMNLP/seed/150000/iteration0/phrase-table.gz
LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=/scratch/ae1541/EMNLP/seed/150000/iteration0/reordering-table.gz
Distortion KENLM name=LM0 factor=0
path=/scratch/ae1541/EMNLP/seed/LM/LM.lev.lm order=5
input-factors: 0
input-file: seed/150000/iteration1/del00.egy
mapping: 0 T 0
threads: 12
weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3
0.3 Distortion0= 0.3 LM0= 0.5
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhrasePenalty
FeatureFunction: PhrasePenalty0 start: 2 end: 2
line=PhraseDictionaryMemory name=TranslationModel0 num-features=4
path=/scratch/ae1541/EMNLP/seed/150000/iteration0/phrase-table.gz
FeatureFunction: TranslationModel0 start: 3 end: 6
line=LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=/scratch/ae1541/EMNLP/seed/150000/iteration0/reordering-table.gz
FeatureFunction: LexicalReordering0 start: 7 end: 12
Initializing LexicalReordering..
line=Distortion
FeatureFunction: Distortion0 start: 13 end: 13
line=KENLM name=LM0 factor=0 path=/scratch/ae1541/EMNLP/seed/LM/LM.lev.lm
order=5
FeatureFunction: LM0 start: 14 end: 14
Loading the LM will be faster if you build a binary file.
Reading /scratch/ae1541/EMNLP/seed/LM/LM.lev.lm
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Loading UnknownWordPenalty0
Loading WordPenalty0
Loading PhrasePenalty0
Loading LexicalReordering0
Loading table into memory...done.
Loading Distortion0
Loading LM0
Loading TranslationModel0
Start loading text phrase table. Moses format : [1574.737] seconds
Reading /scratch/ae1541/EMNLP/seed/150000/iteration0/phrase-table.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Created input-output object : [1608.439] seconds
Translating: Translating: A y Translating: jAC Translating: Translating:
AbwTranslating: RRR w hA Al mAd mrwAn
sAdRRR jrydp A m trtr . . jrydp ywmy p . . bTyx yp . . mhlb yp bAl
tEycTranslating: w nA Al byAn Al tAly p
Line 1: Initialize search took Line 8: Initialize search took tqly p 0.006
seconds total
0.005 seconds total
Translating: t h y Translating: qwqAl A b mSr RRR rytc bAl Al dAxly p y
AxbAr ?
Line 9: Initialize search took 0.007 seconds total
w b h mfyc RRR
AxwAn hm Hd m Al mslmyn AnDrb
Line 11: Initialize search took Line 0: Initialize search took Al0.010
seconds total
Hrp 0.009 seconds total
Al Translating: AfAd Al mhlb jrydpmtHdv Al rsmy l wzArp Al nSAry bEd
Translating: yp Al m m A y h Avl Al dAxly w
An p AfAdt Line 7: Initialize search took qyAd ( Line xyr p ll TbE
Al mSlHp 8: Collecting options took Line 2: Initialize search took lhm Al
0.005 seconds at moses/Manager.cpp:AjElh bHv 0.015 seconds total
0.013 seconds total
wrAC Amn xyr mvyr Al Al y dwlp tlyfzywn Al sAbq Al mSry A ) Almbjl ytHdv b
Al An lsAn jnAQy h jrydp An bEd Al ftn A tHry w m w Al trtr Al bHv w mtTrf
f yTlb yn y mn Al AHdAv Al mslmyn
hm Line 6: Initialize search took 117
nA 0.020 seconds total
s *** bSegmentation fault
Register dump:

RAX: 0000000000000000 RBX: 00000000012e28e0 RCX: 0000000000000000
RDX: 00007f91b7413010 RSI: 0000000000000000 RDI: 00007f917400b9c0
RBP: 0000000000000000 R8 : 0000000000000002 R9 : 0000000000000038
R10: 00007f91aa8f3310 R11: 00007f91b64cfbf0 R12: 00007f917400b028
R13: 00007f9174004f60 R14: 00007f917400b9c0 R15: 0000000000000002
RSP: 00007f91aa8f3730

RIP: 00007f91b66e8833 EFLAGS: 00010246

CS: 0033 FS: 0000 GS: 0000

Trap: 0000000e Error: 00000004 OldMask: 00000000 CR2: 00000010

FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000
RIP: 00000000 RDP: 00000000

ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000
ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000
ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000
ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000
mxcsr: 1fa4
XMM0: 00000000000000000000000000000000 XMM1:
00000000000000000000000000000000
XMM2: 00000000000000000000000000000000 XMM3:
00000000000000000000000000000000
XMM4: 00000000000000000000000000000000 XMM5:
00000000000000000000000000000000
XMM6: 00000000000000000000000000000000 XMM7:
00000000000000000000000000000000
XMM8: 00000000000000000000000000000000 XMM9:
00000000000000000000000000000000
XMM10: 00000000000000000000000000000000 XMM11:
00000000000000000000000000000000
XMM12: 00000000000000000000000000000000 XMM13:
00000000000000000000000000000000
XMM14: 00000000000000000000000000000000 XMM15:
00000000000000000000000000000000

Backtrace:
y Sdd lA tjmyE hyj rAs wA
AHd/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZNK5Moses16LanguageModelKenIN2lm5ngram12ProbingModelEE19EvaluateWhenAppliedERKNS_10HypothesisEPKNS_7FFStateEPNS_24ScoreComponentCollectionE+0xa3)[0x7f91b66e8833]
Ely/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses10Hypothesis19EvaluateWhenAppliedERKNS_12SquareMatrixE+0xf0)[0x7f91b64cfab0]
Al
/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses12SearchNormal16ExpandHypothesisERKNS_10HypothesisERKNS_17TranslationOptionEf+0x165)[0x7f91b65484a5]
bED
/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses12SearchNormal19ExpandAllHypothesesERKNS_10HypothesisEmm+0x79)[0x7f91b6548829]

mALine
/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses12SearchNormal20ProcessOneHypothesisERKNS_10HypothesisE+0x34b)[0x7f91b6548beb]
10: Initialize search took
/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses12SearchNormal6DecodeEv+0x11c)[0x7f91b6548ffc]
DHAyA/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses7Manager6DecodeEv+0xe3)[0x7f91b64f9173]
sbyrw
/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses15TranslationTask3RunEv+0x1a5)[0x7f91b656f505]
f
/share/apps/NYUAD/mosesdecoder/3.0/lib/libmoses.so(_ZN5Moses10ThreadPool7ExecuteEv+0x89)[0x7f91b6724b99]
./share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_thread.so.1.57.0(+0x1211a)[0x7f91b54ee11a]
/usr/lib64/libpthread.so.0(+0x7dc5)[0x7f91b489cdc5]
y /usr/lib64/libc.so.6(clone+0x6d)[0x7f91b45ca28d]
.
Memory map:

tlk 00400000-0040b000 r-xp 00000000 00:2b 1254532671
/share/apps/NYUAD3/mosesdecoder/3.0/bin/moses
0060b000-0060c000 rw-p 0000b000 00:2b 1254532671
/share/apps/NYUAD3/mosesdecoder/3.0/bin/moses
012ce000-52974a000qd rw-p 00000000 00:00 0
[heap]
7f9174000000-7f9174021000 rw-p 00000000 00:00 0
7f9174021000-7f9178000000 ---p 00000000 00:00 0
7f9178000000-7f917807b000 rw-p 00000000 00:00 0
7f917807b000-7f917c000000 ---p 00000000 00:00 0
7f917c000000-7f917c034000 rw-p 00000000 00:00 0
7f917c034000-7f9180000000 ---p 00000000 00:00 0
7f9184000000-7f918409e000 rw-p 00000000 00:00 0
7f918409e000-7f9188000000 ---p 00000000 00:00 0
7f9188000000-7f9188047000 rw-p 00000000 00:00 0
7f9188047Al000-7f918c000000 ---p 00000000 00:00 0
7f918c000000-7f918c23d000 rw-p 00000000 00:00 0
7f918c23d000-7f9190000000 ---p 00000000 00:00 0
7f9190000000-7f9190106000 rw-p 00000000 00:00 0
7f9190106000-7f9194000000 ---p 00000000 00:00 0
7f9194000000-7f91940 c7000 rw-p 00000000 00:00 0
7f91940c9000-7f9198000000 ---p 00000000 00:00 0
7f9198000000-7f9198106000 rw-p 00000000 00:00 0
7f9198106000-7f919c000000 ---p 00000000 00:00 0
7f919c000000-7f919c165000 rw-p 00000000 00:00 0
7f919c165000-7f91a0000000 ---p tbyn00000000 00:00 0
7f919c167000-7f91a0000000 ---p 00000000 00:00 0
7f91a0000000-7f91a009f000 rw-p 00000000 00:00 0
7f91a009f000-7f91a4000000 ---p 00000000 00:00 0
7f91a4000000-7f91a4047000 rw-p 00000000 00:00 0
7f91a4047000-7f91a8000000 ---p 00000000 00 :00 0
7f91a80f0000-7f91a80f1000 ---p 00000000 00:00 0
7f91a80f1000-7f91a88f1000 rw-p 00000000 00:00 0
[stack:10106]
7f91a88f1000-7f91a88f2000 ---p 00000000 00:00 0
7f91a88f2000-7f91a90f2000 rw-p 00000000 00:00 0 AHdAv
[stack:10105]
7f91a90f2000-7f91a90f3000 ---p 00000000 00:00 0
7f91a90f3000-7f91a98f3000 rw-p 00000000 00:00 0
[stack:10104]
7f91a98f3000-7f91a98f4000 ---p 00000000 00:00 0
7f91a98f4000-7f91aa0f4000 rw-p 00000000 00:00 0
[stack:10103]
7f91aa0f4000-7f91aa0f5000 ---p 00000000 00:00 0
7f91aa0f5000-7f91aa8f5000 rw-p 00000000 00:00 0
[stack:10102]
7f91aa8f5000-7f91aa8f6000 ---p 00000000 00:00 0
7f91aa8f6000-7f91ab0f6000 rw-p 0lnA0000000 00:00 0
[stack:10101]
7f91ab0f6000-7f91ab0f7000 ---p 00000000 00:00 0
7f91ab0f7000-7f91ab8f7000 rw-p 00000000 00:00 0
[stack:10100]
7f91ab8f7000-7f91ab8f8000 ---p 00000000 00:00 0
7f91ab8f8000-7f9 1ac0f8000 rw-p 00000000 00:00 0
[stack:10099]
7f91ac0f8000-7f91ac0f9000 ---p 00000000 00:00 0
7f91ac0f9000-7f91ae8fa000 rw-p 00000000 00:00 0
[stack:10098]
7f91aef80000-7f91aef81000 ---p 00000000 00:00 0
w7f91aef81000-7f91af781000 rw-p 00000000 00:00 0
[stack:10097]
7f91af781000-7f91af782000 ---p 00000000 00:00 0
7f91af782000-7f91aff82000 rw-p 00000000 00:00 0
[stack:10096]
7f91aff82000-7f91aff83000 ---p 00 000000 00:00 0
7f91aff83000-7f91b1784000 rw-p 00000000 00:00 0
[stack:10095]
7f91b1886000-7f91b18c7000 rw-p 00000000 00:00 0
7f91b18c7000-7f91b409f000 rw-p 00000000 00:00 0
7f91b409f000-7f91b40ae000 r-xp 00000000 00:12 54468 An
/usr/lib64/libbz2.so.1.0.6
7f91b40ae000-7f91b42ad000 ---p 0000f000 00:12 54468
/usr/lib64/libbz2.so.1.0.6
7f91b42ad000-7f91b42ae000 r--p 0000e000 00:12 54468
/usr/lib64/libbz2.so.1.0.6
7f91b42ae00 0-7f91b42af000 rw-p 0000f000 00:12 54468
/usr/lib64/libbz2.so.1.0.6
7f91b42af000-7f91b42d3000 r-xp 00000000 00:12 54446
/usr/lib64/liblzma.so.5.0.99
7f91b42d3000-7f91b44d2000 ---p 00024000 00:12 54446 AlVy
/usr/lib64/liblzma.so.5.0.99
7f91b44d2000-7f91b44d3000 r--p 00023000 00:12 54446
/usr/lib64/liblzma.so.5.0.99
7f91b44d3000-7f91b44d4000 rw-p 00024000 00:12 54446
/usr/lib64/liblzma.so.5.0.99
7f91b44d4000-7f 91b468a000 r-xp 00000000 00:12 54398
/usr/lib64/libc-2.17.so
7f91b468a000-7f91b488a000 ---p 001b6000 00:12 54398
/usr/lib64/libc-2.17.so
7f91b488a000-7f91b488e000 r--p 001b6000 00:12 54398
/usrAl/lib64/libc-2.17.so
7f91b488e000-7f91b4890000 rw-p 001ba000 00:12 54398
/usr/lib64/libc-2.17.so
7f91b4890000-7f91b4895000 rw-p 00000000 00:00 0
7f91b4895000-7f91b48ab000 r-xp 00000000 00:12 54451
/usr/lib64/libpth read-2.17.so
7f91b48ab000-7f91b4aab000 ---p 00016000 00:12 54451
/usr/lib64/libpthread-2.17.so
7f91b4aab000-7f91b4aac000 r--p 00016000 00:12 54451
/usr/lib64/libpthread-2.17.so
7f91b4aac000-7f91b4aad000 rw-p 000170t00 00:12 54451
/usr/lib64/libpthread-2.17.so
7f91b4aad000-7f91b4ab1000 rw-p 00000000 00:00 0
7f91b4ab1000-7f91b4ac7000 r-xp 00000000 00:2b 390181951
/share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown-linux-gn
u/lib64/libgcc_s.so.1
7f91b4ac7000-7f91b4cc6000 ---p 00016000 00:2b 390181951
/share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown-linux-gnu/lib64/libgcc_s.so.1
7f91b4cc6000-7f91b4cc7000 r--p 00015000 00:2b 390181951
slfyyn
/share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown-linux-gnu/lib64/libgcc_s.so.1
7f91b4cc7000-7f91b4cc8000 rw-p 00016000 00:2b 390181951
/share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown-linux-gnu/lib64/libgcc_s.so.1
7 f91b4cc8000-7f91b4dc9000 r-xp 00000000 00:12 54414
/usr/lib64/libm-2.17.so
7f91b4dc9000-7f91b4fc8000 ---p 00101000 00:12 54414
/usr/lib64/libm-2.17.so
7f91b4fc8000-7f91b4fc9000 r--p 00100000 00:12 54414 bdw
/usr/lib64/libm-2.17.so
7f91b4fc9000-7f91b4fca000 rw-p 00101000 00:12 54414
/usr/lib64/libm-2.17.so
7f91b4fca000-7f91b50b5000 r-xp 00000000 00:2b 3536542694
/share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-
unknown-linux-gnu/4.9.3/libstdc++.so.6.0.20
7f91b50b5000-7f91b52b5000 ---p 000eb000 00:2b 3536542694
/share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/libstdc++.so.6.0.20
7f91b52b5000-7f91b52bd000 r--p 000eb000 00:hm 2b 3536542694
/share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknown-linux-gnu/4.9.3/libstdc++.so.6.0.20
7f91b52bd000-7f91b52bf000 rw-p 000f3000 00:2b 3536542694
/share/apps/NYUAD3/gcc/gcc/4.9.3/el7/lib/gcc/x86_64-unknomlAmHwn-linux-gnu/4.9.3/libstdc++.so.6.0.20
7f91b52bf000-7f91b52d4000 rw-p 00000000 00:00 0
7f91b52d4000-7f91b52db000 r-xp 00000000 00:12 54459
/usr/lib64/librt-2.17.so
7f91b52db000-7f91b54da000 ---p 00007000 00:12 54459
/usr/lib64/librt-2.17.so
7f91b54da000-7f91b54db000 r--p 00006000 00:12 54459
/usr/lib64/librt-2.17.so
7f91b54db000-7f91b54dc000 rw-p 00007000 00:12 54459
/usr/lib64/librt-2.17.so
7f91b54dc000-7f91b54fd000 r-xp 0mn0000000 00:2b 189775209
/share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_thread.so.1.57.0
7f91b54fd000-7f91b56fc000 ---p 00021000 00:2b 189775209
/share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/
avx2/1.57.0/lib/libboost_thread.so.1.57.0
7f91b56fc000-7f91b56ff000 rw-p 00020000 00:2b 189775209
/share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_thread.so.1.57.0
7f91b56ff000-7f91b5714000 r-xp 00000000 00:12 544h64
/usr/lib64/libz.so.1.2.7
7f91b5714000-7f91b5913000 ---p 00015000 00:12 54464
/usr/lib64/libz.so.1.2.7
7f91b5913000-7f91b5914000 r--p 00014000 00:12 54464
/usr/lib64/libz.so.1.2.7
7f91b591400 0-7f91b5915000 rw-p 00015000 00:12 54464
/usr/lib64/libz.so.1.2.7
7f91b5915000-7f91b5918000 r-xp 00000000 00:2b 189775207
/share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_system.so.1.57.0
7f91AcElb5918000-7f91b5b17000 ---p 00003000 00:2b 189775207
/share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_system.so.1.57.0
7f91b5b17000-7f91b5b18000 rw-p 00002000 00:2b 189775207
/share/apps/NYUAD3/boo
st/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_system.so.1.57.0
7f91b5b18000-7f91b5b2e000 r-xp 00000000 00:2b 189756212
/share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_iostreams.so.1.57.0
7f91b5b2e000-7f91bwADH5d2e000 ---p 00016000 00:2b 189756212
/share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_iostreams.so.1.57.0
7f91b5d2e000-7f91b5d30000 rw-p 00016000 00:2b 189756212
/share/apps/NYUAD3/boost/gcc_4.9.
3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_iostreams.so.1.57.0
7f91b5d30000-7f91b5d34000 r-xp 00000000 00:12 54392
/usr/lib64/libSegFault.so
7f91b5d34000-7f91b5f33000 ---p 00004000 00:12 54392
/usr/lib64/libSegFaultwA.so
7f91b5f33000-7f91b5f34000 r--p 00003000 00:12 54392
/usr/lib64/libSegFault.so
7f91b5f34000-7f91b5f35000 rw-p 00004000 00:12 54392
/usr/lib64/libSegFault.so
7f91b5f35000-7f91b5f38000 r-xp 00000000 00:12 54410
/usr/lib64/libdl-2.17.so
7f91b5f38000-7f91b6137000 ---p 00003000 00:12 54410
/usr/lib64/libdl-2.17.so
7f91b6137000-7f91b6138000 r--p 00002000 00:12 54410
/usr/lib64/libdl-2.17.so
7f91b6138000-7f90.0221b6139000 rw-p 00003000 00:12 54410
/usr/lib64/libdl-2.17.so
7f91b6139000-7f91b6150000 r-xp 00000000 00:2b 189756209
/share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_filesystem.so.1.57.0
7f91b seconds total6150000-7f91b634f000 ---p 00017000 00:2b 189756209

/share/apps/NYUAD3/boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_filesystem.so.1.57.0
7f91b634f000-7f91b6350000 rw-p 00016000 00:2b 189756209
/share/apps/NYUAD3/
boost/gcc_4.9.3/openmpi_1.10.2/avx2/1.57.0/lib/libboost_filesystem.so.1.57.0
7f91b6350000-7f91b6868000 r-xp 00000000 00:2b 2421011537
/share/apps/NYUAD3/mosesdecoder/3.0/lib/libmoses.so
7f91b6868000-7f91b6a67000 ---p 00518000 00:2b 24210115ftyl37
/share/apps/NYUAD3/mosesdecoder/3.0/lib/libmoses.so
7f91b6a67000-7f91b6a82000 rw-p 00517000 00:2b 2421011537
/share/apps/NYUAD3/mosesdecoder/3.0/lib/libmoses.so
7f91b6a82000-7f91b73b2000 rw-p 00000000 00:00 0
7f91b73b200 0-7f91b73d3000 r-xp 00000000 00:12 54384
/usr/lib64/ld-2.17.so
7f91b7413000-7f91b743b000 rw-p 00000000 00:00 0
7f91b745c000-7f91b74c6000 rw-p 00000000 00:00 0
7f91b75c7000-7f91b75d3000 rw-p 00000000 00:00 0
7f91b75d3000-7f91b75d4000Line r--p 00021000 00:12 54384
/usr/lib64/ld-2.17.so
7f91b75d4000-7f91b75d5000 rw-p 00022000 00:12 54384
/usr/lib64/ld-2.17.so
7f91b75d5000-7f91b75d6000 rw-p 00000000 00:00 0
7ffcb1849000-7ffcb186c000 rw-p 00000000 00p:00 0
[stack]
7ffcb191f000-7ffcb1921000 r-xp 00000000 00:00 0
[vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0
[vsyscall]
9: Collecting options took

On Thu, Apr 6, 2017 at 1:58 AM, Alexander Erdmann <ae1541@nyu.edu> wrote:

> Hi Moses Team,
>
> I'm running an MT system with 2 phrase tables and 2 reordering tables in
> conjunction using the default 4 probabilities in the former and 6
> probabilities in the latter.
>
> When I call it with this (and I'm running lots of jobs in parallel so I
> may run this command 20 times at once just varying the suffix on the input
> file 'del')
>
> $MOSESDECODER_ROOT/bin/moses -threads 4 -config
> $PWD/seed/$seedSize/iteration$iterations/moses.ini -input-file
> data/del$var 1> seed/$seedSize/iteration$iterations/del$var.lev 2>
> seed/$seedSize/iteration$iterations/del$var.log
>
> it never translates a single line, but it also doesn't die in the same
> place every time. Sometimes the error message is:
>
> Loading UnknownWordPenalty0
>
> Loading WordPenalty0
>
> Loading PhrasePenalty0
>
> Loading LexicalReordering0
>
> Loading table into memory...found inconsistent number of probabilities...
> found 4 expected 6
>
> and sometimes it loads the lexical reordering table just fine but
> segFaults while trying to translate the first line.
>
> I am positive that there are in fact 6 probabilities in every line of the
> reordering table so I'm especially confused about that error... any ideas?
>
> Here is my config file:
>
> # input factors
> [input-factors]
> 0
>
> # mapping steps
> [mapping]
> 0 T 0
> 1 T 1
>
> [distortion-limit]
> 6
>
> # feature functions
> [feature]
> UnknownWordPenalty
> WordPenalty
> PhrasePenalty
> PhraseDictionaryMemory name=TranslationModel0 num-features=4
> path=/scratch/ae1541/EMNLP/seed/$$$/iteration0/phrase-table
> PhraseDictionaryMemory name=TranslationModel1 num-features=4
> path=/scratch/ae1541/EMNLP/seed/$$$/iteration***/phrase-table
> input-factor=0 output-factor=0
> LexicalReordering name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/scratch/ae1541/EMNLP/seed/$$$/iteration0/reordering-table
> LexicalReordering name=LexicalReordering1 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/scratch/ae1541/EMNLP/seed/$$$/iteration***/reordering-table
> Distortion
> KENLM name=LM0 factor=0 path=/scratch/ae1541/EMNLP/seed/LM/LM.lev.lm
> order=5
>
> # dense weights for feature functions
> [weight]
> UnknownWordPenalty0= 1
> WordPenalty0= -1
> PhrasePenalty0= 0.2
> TranslationModel0= 0.2 0.2 0.2 0.2
> TranslationModel1= 0.2 0.2 0.2 0.2
> LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
> LexicalReordering1= 0.3 0.3 0.3 0.3 0.3 0.3
> Distortion0= 0.3
> LM0= 0.5
>
> Thanks!
>
> --
> Alex Erdmann
> PhD Student in Linguistics at The Ohio State University
> Visiting Scholar at NYU Abu Dhabi
>



--
Alex Erdmann
PhD Student in Linguistics at The Ohio State University
Visiting Scholar at NYU Abu Dhabi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170406/c9ea5a6c/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 126, Issue 6
*********************************************

0 Response to "Moses-support Digest, Vol 126, Issue 6"

Post a Comment

Subscribe to: Post Comments (Atom)