Moses-support Digest, Vol 107, Issue 9

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: really weird phrase table crash ..... (Vincent Nguyen)
2. Re: Several Issues with Baseline and EMS (Anita Pal)


----------------------------------------------------------------------

Message: 1
Date: Wed, 2 Sep 2015 13:34:28 +0200
From: Vincent Nguyen <vnguyen@neuf.fr>
Subject: Re: [Moses-support] really weird phrase table crash .....
To: "'moses-support'" <moses-support@mit.edu>
Message-ID: <55E6DEC4.9050400@neuf.fr>
Content-Type: text/plain; charset=windows-1252; format=flowed


unsure but might be a bug.

this one crashes everytime

training-options = "-mgiza -mgiza-cpus 8 -sort-compress gzip -sort-parallel 8 -cores 8"

but this works

training-options = "-mgiza -mgiza-cpus 8 -sort-compress gzip -sort-parallel 4 -cores 4"


multithreading issue ?


Le 01/09/2015 17:41, Christophe Servan a ?crit :
> Hello Vincent,
> Did you checked whether you have enough disk space?
>
> Best,
>
> Christophe
>
>
> -----Message d'origine-----
> De : moses-support-bounces@mit.edu [mailto:moses-support-bounces@mit.edu] De
> la part de Vincent Nguyen
> Envoy? : mardi 1 septembre 2015 17:07
> ? : moses-support <moses-support@mit.edu>
> Objet : [Moses-support] really weird phrase table crash .....
>
> Hi,
>
> I don't know what is happening, but during the phrase table building
> (inverse part) in the ../model/tmp.23625 directory I have plenty of files :
> but 4 .coc files are missing (number 14 , 15 , 23, 24 don't know why) and
> then when putting all together it crashes because can't find these 4
>
> phrase-table.half.0000014.gz.coc (and same for 15 23 24)
>
> (out of 37 parts)
>
> any clue ?
>
>
> Using SCRIPTS_ROOTDIR: /home/moses/mosesdecoder/scripts using gzip
> (6) score phrases @ Tue Sep 1 07:17:00 CEST 2015
> (6.1) creating table half
> /home/moses/working/model/phrase-table.2.half.f2e @ Tue Sep 1 07:17:00 CEST
> 2015
> /home/moses/mosesdecoder/scripts/generic/score-parallel.perl 8 "sort
> --compress-program gzip --parallel 8"
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/extract.sorted.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/phrase-table.2.half.f2e.gz --GoodTuring 0
> Executing: /home/moses/mosesdecoder/scripts/generic/score-parallel.perl
> 8 "sort --compress-program gzip --parallel 8"
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/extract.sorted.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/phrase-table.2.half.f2e.gz --GoodTuring 0
> (6.1) creating table half
> /home/moses/working/model/phrase-table.2.half.e2f @ Tue Sep 1 07:17:00 CEST
> 2015
> /home/moses/mosesdecoder/scripts/generic/score-parallel.perl 8 "sort
> --compress-program gzip --parallel 8"
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/extract.inv.sorted.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/phrase-table.2.half.e2f.gz --Inverse 1
> Executing: /home/moses/mosesdecoder/scripts/generic/score-parallel.perl
> 8 "sort --compress-program gzip --parallel 8"
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/extract.inv.sorted.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/phrase-table.2.half.e2f.gz --Inverse 1 using gzip
> Started Tue Sep 1 07:17:00 2015 Started Tue Sep 1 07:17:00 2015
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.0.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000000.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.1.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000001.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.2.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000002.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.3.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000003.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.4.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000004.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.5.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000005.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.6.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000006.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.7.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000007.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.8.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000008.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.9.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000009.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.10.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000010.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.11.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000011.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.12.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000012.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.13.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000013.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.14.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000014.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.15.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000015.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.16.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000016.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.17.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000017.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.18.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000018.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.19.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000019.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.20.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000020.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.21.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000021.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.22.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000022.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.23.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000023.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.24.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000024.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.25.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000025.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.26.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000026.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.27.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000027.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.28.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000028.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.29.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000029.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.30.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000030.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.31.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000031.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.32.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000032.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.33.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000033.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.34.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000034.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.35.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000035.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.36.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000036.gz
> --Inverse 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23626/extract.37.gz
> /home/moses/working/model/lex.2.e2f
> /home/moses/working/model/tmp.23626/phrase-table.half.0000037.gz
> --Inverse 2>> /dev/stderr
> /home/moses/working/model/tmp.23626/run.0.sh/home/moses/working/model/tmp.23
> 626/run.1.sh/home/moses/working/model/tmp.23626/run.2.sh/home/moses/working/
> model/tmp.23626/run.3.sh/home/moses/working/model/tmp.23626/run.4.sh/home/mo
> ses/working/model/tmp.23626/run.5.sh/home/moses/working/model/tmp.23626/run.
> 6.sh/home/moses/working/model/tmp.23626/run.7.sh/home/moses/mosesdecoder/scr
> ipts/../bin/score
> /home/moses/working/model/tmp.23625/extract.0.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000000.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.1.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000001.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.2.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000002.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.3.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000003.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.4.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000004.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.5.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000005.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.6.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000006.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.7.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000007.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.8.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000008.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.9.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000009.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.10.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000010.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.11.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000011.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.12.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000012.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.13.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000013.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.14.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000014.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.15.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000015.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.16.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000016.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.17.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000017.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.18.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000018.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.19.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000019.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.20.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000020.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.21.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000021.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.22.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000022.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.23.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000023.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.24.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000024.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.25.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000025.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.26.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000026.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.27.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000027.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.28.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000028.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.29.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000029.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.30.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000030.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.31.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000031.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.32.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000032.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.33.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000033.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.34.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000034.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.35.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000035.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.36.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000036.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/mosesdecoder/scripts/../bin/score
> /home/moses/working/model/tmp.23625/extract.37.gz
> /home/moses/working/model/lex.2.f2e
> /home/moses/working/model/tmp.23625/phrase-table.half.0000037.gz
> --GoodTuring 2>> /dev/stderr
> /home/moses/working/model/tmp.23625/run.7.sh/home/moses/working/model/tmp.23
> 625/run.5.sh/home/moses/working/model/tmp.23625/run.0.sh/home/moses/working/
> model/tmp.23625/run.3.sh/home/moses/working/model/tmp.23625/run.2.sh/home/mo
> ses/working/model/tmp.23625/run.4.shgunzip
> -c /home/moses/working/model/tmp.23625/phrase-table.half.*.gz 2>>
> /dev/stderr | gzip -c > /home/moses/working/model/phrase-table.2.half.f2e.gz
> 2>> /dev/stderr gunzip -c
> /home/moses/working/model/tmp.23626/phrase-table.half.*.gz 2>>
> /dev/stderr| LC_ALL=C sort --compress-program gzip --parallel 8 -T
> /home/moses/working/model/tmp.23626 | gzip -c >
> /home/moses/working/model/phrase-table.2.half.e2f.gz 2>> /dev/stderr can't
> open pipe to
> /home/moses/working/model/tmp.23625/phrase-table.half.0000014.gz.coc at
> /home/moses/mosesdecoder/scripts/generic/score-parallel.perl line 269.
> Exit code: 2
> ERROR: Scoring of phrases failed at
> /home/moses/mosesdecoder/scripts/training/train-model.perl line 1768.
> rm -rf /home/moses/working/model/tmp.23626
> Finished Tue Sep 1 15:37:15 2015
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support


------------------------------

Message: 2
Date: Wed, 2 Sep 2015 17:34:18 +0200
From: Anita Pal <fujitahibaraki@gmail.com>
Subject: Re: [Moses-support] Several Issues with Baseline and EMS
To: Jeroen Vermeulen <jtv@precisiontranslationtools.com>
Cc: moses-support@mit.edu
Message-ID:
<CABX1R6=aVytgNOW8s74tqQU6oTQAJ10k9LS4+fbybbng5m4C1w@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

For the time being, I'm trying to finish building the baseline system. I've
just been following the commands as listed on the Moses website.

It's still not running. I tried out countless variants -- it's still
telling me to Set IRSTLM environment variable with path to irstlm.

liam@liam-Aspire-E1-571G:~$ export=$HOME/irstlm; ~/irstlm/bin/build-lm.sh \
> -i news-commentary-v8.fr-en.sb.en \
> -t ./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en
~/irstlm/bin/compile-lm \
> --text=yes \
> news-commentary-v8.fr-en.lm.en.gz \
> news-commentary-v8.fr-en.arpa.en
Set IRSTLM environment variable with path to irstlm
liam@liam-Aspire-E1-571G:~$

--> this was the last command I entered.

The full (recommended) command via the website is:

mkdir ~/lm
cd ~/lm
~/irstlm/bin/add-start-end.sh \
< ~/corpus/news-commentary-v8.fr-en.true.en \
> news-commentary-v8.fr-en.sb.en
export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh \
-i news-commentary-v8.fr-en.sb.en \
-t ./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en
~/irstlm/bin/compile-lm \
--text=yes \
news-commentary-v8.fr-en.lm.en.gz \
news-commentary-v8.fr-en.arpa.en

Has anyone got a sample command of how they got this to run?




On Wed, Sep 2, 2015 at 11:59 AM, Jeroen Vermeulen <
jtv@precisiontranslationtools.com> wrote:

> Yes, the semicolon ends one command and starts another.
>
> Don't set HOME. The shell does that for you. If you want a different
> setting than your actual home directory, give it a different name or you
> might confuse programs you run from that shell.
>
>
> Jeroen
>
> On September 2, 2015 1:26:43 AM GMT+07:00, Anita Pal <
> fujitahibaraki@gmail.com> wrote:
>
>> Hey!
>>
>> I'm really, really new to Linux and have no idea how to set /change the
>> config file when it comes to running the experiments as described here (
>> http://www.statmt.org/moses/?n=moses.baseline)
>>
>> home-dir = /home/liam/
>>
>> working-dir =/home/liam/working/experiments
>> moses-src-dir = /home/liam/mosesdecoder
>> moses-script-dir = home/liam/working/experiments (this is where config is
>> located?)
>> moses-bin-dir = /home/liam/mosesdecoder/bin
>> external-bin-dir =/home/liam/mosesdecoder/tools
>> data-dir =/home/liam/corpus
>> train-dir =/home/liam/corpus/training
>> dev-dir = /home/liam/corpus/dev
>> irstlm-dir =/home/liam/irstlm/bin
>>
>> Is this correct? I really have no idea ):
>>
>> Because I just keep getting errors no matter what I do. I have the same
>> problem when it comes to training the language model via IRSTLM. For
>> example:
>>
>> export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh \
>> -i news-commentary-v8.fr-en.sb.en \
>> -t ./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en
>> ~/irstlm/bin/compile-lm \
>> --text=yes \
>> news-commentary-v8.fr-en.lm.en.gz \
>> news-commentary-v8.fr-en.arpa.en
>>
>> I assume this is a separate command. I keep getting errors though I do set HOME=/home/liam/irstlm.
>>
>> What am I doing wrong?
>>
>>
>>
>> ------------------------------
>>
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150902/19366383/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 107, Issue 9
*********************************************

Tweet