Moses-support Digest, Vol 99, Issue 13

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: ERROR: Lexical reordering scoring failed (Philipp Koehn)
2. Re: Training transliteration model outside EMS (Rajnath Patel)
3. Europarl monolingual pipeline (Kenneth Heafield)
4. Re: Europarl monolingual pipeline (Philipp Koehn)


----------------------------------------------------------------------

Message: 1
Date: Tue, 6 Jan 2015 10:36:23 -0800
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] ERROR: Lexical reordering scoring failed
To: Jon Olds <joft_uk@yahoo.co.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDDb9f9eeah5V=NHd3w1r+6XRdDW9E0dfbSzibYDDQbGig@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hi,

you are building a hierarchical phrase-based model,
so you cannot have a lexicalized reordering model.

Remove "-reordering msd-bidirectional-fe" from your
command line.

-phi

On Mon, Jan 5, 2015 at 12:51 AM, Jon Olds <joft_uk@yahoo.co.uk> wrote:
> Hi,
>
> I?m trying to build a hierarchical model using the same (cleaned) data
> that I have used successfully to build a phrase model and I keep getting
> the following error:
>
> ERROR: Lexical reordering scoring failed at
> /home/ubuntu/tools/mosesdecoder/scripts/training/train-model.perl line 1776.
>
> The command I used was:
>
> nohup nice ~/tools/mosesdecoder/scripts/training/train-model.perl
> -root-dir train -corpus ~/data/tok/base.clean -f fr -e en -alignment
> grow-diag-final-and -reordering msd-bidirectional-fe -lm
> 0:3:$HOME/data/lm/base.blm.en:8 -external-bin-dir ~/tools/trainingtools
> -hierarchical -glue-grammar -max-phrase-length 5 >& training.out &
>
> Any ideas?
>
> Many thanks,
>
> Jon
>
> /home/ubuntu/tools/mosesdecoder/scripts/generic/score-parallel.perl 1
> "sort " /home/ubuntu/tools/mosesdecoder/scripts/../bin/score
> /home/ubuntu/working/train/model/extract.inv.sorted.gz
> /home/ubuntu/working/train/model/lex.e2f
> /home/ubuntu/working/train/model/rule-table.half.e2f.gz --Inverse
> --Hierarchical 1
> Executing:
> /home/ubuntu/tools/mosesdecoder/scripts/generic/score-parallel.perl 1
> "sort " /home/ubuntu/tools/mosesdecoder/scripts/../bin/score
> /home/ubuntu/working/train/model/extract.inv.sorted.gz
> /home/ubuntu/working/train/model/lex.e2f
> /home/ubuntu/working/train/model/rule-table.half.e2f.gz --Inverse
> --Hierarchical 1
> Started Sun Jan 4 12:55:12 2015
> ln -s /home/ubuntu/working/train/model/extract.inv.sorted.gz
> /home/ubuntu/working/train/model/tmp.25446/extract.0.gz
> /home/ubuntu/tools/mosesdecoder/scripts/../bin/score
> /home/ubuntu/working/train/model/tmp.25446/extract.0.gz
> /home/ubuntu/working/train/model/lex.e2f
> /home/ubuntu/working/train/model/tmp.25446/phrase-table.half.0000000.gz
> --Inverse --Hierarchical 2>> /dev/stderr
> /home/ubuntu/working/train/model/tmp.25446/run.0.shgunzip -c
> /home/ubuntu/working/train/model/tmp.25446/phrase-table.half.*.gz 2>>
> /dev/stderr| LC_ALL=C sort -T
> /home/ubuntu/working/train/model/tmp.25446 | gzip -c >
> /home/ubuntu/working/train/model/rule-table.half.e2f.gz 2>> /dev/stderr
> rm -rf /home/ubuntu/working/train/model/tmp.25446
> Finished Sun Jan 4 13:15:56 2015
> (6.6) consolidating the two halves @ Sun Jan 4 13:15:56 UTC 2015
> Executing: /home/ubuntu/tools/mosesdecoder/scripts/../bin/consolidate
> /home/ubuntu/working/train/model/rule-table.half.f2e.gz
> /home/ubuntu/working/train/model/rule-table.half.e2f.gz /dev/stdout
> --Hierarchical | gzip -c > /home/ubuntu/working/train/model/rule-table.gz
> Consolidate v2.0 written by Philipp Koehn
> consolidating direct and indirect rule tables
> processing hierarchical rules
> Executing: rm -f /home/ubuntu/working/train/model/rule-table.half.*
> (7) learn reordering model @ Sun Jan 4 13:25:58 UTC 2015
> (7.1) [no factors] learn reordering model @ Sun Jan 4 13:25:58 UTC 2015
> (7.2) building tables @ Sun Jan 4 13:25:58 UTC 2015
> Executing:
> /home/ubuntu/tools/mosesdecoder/scripts/../bin/lexical-reordering-score
> /home/ubuntu/working/train/model/extract.o.sorted.gz 0.5
> /home/ubuntu/working/train/model/reordering-table. --model "wbe msd
> wbe-msd-bidirectional-fe"
> Lexical Reordering Scorer
> scores lexical reordering models of several types (hierarchical,
> phrase-based and word-based-extraction
> terminate called after throwing an instance of 'util::ErrnoException'
> what(): util/file.cc:68 in int util::OpenReadOrThrow(const char*)
> threw ErrnoException because `-1 == (ret = open(name, 00))'.
> No such file or directory while opening
> /home/ubuntu/working/train/model/extract.o.sorted.gz
> Aborted (core dumped)
> Exit code: 134
> ERROR: Lexical reordering scoring failed at
> /home/ubuntu/tools/mosesdecoder/scripts/training/train-model.perl line 1776.
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support



------------------------------

Message: 2
Date: Wed, 7 Jan 2015 00:38:14 +0530
From: Rajnath Patel <patelrajnath@gmail.com>
Subject: Re: [Moses-support] Training transliteration model outside
EMS
To: Anoop (?????) <anoop.kunchukuttan@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAE-r4umDcdfk1uHSS=Fd1bD__knzXRf8tPNOJPT07M0H9Bq0Rg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Thank you.

The shared document was very useful. Finally I was able to train the
transliteration system. For English-Hindi training, I had have to comment
line no 60 in ~/mosesdecoder/scripts/Transliteration/clean.pl {#$retur =
deleteEnglish($lang1, $lang2, $_); }.


On Tue, Jan 6, 2015 at 7:05 PM, Anoop (?????) <anoop.kunchukuttan@gmail.com>
wrote:

> Hi Rajnath,
>
> This was discussed earlier in the following thread:
>
> https://www.mail-archive.com/moses-support%40mit.edu/msg10740.html
>
> Ratish compiled this document on using transliteration without EMS based
> on the discussion:
>
>
> https://docs.google.com/document/d/1G9RjczZXWGHU6byJFORf6uToItph1jU_piL53wQhGXg/edit
>
> I have tried following the documentation and was able to run it
> successfully.
>
> Regards
> Anoop.
>
> On Tue, Jan 6, 2015 at 4:27 PM, Rajnath Patel <patelrajnath@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I am trying to use instructions given on moses site (
>> http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc5) to train a
>> transliteration model using parallel bilingual corpus, but unable to train.
>> Actually its not clear to me what exactly 'alignment' switch is suppose to
>> refer? If possible put a dummy script with a sample training data on moses
>> site. It will be easy to use. Please have a look at following script and
>> suggest what I am doing wrong?
>>
>> Thank You.
>>
>> *Shell Script:*
>> $SCRIPTS_ROOTDIR/Transliteration/train-transliteration-module.pl \
>> --corpus-f /home/raj/anuvad/xlit/data/corpus.mar \
>> --corpus-e /home/raj/anuvad/xlit/data/corpus.hin \
>> --alignment /home/raj/anuvad/xlit/align \
>> --moses-src-dir /home/raj/smt/decoder/mosesdecoder \
>> --external-bin-dir /home/raj/smt/decoder/mgiza-master/mgizapp/bin \
>> --input-extension mar \
>> --output-extension hin \
>> --srilm-dir /home/raj/smt/decoder/srilm/bin \
>> --out-dir /home/raj/anuvad/xlit/exp1 \
>> >& /home/raj/anuvad/xlit/training.out &
>>
>> Note: '*align*' is empty directory in '*alignment*' switch.
>>
>> --
>> Regards:
>> Raj Nath Patel
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
>
> --
> I claim to be a simple individual liable to err like any other fellow
> mortal. I own, however, that I have humility enough to confess my errors
> and to retrace my steps.
>
> http://flightsofthought.blogspot.com
>



--
Regards:
Raj Nath Patel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150107/4cf9ed61/attachment-0001.htm

------------------------------

Message: 3
Date: Tue, 06 Jan 2015 14:09:38 -0500
From: Kenneth Heafield <moses@kheafield.com>
Subject: [Moses-support] Europarl monolingual pipeline
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <54AC32F2.5090207@kheafield.com>
Content-Type: text/plain; charset=utf-8

Dear Moses,

Where does this data come from?

http://www.statmt.org/wmt13/training-monolingual-europarl-v7.tgz

Specifically, if I wanted non-WMT languages, then I can download
Europarl from http://www.statmt.org/europarl/ .

There are some tools, like a perl script to strip XML, but that also
strips out <P> tags which are meant to be preserved for
split-sentences.perl. And I don't think split-sentences.perl was
designed to run before stripping XML but could be wrong.

Does one write a custom XML strip program to remove all the tags except
<P> then pass it to split-sentences.perl?

Kenneth


------------------------------

Message: 4
Date: Tue, 6 Jan 2015 11:24:37 -0800
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] Europarl monolingual pipeline
To: Kenneth Heafield <moses@kheafield.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDC8SG4KyYqWgxNWcSMZVjJBgJ7hMtpRt_HvD2QJjMYf+Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

the Perl script that was used to build this corpus is:

#!/usr/bin/perl -w

use strict;
my ($l) = @ARGV;

my $data = "/home/pkoehn/statmt/data/europarl-v7";
my $tools = "/home/pkoehn/statmt/data/europarl-v7/tools";
my $preprocessor = "$tools/split-sentences.perl -q";

die("ERROR: no data for language $l") unless -e "$data/txt/$l";
open(SPLIT,"cat $data/txt/$l/ep-00-0* $data/txt/$l/ep-0[123456789]*
$data/txt/$l/ep-[19]* | $preprocessor -l $l |");
while(<SPLIT>) {
next if /^\s*$/;
next if /^</;
print $_;
}
close(SPLIT);


The sentence splitting code is in the tools package that comes
with the Europarl source release.

-phi

On Tue, Jan 6, 2015 at 11:09 AM, Kenneth Heafield <moses@kheafield.com>
wrote:

> Dear Moses,
>
> Where does this data come from?
>
> http://www.statmt.org/wmt13/training-monolingual-europarl-v7.tgz
>
> Specifically, if I wanted non-WMT languages, then I can download
> Europarl from http://www.statmt.org/europarl/ .
>
> There are some tools, like a perl script to strip XML, but that
> also
> strips out <P> tags which are meant to be preserved for
> split-sentences.perl. And I don't think split-sentences.perl was
> designed to run before stripping XML but could be wrong.
>
> Does one write a custom XML strip program to remove all the tags
> except
> <P> then pass it to split-sentences.perl?
>
> Kenneth
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150106/01773273/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 99, Issue 13
*********************************************

Tweet