Moses-support Digest, Vol 113, Issue 15

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Reordering Constraints with "zone" and "wall" markup
(Patrik Lambert)
2. Re: What is sentence alignment score in training step 2 RUN
GIZA++ (Philipp Koehn)
3. Re: In-domain dictionary - where to include it (Philipp Koehn)


----------------------------------------------------------------------

Message: 1
Date: Mon, 7 Mar 2016 20:01:34 +0100
From: Patrik Lambert <patrik.l@webinterpret.com>
Subject: [Moses-support] Reordering Constraints with "zone" and "wall"
markup
To: moses-support@mit.edu
Message-ID:
<CAOXrhy2M6QmX2yZAwerGBcO7THOW4vMg2hAvArDV6D=3vqMcCw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

I am trying to specify reordering constraints with the <zone> tag and it
does not work as I would expect. The output corresponding to an input
sentence I translated from English to Spanish is as follows:

input: <zone> My favourite </zone> hotel in Cologne
output: mi |0-0| hotel favorito |1-2| en Colonia |3-4|

I understand a "zone" as a sequence to be translated as a block, with no
reordering with outside material. According to this, the translation of "My
favourite" should be contiguous. Here it is not the case since the
translation of "hotel" is inserted between "mi" and "favorito".

Actually even with the <wall/> tag I obtain the same output:
input: My favourite <wall/> hotel in Cologne
output: mi |0-0| hotel favorito |1-2| en Colonia |3-4|

The command I use is the following one:
moses -search-algorithm 1 -cube-pruning-pop-limit 5000 -s 5000 -t
-text-type "test" -v 0 -f filtered.ini -xml-input exclusive < input

Do I do something wrong or did I misunderstand what <zone> and <wall/>
imply?

Thanks,

Patrik

--

*Patrik Lambert*

NLP Researcher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160307/e2b0bac2/attachment-0001.html

------------------------------

Message: 2
Date: Mon, 7 Mar 2016 15:36:04 -0500
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] What is sentence alignment score in
training step 2 RUN GIZA++
To: Jyoti Srivastava <jyoti11s@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDCKvreVOxmyQ=et53HewRRr2F21=tMtLwPQKxoTbNh88Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

this ought to be the score for p(a|e,f) (or p(e,a|f)?) according the the
last IBM Model that was run - typically IBM Model 4.

The IBM Model 4 formula is quite complex. It considers word alignment,
fertility, reordering, etc. You can find it here:
Peter F. Brown and Stephen A. Della-Pietra and Vincent J. Della-Pietra and
Robert L. Mercer (1993): The Mathematics of Statistical Machine
Translation, Computational Linguistics
http://acl.ldc.upenn.edu/J/J93/J93-2003.pdf

-phi

On Sun, Mar 6, 2016 at 6:25 PM, Jyoti Srivastava <jyoti11s@gmail.com> wrote:

> Dear Sir/Madam
>
> I have following doubt. Please clear it.
>
> http://www.statmt.org/moses/?n=FactoredTraining.RunGIZA
>
> Above link is for "Training step 2 RUN GIZA++"
>
> there for every sentence pair, following three information are given:
> 1. Source length
> 2. Target length
> 3.
> * Alignment score*
> What is this alignment score. This is not for word alignment then how did
> this score is computed.
>
> Thanks and regards
>
> Jyoti Srivastava
> +91 9687404917
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160307/d2ef1539/attachment-0001.html

------------------------------

Message: 3
Date: Mon, 7 Mar 2016 15:39:25 -0500
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] In-domain dictionary - where to include
it
To: Joze Kadivec <joze.kadivec@zop-cr.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDAAfG6v-mR=J5npzvZGgB7Zhv=G=OKCf3H72R-Ff_0NOw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

it is likely that it did not run the decoder et al.

Check the STDERR or STDOUT for the command in which the moses decoder was
invoked.
Run just that command manually and check for error messages.

-phi

On Sun, Mar 6, 2016 at 1:18 AM, Joze Kadivec <joze.kadivec@zop-cr.com>
wrote:

> Hello,
>
>
>
> Can you please help me with the following error?
>
>
>
> When I run the Tuning process with the following command ((nohup nice
> ~/mosesdecoder/scripts/training/mert-moses.pl ~/corpus/
> news-test2008.true.fr ~/corpus/news-test2008.true.en
> ~/mosesdecoder/bin/moses train/model/moses.ini --mertdir
> ~/mosesdecoder/bin/ --decoder-flags="-threads 4" &> mert.out)) the process
> stops immediately after it finishes calculating the best translation with
> the following exit codes:
>
>
>
> Failed to get order of scores from nbestlist 'run1.best100.out': No such
> file or directory at /home/kjoze/mosesdecoder/scripts/training/
> mert-moses.pl line 1474.
>
> coring the nbestlist.
>
> exec: /home/kjoze/working/mert-work/extractor.sh
>
> Executing: /home/kjoze/working/mert-work/extractor.sh > extract.out 2>
> extract.err
>
> Exit code: 1
>
> ERROR: Failed to run '/home/kjoze/working/mert-work/extractor.sh'. at
> /home/kjoze/mosesdecoder/scripts/training/mert-moses.pl line 1744.
>
>
>
> The contents of mert-moses.pl file at line 1474 is:
>
>
>
> sub get_order_of_scores_from_nbestlist {
>
> # read the first line and interpret the ||| label: num num num label2:
> num ||| column in nbestlist
>
> # return the score labels in order
>
> my $fname_or_source = shift;
>
> # print STDERR "Peeking at the beginning of nbestlist to get order of
> scores: $fname_or_source\n";
>
> 1474? open my $fh, $fname_or_source or die "Failed to get order of
> scores from nbestlist '$fname_or_source': $!";
>
> my $line = <$fh>;
>
> close $fh;
>
> die "Line empty in nbestlist '$fname_or_source'" if !defined $line;
>
> my ($sent, $hypo, $scores, $total) = split /\|\|\|/, $line;
>
> $scores =~ s/^\s*|\s*$//g;
>
> die "No scores in line: $line" if $scores eq "";
>
>
>
> my @order = ();
>
> my $label = undef;
>
> my $sparse = 0; # we ignore sparse features here
>
> foreach my $tok (split /\s+/, $scores) {
>
>
>
> Please advise. Does changing the news-test2008.true.fr file contents has
> any influence?
>
>
>
> Thank you.
>
> Kind regards,
>
> Jo?e
>
>
>
>
>
> *From:* phkoehn@gmail.com [mailto:phkoehn@gmail.com] *On Behalf Of *Philipp
> Koehn
> *Sent:* Friday, March 04, 2016 11:03 PM
> *To:* Joze Kadivec
> *Cc:* moses-support@mit.edu
> *Subject:* Re: [Moses-support] In-domain dictionary - where to include it
>
>
>
> Hi,
>
>
>
> the most straight-forward way is to add it to the phrase table.
>
>
>
> You could add it multiple times, if you want it to be given more weight
> than the regular parallel corpus.
>
>
>
> -phi
>
>
>
> On Tue, Feb 23, 2016 at 3:10 PM, Joze Kadivec <joze.kadivec@zop-cr.com>
> wrote:
>
> Hello,
>
> I just recently installed Moses and would like to use it as part of my
> masters thesis on Domain adaptation for less-resourced Slovenian language.
> I
> was given some domain specific dictionaries for which I need your advice.
> Where should I include it? Should I simply paste their contents to the end
> of aligned corpora files used to create translation model? Is there another
> way to use the dictionaries more efficiently?
>
> Thank you.
>
> Kind regards,
> Jo?e Kadivec
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160307/9bbe0005/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 113, Issue 15
**********************************************

Tweet