Home » Uncategories » Moses-support Digest, Vol 103, Issue 46

Moses-support Digest, Vol 103, Issue 46

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Sentence-level features (Varvara Logacheva)
2. decoder fails to reach hypothesis stage (kwame porter robinson)
3. Re: decoder fails to reach hypothesis stage
(kwame porter robinson)


----------------------------------------------------------------------

Message: 1
Date: Tue, 19 May 2015 19:45:30 +0100
From: Varvara Logacheva <varvara.logacheva@gmail.com>
Subject: [Moses-support] Sentence-level features
To: moses-support@mit.edu
Message-ID:
<CAN79iOkBQdggfuJ-teNBXnwDVEifoDgJ0Md-njOf25V-GOB41w@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Dear Moses users,

could you please tell if Moses has any mechanism of adding sentence-level
features?
I mean features that can be evaluated only when the whole hypothesis is
generated.

Thank you.

Best,

Varvara.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150519/973cb2d0/attachment-0001.htm

------------------------------

Message: 2
Date: Tue, 19 May 2015 14:49:04 -0400
From: kwame porter robinson <k.porter.robinson@gmail.com>
Subject: [Moses-support] decoder fails to reach hypothesis stage
To: moses-support <moses-support@mit.edu>
Message-ID:
<CA+RrxgWJY8PZXeBiJS0CNmTDBfj3k6MrvO-Kcxfwzx7h3meu7Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

>From the toy phrase based tutorial at [1], I'm using a modified phrase
table (I replaced the German with phrases in a different source language,
the target language is still english). And I'm finding that a certain
source phrase causes the decoder to seg fault.

Digging deeper into the verbose output I see that after the translation
option collection it never enters the hypothesis stage even though there is
at least one path available (see the verbose output below). The source
phrase has 8 tokens and the translation "this is a" covers [0-5][6-7].

What would prevent the decoder from entering the hypothesis generation
stage? I did notice that the last 3 lines ([1 ; 1-1], etc) are not present
in successful source translation output. Thank you for your time.

[1] http://www.statmt.org/moses/?n=Moses.Tutorial

cheers
kwame
---
Defined parameters (per moses.ini or switch):
beam-threshold: 0
config: phrase-model/moses.ini
feature: KENLM name=LM factor=0 order=3 num-features=1
path=/home/kwame/Moses/moses-linux-64bit/sample-models/lm/europarl.srilm.gz
WordPenalty PhraseDictionaryMemory input-factor=0 output-factor=0
path=/home/kwame/Moses/moses-linux-64bit/sample-models/phrase-model/fixed.sorted.bit
num-features=1 table-limit=100
input-factors: 0
mapping: T 0
report-segmentation:
stack: 1000
verbose: 3 2
weight: WordPenalty0= 0 LM= 1 PhraseDictionaryMemory0= 1
input type is: text input
line=KENLM name=LM factor=0 order=3 num-features=1
path=/home/kwame/Moses/moses-linux-64bit/sample-models/lm/europarl.srilm.gz
FeatureFunction: LM start: 0 end: 0
Loading the LM will be faster if you build a binary file.
Reading
/home/kwame/Moses/moses-linux-64bit/sample-models/lm/europarl.srilm.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
**The ARPA file is missing <unk>. Substituting log10 probability -100.000.
**************************************************************************************************
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhraseDictionaryMemory input-factor=0 output-factor=0
path=/home/kwame/Moses/moses-linux-64bit/sample-models/phrase-model/fixed.sorted.bit
num-features=1 table-limit=100
FeatureFunction: PhraseDictionaryMemory0 start: 2 end: 2
DecodeFeature:
Loading LM
Loading WordPenalty0
Loading PhraseDictionaryMemory0
Start loading text phrase table. Moses format : [0.824] seconds
Reading
/home/kwame/Moses/moses-linux-64bit/sample-models/phrase-model/fixed.sorted.bit
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
DecodeStep():
outputFactors=FactorMask<0>
conflictFactors=FactorMask<>
newOutputFactors=FactorMask<0>
Created input-output object : [0.825] seconds
The global weight vector looks like this: core=(1.000,0.000,1.000)
Translating line 0 in thread id 140062576498432
Translating: 0 1 1 0 1 0 0 0
InitializeForInput( LM ) = 0.000
InitializeForInput( WordPenalty0 ) = 0.000
InitializeForInput( PhraseDictionaryMemory0 ) = 0.000
Line 0: Initialize search took 0.001 seconds total

...

Translation Option Collection
a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515
[[0..0]]core=(0.000,-1.000,0.000)
the :: term=: nonterm=: c=-5.789 core=(0.000,-1.000,-0.916) c=-5.789
[[0..2]]core=(0.000,-1.000,-0.916)
this :: term=: nonterm=: c=-8.002 core=(0.000,-1.000,-2.303) c=-8.002
[[0..3]]core=(0.000,-1.000,-2.303)
this is :: term=: nonterm=: c=-10.291 core=(0.000,-2.000,-0.223) c=-10.291
[[0..5]]core=(0.000,-2.000,-0.223)
or :: term=: nonterm=: c=-7.720 core=(0.000,-1.000,-2.303) c=-7.720
[[1..2]]core=(0.000,-1.000,-2.303)
is :: term=: nonterm=: c=-4.922 core=(0.000,-1.000,0.000) c=-4.922
[[2..3]]core=(0.000,-1.000,0.000)
it is :: term=: nonterm=: c=-8.821 core=(0.000,-2.000,-0.223) c=-8.821
[[2..5]]core=(0.000,-2.000,-0.223)
a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515
[[3..3]]core=(0.000,-1.000,0.000)
is :: term=: nonterm=: c=-4.922 core=(0.000,-1.000,0.000) c=-4.922
[[4..5]]core=(0.000,-1.000,0.000)
old :: term=: nonterm=: c=-8.945 core=(0.000,-1.000,-0.223) c=-8.945
[[4..6]]core=(0.000,-1.000,-0.223)
a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515
[[5..5]]core=(0.000,-1.000,0.000)
's :: term=: nonterm=: c=-6.116 core=(0.000,-1.000,0.000) c=-6.116
[[5..6]]core=(0.000,-1.000,0.000)
an :: term=: nonterm=: c=-6.413 core=(0.000,-1.000,0.000) c=-6.413
[[5..6]]core=(0.000,-1.000,0.000)
a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515
[[6..6]]core=(0.000,-1.000,0.000)
's :: term=: nonterm=: c=-6.116 core=(0.000,-1.000,0.000) c=-6.116
[[6..7]]core=(0.000,-1.000,0.000)
an :: term=: nonterm=: c=-6.413 core=(0.000,-1.000,0.000) c=-6.413
[[6..7]]core=(0.000,-1.000,0.000)
a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515
[[7..7]]core=(0.000,-1.000,0.000)

[1 ; 1-1]

[1 ; 2-2]

[1 ; 4-4]
---
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150519/233eb609/attachment-0001.htm

------------------------------

Message: 3
Date: Tue, 19 May 2015 21:06:16 -0400
From: kwame porter robinson <k.porter.robinson@gmail.com>
Subject: Re: [Moses-support] decoder fails to reach hypothesis stage
To: moses-support <moses-support@mit.edu>
Message-ID:
<CA+RrxgXsYOwxuWG=2yh46yi_8RaYPjcBGETtTctBvcaaBeww9w@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

correction:

Adding UnknownWordPenalty to my moses.ini corrected this issue and the
decoder chose the expected translation. A backtrace indicated that
ProcessOneUnknownWord was trying to assign a score but there was no
UnknownWordPenalty feature (and hence a score).

On Tue, May 19, 2015 at 2:49 PM, kwame porter robinson <
k.porter.robinson@gmail.com> wrote:

> Hi,
>
> From the toy phrase based tutorial at [1], I'm using a modified phrase
> table (I replaced the German with phrases in a different source language,
> the target language is still english). And I'm finding that a certain
> source phrase causes the decoder to seg fault.
>
> Digging deeper into the verbose output I see that after the translation
> option collection it never enters the hypothesis stage even though there is
> at least one path available (see the verbose output below). The source
> phrase has 8 tokens and the translation "this is a" covers [0-5][6-7].
>
> What would prevent the decoder from entering the hypothesis generation
> stage? I did notice that the last 3 lines ([1 ; 1-1], etc) are not present
> in successful source translation output. Thank you for your time.
>
> [1] http://www.statmt.org/moses/?n=Moses.Tutorial
>
> cheers
> kwame
> ---
> Defined parameters (per moses.ini or switch):
> beam-threshold: 0
> config: phrase-model/moses.ini
> feature: KENLM name=LM factor=0 order=3 num-features=1
> path=/home/kwame/Moses/moses-linux-64bit/sample-models/lm/europarl.srilm.gz
> WordPenalty PhraseDictionaryMemory input-factor=0 output-factor=0
> path=/home/kwame/Moses/moses-linux-64bit/sample-models/phrase-model/fixed.sorted.bit
> num-features=1 table-limit=100
> input-factors: 0
> mapping: T 0
> report-segmentation:
> stack: 1000
> verbose: 3 2
> weight: WordPenalty0= 0 LM= 1 PhraseDictionaryMemory0= 1
> input type is: text input
> line=KENLM name=LM factor=0 order=3 num-features=1
> path=/home/kwame/Moses/moses-linux-64bit/sample-models/lm/europarl.srilm.gz
> FeatureFunction: LM start: 0 end: 0
> Loading the LM will be faster if you build a binary file.
> Reading
> /home/kwame/Moses/moses-linux-64bit/sample-models/lm/europarl.srilm.gz
>
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> **The ARPA file is missing <unk>. Substituting log10 probability -100.000.
>
> **************************************************************************************************
> line=WordPenalty
> FeatureFunction: WordPenalty0 start: 1 end: 1
> line=PhraseDictionaryMemory input-factor=0 output-factor=0
> path=/home/kwame/Moses/moses-linux-64bit/sample-models/phrase-model/fixed.sorted.bit
> num-features=1 table-limit=100
> FeatureFunction: PhraseDictionaryMemory0 start: 2 end: 2
> DecodeFeature:
> Loading LM
> Loading WordPenalty0
> Loading PhraseDictionaryMemory0
> Start loading text phrase table. Moses format : [0.824] seconds
> Reading
> /home/kwame/Moses/moses-linux-64bit/sample-models/phrase-model/fixed.sorted.bit
>
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>
> ****************************************************************************************************
> DecodeStep():
> outputFactors=FactorMask<0>
> conflictFactors=FactorMask<>
> newOutputFactors=FactorMask<0>
> Created input-output object : [0.825] seconds
> The global weight vector looks like this: core=(1.000,0.000,1.000)
> Translating line 0 in thread id 140062576498432
> Translating: 0 1 1 0 1 0 0 0
> InitializeForInput( LM ) = 0.000
> InitializeForInput( WordPenalty0 ) = 0.000
> InitializeForInput( PhraseDictionaryMemory0 ) = 0.000
> Line 0: Initialize search took 0.001 seconds total
>
> ...
>
> Translation Option Collection
> a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515
> [[0..0]]core=(0.000,-1.000,0.000)
> the :: term=: nonterm=: c=-5.789 core=(0.000,-1.000,-0.916) c=-5.789
> [[0..2]]core=(0.000,-1.000,-0.916)
> this :: term=: nonterm=: c=-8.002 core=(0.000,-1.000,-2.303) c=-8.002
> [[0..3]]core=(0.000,-1.000,-2.303)
> this is :: term=: nonterm=: c=-10.291 core=(0.000,-2.000,-0.223)
> c=-10.291 [[0..5]]core=(0.000,-2.000,-0.223)
> or :: term=: nonterm=: c=-7.720 core=(0.000,-1.000,-2.303) c=-7.720
> [[1..2]]core=(0.000,-1.000,-2.303)
> is :: term=: nonterm=: c=-4.922 core=(0.000,-1.000,0.000) c=-4.922
> [[2..3]]core=(0.000,-1.000,0.000)
> it is :: term=: nonterm=: c=-8.821 core=(0.000,-2.000,-0.223) c=-8.821
> [[2..5]]core=(0.000,-2.000,-0.223)
> a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515
> [[3..3]]core=(0.000,-1.000,0.000)
> is :: term=: nonterm=: c=-4.922 core=(0.000,-1.000,0.000) c=-4.922
> [[4..5]]core=(0.000,-1.000,0.000)
> old :: term=: nonterm=: c=-8.945 core=(0.000,-1.000,-0.223) c=-8.945
> [[4..6]]core=(0.000,-1.000,-0.223)
> a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515
> [[5..5]]core=(0.000,-1.000,0.000)
> 's :: term=: nonterm=: c=-6.116 core=(0.000,-1.000,0.000) c=-6.116
> [[5..6]]core=(0.000,-1.000,0.000)
> an :: term=: nonterm=: c=-6.413 core=(0.000,-1.000,0.000) c=-6.413
> [[5..6]]core=(0.000,-1.000,0.000)
> a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515
> [[6..6]]core=(0.000,-1.000,0.000)
> 's :: term=: nonterm=: c=-6.116 core=(0.000,-1.000,0.000) c=-6.116
> [[6..7]]core=(0.000,-1.000,0.000)
> an :: term=: nonterm=: c=-6.413 core=(0.000,-1.000,0.000) c=-6.413
> [[6..7]]core=(0.000,-1.000,0.000)
> a :: term=: nonterm=: c=-5.515 core=(0.000,-1.000,0.000) c=-5.515
> [[7..7]]core=(0.000,-1.000,0.000)
>
> [1 ; 1-1]
>
> [1 ; 2-2]
>
> [1 ; 4-4]
> ---
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150519/6febfb04/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 103, Issue 46
**********************************************

0 Response to "Moses-support Digest, Vol 103, Issue 46"

Post a Comment

Subscribe to: Post Comments (Atom)