Moses-support Digest, Vol 93, Issue 5

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Smoothing in Language Model (Pranjal Das)
2. Regarding language model (Pranjal Das)
3. Filtering Issue (Judah Schvimer)
4. Re: Moses and BiDi Languages (Michal Ziemski)


----------------------------------------------------------------------

Message: 1
Date: Thu, 3 Jul 2014 12:32:56 +0530
From: Pranjal Das <pranjal4456@gmail.com>
Subject: [Moses-support] Smoothing in Language Model
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAAGh44yCsFT_Q+MdyUnr_roz-henAUABzKq9P8Wv6wxt5AbtsA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

Hi everyone, while coounti n-grams, sometimes we come to a situation
where we get zero probability. What method is applied then ? Please
explain.
In some websites I have seen that Add-one smoothing is used to solve this.

--

Regards,

*Pranjal Das*
Department of Information Technology,
Gauhati University Institute of Science and Technology,
Phone- +91-8399879454


------------------------------

Message: 2
Date: Thu, 3 Jul 2014 12:50:07 +0530
From: Pranjal Das <pranjal4456@gmail.com>
Subject: [Moses-support] Regarding language model
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAAGh44zajUF619UdW1At2t62znq5P03JaJGJbbdN7wWaE9pAHw@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

While calculating bigram probability, suppose for a sentence "He likes
to play football", does P(He|<start>) and P(<end>|football) calculated
?
--

Regards,

*Pranjal Das*
Department of Information Technology,
Gauhati University Institute of Science and Technology,
Phone- +91-8399879454


------------------------------

Message: 3
Date: Thu, 3 Jul 2014 09:50:19 -0400
From: Judah Schvimer <judah.schvimer@mongodb.com>
Subject: [Moses-support] Filtering Issue
To: moses-support@mit.edu
Message-ID:
<CALF9aB7aEQTO1YNm2aMXNhH7reZYX8MNTwRbD-CASysF7hjijQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

I have a script that basically does all of the training steps for me and
it's worked many times before, but suddenly I got this error with a
different corpus. It doesn't occur when I use the moses.ini file in
train/model/ or in mert-work/, just when I use the one in filtered/. Any
idea what's wrong? Something appears to be weird around the 39-grams step.

Defined parameters (per moses.ini or switch):
config: /home/judah/archive8/3/working/filtered/moses.ini
distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty
PhraseDictionaryBinary name=TranslationModel0 table-limit=20 num-features=4
path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1
input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0
num-features=6 type=hier-msd-bidirectional-fe-allff input-factor=0
output-factor=0
path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe
Distortion IRSTLM name=LM0 factor=0 path=/home/judah/archive8/3/lm/
train.en-es.blm.es order=3
input-factors: 0
mapping: 0 T 0
weight: LexicalReordering0= 0.389589 0.0418995 0.0286706 0.0187875
0.0287628 0.00958344 Distortion0= 0.0583275 LM0= -0.00740405 WordPenalty0=
0.0123212 PhrasePenalty0= 0.0413057 TranslationModel0= 0.0360237 0.040722
0.266723 0.0198795 UnknownWordPenalty0= 1
/home/judah/mosesdecoder/bin
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhrasePenalty
FeatureFunction: PhrasePenalty0 start: 2 end: 2
line=PhraseDictionaryBinary name=TranslationModel0 table-limit=20
num-features=4
path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1
input-factor=0 output-factor=0
FeatureFunction: TranslationModel0 start: 3 end: 6
line=LexicalReordering name=LexicalReordering0 num-features=6
type=hier-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe
FeatureFunction: LexicalReordering0 start: 7 end: 12
Initializing LexicalReordering..
line=Distortion
FeatureFunction: Distortion0 start: 13 end: 13
line=IRSTLM name=LM0 factor=0 path=/home/judah/archive8/3/lm/
train.en-es.blm.es order=3
FeatureFunction: LM0 start: 14 end: 14
Loading UnknownWordPenalty0
Loading WordPenalty0
Loading PhrasePenalty0
Loading LexicalReordering0
binary file loaded, default OFF_T: -1
Loading Distortion0
Loading LM0
In LanguageModelIRST::Load: nGramOrder = 3
Language Model Type of /home/judah/archive8/3/lm/train.en-es.blm.es is 1
Language Model Type is 1
mmap
loadtxt_ram()
3-grams: reading 0 entries
done level 3
1-grams: reading 0 entries
done level 1
8-grams: reading 0 entries
done level 8
2-grams: reading 0 entries
done level 2
39-grams: reading 3991252117 entries
moses: util.cpp:289: int parseline(std::istream&, int, ngram&, float&,
float&): Assertion `howmany == (Order+ 1) || howmany == (Order + 2)' failed.
[1] 5062 abort (core dumped) /home/judah/mosesdecoder/bin/moses -f < >


Thanks,
Judah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140703/433b88f5/attachment-0001.htm

------------------------------

Message: 4
Date: Thu, 3 Jul 2014 10:19:17 -0400
From: Michal Ziemski <ziemski@un.org>
Subject: Re: [Moses-support] Moses and BiDi Languages
To: falko.schaefer@sap.com, pkoehn@inf.ed.ac.uk
Cc: moses-support@mit.edu
Message-ID:
<OFF4091A25.97D900D1-ON85257D0A.004E272C-85257D0A.004EABDF@un.org>
Content-Type: text/plain; charset="us-ascii"

An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140703/9fbb8e57/attachment-0001.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 93, Issue 5
********************************************

Tweet