Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: FinalState in Moses decoder (Hieu Hoang)
2. Re: low BLEU Score (EN->DE) (Hieu Hoang)
3. Re: System requiremnts for Moses (Hegde, Sujay)
4. Eclipse for moses smt PROBLEM (Despina Mouratidi)
5. MERT's Powell Search (liling tan)
----------------------------------------------------------------------
Message: 1
Date: Tue, 8 Dec 2015 22:41:40 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] FinalState in Moses decoder
To: Vu Thuong Huyen <huyenvt2211@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbjV8Aw0qqXGhfh-qjtSC+W1-UqOv0_Nm6hCO1VGDJDwKA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi Huyen
Hieu Hoang
http://www.hoang.co.uk/hieu
On 5 December 2015 at 04:32, Vu Thuong Huyen <huyenvt2211@gmail.com> wrote:
> Hi all,
>
> I?m integrating my LM in Mosesdecoder. I followed the SRI.cpp and
> NeuralLMWrapper.cpp. I don?t know how to assign value for *finalState
> variable. Could you explain for me how does it affect when decoding? I
> logged out the context when the system call ?GetValue? function in my LM
> and in SRILM LM, the number of contexts were different.
>
The set the finalState to a pointer that represent the context. For
example, if you have a trigram LM, and the last trigram was 'a b c', then
the finaState should represent 'b c'.
SRI and IRST store their LM in a tree, the finalState is the memory address
of node that stores the bigram 'b c'. KENLM isn't a trie, it sets the
finalState to a hash of the words 'b c'
>
>
> LMResult LanguageModelSRI::GetValue(VocabIndex wordId, VocabIndex
> *context) const
>
> {
>
> LMResult ret;
>
> ret.score = FloorScore(TransformLMScore(m_srilmModel->wordProb( wordId,
> context)));
>
> ret.unknown = (wordId == m_unknownId);
>
> return ret;
>
> }
>
>
>
>
> Which is value of *ret.score*? Log Probability or Probability?
>
log probability
>
>
> Best Regards,
>
> Huyen.
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151208/0e7a1520/attachment-0001.html
------------------------------
Message: 2
Date: Tue, 8 Dec 2015 22:45:16 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] low BLEU Score (EN->DE)
To: Raphael Hoeps <raphael.hoeps@gmx.net>, "moses-support@mit.edu"
<moses-support@mit.edu>
Message-ID: <56675D7C.4060609@gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
this url has some experiments for en-de
http://www.statmt.org/moses/RELEASE-3.0/models/en-de/evaluation/
It was trained on Europarl, tuned and test on news data, as you can see
if you look at the config files
The BLEU scores are comparable to yours.
On 06/12/15 10:25, Raphael Hoeps wrote:
> Hi,
> two weeks ago you already helped me to find a very stupid mistake in my
> moses system setup. Now everything seems to work quite well, but I am a
> little surprised about my (pretty low?) BLEU Score of only 10.
>
> I sticked to this tutorial, but did it for translating English to German:
> http://www.statmt.org/moses/?n=Moses.Baseline
>
> I used the same English and the corresponding German corpora as it is
> done in the tutorial, but I cut down the development corpora from 2000
> to 500 lines to speed up the process. (My laptop is quite old).
>
> So, do you think a BLEU Score of 10 is realistic or did I make a
> mistake? In the tutorial the score was 24, but of course it will be
> lower in a EN->DE system. Did anyone else ever setup a EN->DE baseline
> system? What scores did you get?
>
> Thanks a lot,
> Raphael
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
--
Hieu Hoang
http://www.hoang.co.uk/hieu
------------------------------
Message: 3
Date: Wed, 9 Dec 2015 07:30:56 +0000
From: "Hegde, Sujay" <Sujay.Hegde@xerox.com>
Subject: Re: [Moses-support] System requiremnts for Moses
To: Philipp Koehn <phi@jhu.edu>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>,
"MudaliarMudaliar, Preeti J" <preeti.mudaliarmudaliar@xerox.com>
Message-ID:
<586EA7C483504E48870F5BF54319B6EC4EDA89@USA7109MB006.na.xerox.net>
Content-Type: text/plain; charset="utf-8"
HI Philip,
I could not find max-phrase-length parameter in the Experiment config file.
I could find only the below:
[CORPUS]
### tools to use to prepare the data
#
#tokenizer =
#lowercaser =
### long sentences are filtered out, since they slow down GIZA++
# and are a less reliable source of data. set here the maximum
# length of a sentence
#
max-sentence-length = 80
Thanks and Regards,
Sujay,
Xerox Business Services, Bangalore, India
-----Original Message-----
From: phkoehn@gmail.com [mailto:phkoehn@gmail.com] On Behalf Of Philipp Koehn
Sent: 04 December 2015 19:44
To: Hegde, Sujay
Cc: moses-support@mit.edu; MudaliarMudaliar, Preeti J
Subject: Re: [Moses-support] System requiremnts for Moses
Hi,
if you are using experiment.perl, then the config parameter for phrase length is
max-phrase-length = 5
Similarly, if you call train-model.perl directly, then the switch is
$MOSES/scripts/training/train-model.perl -max-phrase-length 5 [...]
You can check the size of your model by looking at the size of the
*minphr* *minlexr* and *binlm* files. These are binary data files that are loaded into RAM (see files referenced in moses.ini).
Typically, the language model is the largest, because typically it is trained with large amounts of additional monolingual data.
-phi
On Fri, Dec 4, 2015 at 12:22 AM, Hegde, Sujay <Sujay.Hegde@xerox.com> wrote:
> Hi Phillip,
>
> How do we limit phrase length during training .Is there a config parameter in moses training config file?
>
> Is the phrase table the biggest model or the language model? ----> We have 6-7 phrase tables that are combined in a log-linear fashion during decoding.
>
>
>
>
> Thanks and Regards,
> Sujay,
> Xerox Business Services, Bangalore, India
>
>
> -----Original Message-----
> From: phkoehn@gmail.com [mailto:phkoehn@gmail.com] On Behalf Of
> Philipp Koehn
> Sent: 03 December 2015 21:52
> To: Hegde, Sujay
> Cc: moses-support@mit.edu; MudaliarMudaliar, Preeti J
> Subject: Re: [Moses-support] System requiremnts for Moses
>
> Hi,
>
> having such long sentences should cause all kinds of problems with word alignment, so I am bit puzzled that they still show up when pruning the phrase table.
>
> A good way to prune the phrase table is to limit the length of phrases (max 5 does no harm, even max 4 is not a big deal), and reduce low probability phrase pairs ($MOSES/scripts/training/threshold-filter.perl).
>
> Is the phrase table the biggest model or the language model? For the latter, there are several compression options.
>
> -phi
>
> On Thu, Dec 3, 2015 at 12:32 AM, Hegde, Sujay <Sujay.Hegde@xerox.com> wrote:
>> HI Philipp,
>>
>>
>>
>> Thanks a lot.
>>
>>
>>
>> Actually it?s a VIRTUAL machine.
>>
>>
>>
>> Also we have compressed the models into .minphr and
>> .minlexr but we couldn?t prune it as while pruning we got an error
>> saying some of the sentences in the Corpus are too long and it cannot be pruned.
>>
>>
>>
>> We used pruning using SALM and get the following error:
>>
>>
>>
>> /mnt/hd1/git/salm/Bin/Linux/Index/IndexSA.O64
>> opensub.train.it
>>
>> Initialize vocabulary file: opensub.train.it.id_voc
>>
>> Loading existing vocabulary file: opensub.train.it.id_voc
>>
>> Total 100 word types loaded
>>
>> Max VocID=100
>>
>> Sentence 4152148 has more than 256 words. Can not handle such long sentence.
>> Please cut it short first!
>>
>>
>>
>> Is there anything we could do about the above?
>>
>>
>>
>>
>>
>>
>>
>> Thanks and Regards,
>>
>> Sujay,
>>
>> Xerox Business Services, Bangalore, India
>>
>>
>>
>> From: phkoehn@gmail.com [mailto:phkoehn@gmail.com] On Behalf Of
>> Philipp Koehn
>> Sent: 03 December 2015 03:13
>> To: Hegde, Sujay
>> Cc: moses-support@mit.edu
>> Subject: Re: [Moses-support] System requiremnts for Moses
>>
>>
>>
>> Hi,
>>
>>
>>
>> the machine you have is certainly sufficient even for large models.
>>
>>
>>
>> If you are running two language pairs in parallel and run into RAM
>> problems, you may want to look into ways to compress the model files
>> (phrase table, reordering table, language model) using either more
>> efficient data structures (e.g., various KENLM options), or pruning the models.
>>
>>
>>
>> -phi
>>
>>
>>
>>
>>
>> On Tue, Dec 1, 2015 at 5:08 AM, Hegde, Sujay <Sujay.Hegde@xerox.com> wrote:
>>
>> Dear Moses Admin,
>>
>>
>>
>> We are using Moses decoder for commercial environment.
>>
>>
>>
>> We have 132GB RAM, 1TB disk and quadcore Virtual
>> Machine with CentOs OS.
>>
>>
>>
>> We have 2 language pairs installed, and when running
>> both the models together the Translation hangs(Takes a LONG time).
>>
>> It is fine when we run only one language model.
>>
>>
>>
>> Is there any Specific System requirements needed for moses?
>>
>> Please let me know
>>
>>
>>
>> Thanks and Regards,
>>
>> Sujay,
>>
>> Xerox Business Services, Bangalore, India
>>
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
------------------------------
Message: 4
Date: Wed, 9 Dec 2015 10:41:47 +0200
From: Despina Mouratidi <dmouratidi@gmail.com>
Subject: [Moses-support] Eclipse for moses smt PROBLEM
To: Moses-support@mit.edu
Message-ID:
<CAJJG54=Gf2ZDbj4Z0dBtnnGuXSAgdJDJY9Oh02wNVBGw8HutnQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Hello, i was follow the instructions of the video for using eclipse
but i dont have eclipse luna. I have eclipse 3.8 which hasnt c++
extension. I downloaded eclipse mars but i cant use it through command
line for moses. Any ideas?
Thanks in advance
Despina
------------------------------
Message: 5
Date: Wed, 9 Dec 2015 12:18:45 +0100
From: liling tan <alvations@gmail.com>
Subject: [Moses-support] MERT's Powell Search
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAKzPaJJ58neQ2uZyt5uXZen7k=axCaqAcMV3neUMmLtDRQ1-Kw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Dear Moses devs/users,
We are going through the slides for MT tuning on
http://mt-class.org/jhu/slides/lecture-tuning.pdf and we couldn't figure
out what does "?ai + bi" on slide 31 refer to.
What are the values for "ai" and "bi"? Are they numbers from the nbest
list?
According to the algorithm on slide 37 of
http://mt-class.org/jhu/slides/lecture-tuning.pdf, is line 6 where the ?ai
+ bi computation occurs?
compute line l: parameter value ? score
>From the nbest list we have lines as such:
0 ||| including options , ????? buy 20 ?????? planes , ??????????? volume -
? ??? 26 ?????????? ???????? . ||| LexicalReordering0= -3.12525 0 0
-7.34869 0 0 Distortion0= 0 LM0= -111.207 WordPenalty0= -18 PhrasePenalty0=
17 TranslationModel0= -12.8271 -8.45991 -11.4888 -11.3076 ||| -746.163
Let's say we are tuning the first parameter for LexicalReodering0 for this
sentence, is it that we only calculate:
? -3.12525 * -746.163
Is ai = 3.12525 for this sentence? Is bi = -746.163? What is bi suppose to
be?
Thanks in advance for the explanation!
Regards,
Liling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151209/e5179c18/attachment.html
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 110, Issue 18
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 110, Issue 18"
Post a Comment