Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: BLEU score difference about 0.13 for one dataset is
normal? (Michael Denkowski)
2. Re: BLEU score difference about 0.13 for one dataset is
normal? (Tom Hoar)
3. Re: Kendall's tau metric (Arefeh Kazemi)
4. error during tuning step (factored training) (Tomasz Gawryl)
----------------------------------------------------------------------
Message: 1
Date: Sun, 11 Oct 2015 00:47:44 -0400
From: Michael Denkowski <michael.j.denkowski@gmail.com>
Subject: Re: [Moses-support] BLEU score difference about 0.13 for one
dataset is normal?
To: Davood Mohammadifar <davood_mf@hotmail.com>
Cc: Moses Support <moses-support@mit.edu>
Message-ID:
<CA+-Geg+N_uNafwcDVW7uCnC2CXQq1KD2L8m96wFbGoByAWRv-A@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi Davood,
Optimizers like MERT will give you a slightly different result every time
you run them, leading to variance in BLEU score. It's generally a good
idea to use multiple optimizer runs, especially when comparing two
systems. There's a good paper on hypothesis testing for MT that goes into
detail on this <http://aclweb.org/anthology/P/P11/P11-2031.pdf>. Some
other parts of a standard system like word alignment can also be
non-deterministic but the optimizer is the most frequent cause of
fluctuating metric scores.
Best,
Michael
On Sun, Oct 11, 2015 at 12:23 AM, Davood Mohammadifar <davood_mf@hotmail.com
> wrote:
> Hello every one
>
> I noticed different BLEU scores for same dataset. Also the difference is
> not so much and is about 0.13.
>
> I trained my dataset and tuned development set for Persian-English
> translation. after testing, the score was 21.95. For second time i did the
> same process and obtained 21.82. (my tools were mgiza, mert, ...)
>
> is this difference normal?
>
> My system:
> CPU: Core i7-4790K
> RAM: 16GB
> OS: ubuntu 12.04
>
> Thanks
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151011/491b8962/attachment-0001.html
------------------------------
Message: 2
Date: Sun, 11 Oct 2015 12:53:37 +0700
From: Tom Hoar <tahoar@precisiontranslationtools.com>
Subject: Re: [Moses-support] BLEU score difference about 0.13 for one
dataset is normal?
To: moses-support@mit.edu
Message-ID:
<4CD5DBFB-784C-49A0-970A-10928DE772F8@precisiontranslationtools.com>
Content-Type: text/plain; charset="utf-8"
Yes. Each tuning with the same test set will give you small variations in the final BLEU. Yours looks like they're in a normal range.
Date: Sun, 11 Oct 2015 04:23:56 +0000
From: Davood Mohammadifar <davood_mf@hotmail.com>
Subject: [Moses-support] BLEU score difference about 0.13 for one
dataset is normal?
To: Moses Support <moses-support@mit.edu>
Hello every one
I noticed different BLEU scores for same dataset. Also the difference is not so much and is about 0.13.
I trained my dataset and tuned development set for Persian-English translation. after testing, the score was 21.95. For second time i did the same process and obtained 21.82. (my tools were mgiza, mert, ...)
is this difference normal?
My system:
CPU: Core i7-4790K
RAM: 16GB
OS: ubuntu 12.04
Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151011/6650d71a/attachment-0001.html
------------------------------
Message: 3
Date: Sun, 11 Oct 2015 14:21:00 +0330
From: Arefeh Kazemi <arefeh.kazeminajafabadi@dcu.ie>
Subject: Re: [Moses-support] Kendall's tau metric
To: moses-support@mit.edu, Philipp Koehn <phi@jhu.edu>
Message-ID:
<CAB2b+QVSBOx-wde=G_v1pP=QAfNmycUZauYwLjuCPvnZLavFEA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi
Thanks Philipp for the reply.
1- when I use the suggested command, after the first run of tuning , It
gives "segmentation fault" error.
the exact command that I use:
nohup nice $SCRIPTS_ROOTDIR/training/mert-moses.pl --mertargs="--sctype
KENDALL,BLEU --scconfig weights:0.6+0.4" m.en m.fa \
$MOSESCHART --decoder-flags="-threads all" $WorkingDir/moses.ini \
--mertdir ~/mosesdecoder/bin/ \
--no-filter-phrase-table > mert.out
the error :
Exit code: 139
ERROR: Failed to run '../mert-work/extractor.sh'
error in extract.err file:
extractor.sh: line 3: 22321 Segmentation fault
2- how can I use KENDALL+Bleu to evaluate the test set?
Thanks
Arefeh
?
--
*Email Disclaimer"This e-mail and any files transmitted with it are confidential and are intended solely for use by the addressee. Any unauthorised dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this e-mail in error, please notify the sender and delete the message. Any views or opinions presented in this e-mail may solely be the views of the author and cannot be relied upon as being those of Dublin City University. E-mail communications such as this cannot be guaranteed to be virus-free, timely, secure or error-free and Dublin City University does not accept liability for any such matters or their consequences. Please consider the environment before printing this e-mail."S?anadh R?omhphoist"T? an r?omhphost seo agus aon chomhad a sheoltar leis faoi r?n agus is lena ?s?id ag an seola? agus sin amh?in ?. T? cosc ioml?n ar scaipeadh, dh?ileadh n? ch?ipe?il neamh?daraithe ar an teachtaireacht seo agus ar ao!
n cheangalt?n at? ag dul leis. M? t? an r?omhphost seo faighte agat tr? dhearmad cuir sin in i?l le do thoil don seolt?ir agus scrios an teachtaireacht. D?fh?adfadh s? gurb iad tuairim? an ?dair agus sin amh?in at? in aon tuairim? no dearctha? at? curtha i l?thair sa r?omhphost seo agus n?or ch?ir glacadh leo mar thuairim? n? dhearctha? Ollscoil Chathair Bhaile ?tha Cliath. N? ghlactar leis go bhfuil cumars?id r?omhphoist den s?rt seo saor ? v?reas, in am, sl?n, n? saor ? earr?id agus n? ghlacann Ollscoil Chathair Bhaile ?tha Cliath le dliteanas in aon ch?s den s?rt sin n? as aon iarmhairt a d?eascr?dh astu. Cuimhnigh ar an timpeallacht le do thoil sula gcuireann t? an r?omhphost seo i gcl?."*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151011/39504e1c/attachment-0001.html
------------------------------
Message: 4
Date: Sun, 11 Oct 2015 18:00:03 +0200
From: "Tomasz Gawryl" <tomasz.gawryl@skrivanek.pl>
Subject: [Moses-support] error during tuning step (factored training)
To: <moses-support@mit.edu>
Message-ID: <000201d1043d$e42f5b40$ac8e11c0$@gawryl@skrivanek.pl>
Content-Type: text/plain; charset="us-ascii"
Hi,
My factored training ends core dumped during tuning step.:
/home/moses/working/experiments/FACTORED/lm/ACROSS=pos.lm.1: line 11:
warning: n
on-zero probability for <unk> in closed-vocabulary LM
terminate called after throwing an instance of 'util::Exception'
what(): moses/TranslationModel/RuleTable/PhraseDictionaryOnDisk.cpp:100
in vi
rtual void Moses::PhraseDictionaryOnDisk::InitializeForInput(const
ttasksptr&) t
hrew util::Exception because `obj->GetMisc("NumTargetFactors") !=
m_output.size(
)'.
On-disk phrase table has 1 target factors.. The ini file specified 2 target
factors
Aborted (core dumped)
Exit code: 134
My config file is similar to example config file for factored trainings.
What should I do to fix it?
Regards
Tomek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151011/b3fba1f4/attachment.html
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 108, Issue 38
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 108, Issue 38"
Post a Comment