Moses-support Digest, Vol 207, Issue 2

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
https://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. WMT24 Metrics Task: Call for Participation (Eleftherios Avramidis)


----------------------------------------------------------------------

Message: 1
Date: Tue, 4 Jun 2024 16:01:32 +0000
From: Eleftherios Avramidis <eleftherios.avramidis@dfki.de>
To: moses-support <moses-support@mit.edu>
Subject: [Moses-support] WMT24 Metrics Task: Call for Participation
Message-ID:
<VI1PR06MB5677A6BDDBC62245E072B9C5F1F82@VI1PR06MB5677.eurprd06.prod.outlook.com>

Content-Type: text/plain; charset="utf-8"

Hi all,

the 9th Conference on Machine Translation (WMT24), collocated with EMNLP 2024, will be featuring this year the Shared Task on evaluation of Automatic Metrics. We are looking for both reference-based metrics and reference-free metrics to evaluate the quality of MT systems. We?ll be using expert-based MQM annotations on English-German, English-Spanish and Japanese?Chinese as the primary gold standard for evaluating metrics. Details are at http://www2.statmt.org/wmt24/metrics-task.html.

We?ll be continuing the challenge sets subtask<http://www2.statmt.org/wmt24/metrics-task.html#_challenge_set_subtask> this year: we invite anyone to submit a new test suite and/or an analysis paper on metric behaviour for specific perturbations/phenomena (you?re welcome to resubmit last year?s challenge set!)

New this year:

*
New language pairs: English-Spanish and Japanese?Chinese
*
Additional 13 typologically diverse African languages in a challenge set
*
We will be using the Codabench platform to improve the metric submission experience

Important dates:
Challenge sets submission deadline: 11th July
Metrics inputs ready to download: 23rd July
Metric submission deadline: 30th July
Metric scores for challenge sets distributed: 6th August
Paper submission deadline to WMT: 20th August

Please register your metric submissions here <https://docs.google.com/forms/d/e/1FAIpQLSenoN9svmqkfJshM8bHd8p3Ofyepdvcg9Mtns0VdEbsFunvEA/viewform?usp=pp_url> and challenge set submissions <https://forms.office.com/e/uhA74RnjMb> here<https://forms.office.com/pages/responsepage.aspx?id=vfGpYaB-aECyMbtKa_y3APGD0hnjQCxCom5Ag3yDFjJUQVZHV1FSUlJRSTBYRU9JV1oyRjNRWk9LRi4u> so we can keep track of participants.

Looking forward to your submissions,
WMT24 Metrics team


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.mit.edu/mailman/private/moses-support/attachments/20240604/7bb6da73/attachment-0001.htm>

------------------------------

Subject: Digest Footer

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
https://mailman.mit.edu/mailman/listinfo/moses-support


------------------------------

End of Moses-support Digest, Vol 207, Issue 2
*********************************************

Tweet