Moses-support Digest, Vol 92, Issue 16

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. OSM - random crashes? (Marcin Junczys-Dowmunt)
2. Re: Moses: Special Characters (Hieu Hoang)
3. Re: OSM - random crashes? (Barry Haddow)
4. Re: OSM - random crashes? (Marcin Junczys-Dowmunt)
5. Re: OSM - random crashes? (Barry Haddow)


----------------------------------------------------------------------

Message: 1
Date: Fri, 06 Jun 2014 12:22:55 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: [Moses-support] OSM - random crashes?
To: moses-support <moses-support@mit.edu>
Message-ID: <5391967F.2070603@amu.edu.pl>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Hi,
is anyone using the OSM feature in Moses and could confirm that it
crashes Moses randomly from time to time? Seems to happen for me ca.
once per 10-20 runs (annoying during tuning). If someone has the same
problem when using OSM I might be inclined to track it down.
Best,
Marcin


------------------------------

Message: 2
Date: Fri, 6 Jun 2014 11:45:18 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Moses: Special Characters
To: Andreas Dolinsek <andreas.dolinsek@gmail.com>, moses-support
<moses-support@mit.edu>
Message-ID:
<CAEKMkbiTBCE2AWf79AYBT9P2VpCRqyWkwDvvEHsQt3BjrN5PiQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

there shouldn't be any problems with diacritics.

The decoder itself is encoding-neutral, however, all the scripts assumes
UTF8 so it's best to encode your training data and input as utf8.

ps. please send questions to the moses mailing list rather than to people
directly so everyone can get to answer


On 6 June 2014 11:41, Andreas Dolinsek <andreas.dolinsek@gmail.com> wrote:

> Thank you for your fast response. Escaping these kind of chars seems
> logically. What I meant especially is escaping german diacritics.
>
> e.g. I want to translate the german sentence: ?Das Wetter ist sch?n?
>
> echo "Das Wetter ist sch?n" | moses -f <config-file> -v 0 -threads all
>
> Die Terminal does not detect ???. Which encoding does moses need for
> non-ASCII chars.
>
>
> Thanks!
> --
> Andreas Dolinsek
> http://www.sonicomobile.com
>
> Am 06. Juni 2014 bei 12:07:24, Hieu Hoang (hieu.hoang@ed.ac.uk) schrieb:
>
> you have to escape certain characters. This is provided by the script
> scripts/tokenizer/escape-special-chars.perl
>
> You should also remove non-printing characters with the script
> scripts/tokenizer/remove-non-printing-char.perl
>
> of course, tokenize your data using the same method you used to tokenize
> the training data
>
>
>
> On 6 June 2014 10:48, Andreas Dolinsek <andreas.dolinsek@gmail.com> wrote:
>
>> Dear Hieu Hoang,
>>
>> my name is Andreas and I work for a company called Sonico Mobile, which
>> ships translation apps (iTranslate) to millions of people with over 4
>> million translation requests per day.
>> Check out our website if you are interested http://www.itranslateapp.com
>> .
>>
>> I installed Moses on a highly scaleable platform and wrote an API for it
>> (for testing), because we would like to try it out for some use cases. I
>> trained it with the europarl parallel corpus.
>>
>> I have a really basic problem now, I don?t know how to translate special
>> chars. I call via my python tornado server:
>>
>> echo <Text> | moses -f <config-file> -v 0 -threads all
>>
>> Non-ASCII chars result in an error. How is it possible to feed it with
>> special chars?
>>
>> Thank you very much!
>>
>> PS: If everything works out, of course we will provide an API access :-)
>> --
>> Andreas Dolinsek
>> http://www.sonicomobile.com
>>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140606/7f74db0d/attachment-0001.htm

------------------------------

Message: 3
Date: Fri, 06 Jun 2014 12:17:47 +0100
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] OSM - random crashes?
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>, moses-support
<moses-support@mit.edu>
Message-ID: <vhao6g32fnsh9i6w9q3fu9cm.1402053467820@email.android.com>
Content-Type: text/plain; charset=utf-8

Hi Marvin

Yes, I confirm. I have checked in a fix this week but have not tested it.

Cheers - Barry

Marcin Junczys-Dowmunt <junczys@amu.edu.pl> wrote:

>Hi,
>is anyone using the OSM feature in Moses and could confirm that it
>crashes Moses randomly from time to time? Seems to happen for me ca.
>once per 10-20 runs (annoying during tuning). If someone has the same
>problem when using OSM I might be inclined to track it down.
>Best,
>Marcin
>_______________________________________________
>Moses-support mailing list
>Moses-support@mit.edu
>http://mailman.mit.edu/mailman/listinfo/moses-support
>

--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




------------------------------

Message: 4
Date: Fri, 06 Jun 2014 13:45:01 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] OSM - random crashes?
To: Barry Haddow <bhaddow@staffmail.ed.ac.uk>, moses-support
<moses-support@mit.edu>
Message-ID: <5391A9BD.5080005@amu.edu.pl>
Content-Type: text/plain; charset=UTF-8; format=flowed

Oh great. I will merge it into my branch and report back. Do you
remember roughly which files you corrected? Something In the FF/OSM
directory itself?

I am running around 20-40 iterations daily, so I will be able to check.
Best,
Marcin

W dniu 06.06.2014 13:17, Barry Haddow pisze:
> Hi Marvin
>
> Yes, I confirm. I have checked in a fix this week but have not tested it.
>
> Cheers - Barry
>
> Marcin Junczys-Dowmunt <junczys@amu.edu.pl> wrote:
>
>> Hi,
>> is anyone using the OSM feature in Moses and could confirm that it
>> crashes Moses randomly from time to time? Seems to happen for me ca.
>> once per 10-20 runs (annoying during tuning). If someone has the same
>> problem when using OSM I might be inclined to track it down.
>> Best,
>> Marcin
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>



------------------------------

Message: 5
Date: Fri, 06 Jun 2014 13:41:04 +0100
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] OSM - random crashes?
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>, moses-support
<moses-support@mit.edu>
Message-ID: <91eb3fi2p21ufssosg4dc113.1402058318933@email.android.com>
Content-Type: text/plain; charset=utf-8

Yes, in the OSM directory. The commit comment was "range check" or something like that. I can't access the repo right now.

Marcin Junczys-Dowmunt <junczys@amu.edu.pl> wrote:

>Oh great. I will merge it into my branch and report back. Do you
>remember roughly which files you corrected? Something In the FF/OSM
>directory itself?
>
>I am running around 20-40 iterations daily, so I will be able to check.
>Best,
>Marcin
>
>W dniu 06.06.2014 13:17, Barry Haddow pisze:
>> Hi Marvin
>>
>> Yes, I confirm. I have checked in a fix this week but have not tested it.
>>
>> Cheers - Barry
>>
>> Marcin Junczys-Dowmunt <junczys@amu.edu.pl> wrote:
>>
>>> Hi,
>>> is anyone using the OSM feature in Moses and could confirm that it
>>> crashes Moses randomly from time to time? Seems to happen for me ca.
>>> once per 10-20 runs (annoying during tuning). If someone has the same
>>> problem when using OSM I might be inclined to track it down.
>>> Best,
>>> Marcin
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>
>

--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 92, Issue 16
*********************************************

Tweet