Moses-support Digest, Vol 100, Issue 61

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: TOKENIZER.PERL (Raj Dabre)
2. Re: Tuning with mert-moses.perl error (Matthias Huck)


----------------------------------------------------------------------

Message: 1
Date: Wed, 18 Feb 2015 01:06:03 +0900
From: Raj Dabre <prajdabre@gmail.com>
Subject: Re: [Moses-support] TOKENIZER.PERL
To: doc <raymond.doctor@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAB3gfjC+6ojPRYNMm4UXV0OhXEJkr+d-OYWJ72=Ap26q9UTApw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hello there,
If it is Indian Languages then this is what you need:
http://www.cfilt.iitb.ac.in/static/download.html
Regards.

On Tue, Feb 17, 2015 at 10:24 AM, doc <raymond.doctor@gmail.com> wrote:

> Hello,
> I am using the tokenizer.perl script which I found on
>
> https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizer/tokenizer.perlI
> have tried to make it work for Indic languages which use the same punctuation
> markers with the exception of the full-stop which is a
> ? U+0964 DEVANAGARI DANDA
> My main issue is that Hindi and other languages using the character also
> use the full-stop as an abbreviation marker. How do I manage to
> keep both characters as tokenising elements? I would really appreciate if
> someone could take some time off and propose modifications to the perl
> script to accommodate also the Devanagari danda as well as the full-stop. I
> work in C and hence the issue.
> I am appending the a small sample of Hindi <raw.txt> for testing
> Many thanks for your help
> Best regards,
>
> Raymond
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Raj Dabre.
Research Student,
Graduate School of Informatics,
Kyoto University.
CSE MTech, IITB., 2011-2014
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150218/5c4c015c/attachment-0001.htm

------------------------------

Message: 2
Date: Tue, 17 Feb 2015 16:19:09 +0000
From: Matthias Huck <mhuck@inf.ed.ac.uk>
Subject: Re: [Moses-support] Tuning with mert-moses.perl error
To: "Hacksawhawk ." <hayo.ce@gmail.com>
Cc: moses-support@mit.edu
Message-ID: <1424189949.2192.381.camel@portedgar>
Content-Type: text/plain; charset="UTF-8"

Hi Hayo,

Can you please do two things:

1.) Send me the file filtered/moses.ini so that I can have a look at the
feature functions and scaling factors in there.

2.) Tell me the Git commit ID of the Moses version you're working with.
A bug was put into master with commit 70e8eb5. It's been fixed a couple
of days later (commit 0de206f). If you've checked out Moses from GitHub
with the bug, you need to update to the most recent code base and the
error most likely will be gone.

Cheers,
Matthias


On Tue, 2015-02-17 at 17:01 +0100, Hacksawhawk . wrote:
> Hi,
>
>
> While trying to tune the translation system I created, I ran into the
> following erorr:
>
> The following weights have no feature function. Maybe incorrectly
> spelt weights: ,Exit code: 1
> The decoder died. CONFIG WAS -weight-overwrite 'PhrasePenalty0=
> 0.043478 WordPenalty0= -0.217391 TranslationModel0= 0.043478 0.043478
> 0.043478 0.043478 Distortion0= 0.065217 LM0= 0.108696
> LexicalReordering0= 0.065217 0.065217 0.065217 0.065217 0.065217
> 0.065217'
>
>
> It seems that mert-moses.pl is rearranging the weight features and
> then trying to overwrite the weight features in the moses.ini file but
> in the wrong order, is this the cause of the error?
>
> I have also attached the mert.out file, hopefully this will provide
> more information.
>
>
> thanks in advance,
>
> Hayo
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support



--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 100, Issue 61
**********************************************

Tweet