Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. merging two translation models (??????? ?????????)
2. Re: Regarding Transliteration (Prasanth K)
3. Re: regarding xml (Christian Buck)
4. Re: merging two translation models (Rico Sennrich)
5. Re: merging two translation models (Philipp Koehn)
6. Warning: Too many arguments while IRSTLM language model
Training (renubalyan)
----------------------------------------------------------------------
Message: 1
Date: Wed, 04 Dec 2013 15:36:26 +0400
From: ??????? ????????? <verbalab@yandex.ru>
Subject: [Moses-support] merging two translation models
To: moses-support@MIT.EDU
Message-ID: <192911386156986@web2g.yandex.ru>
Content-Type: text/plain
Hello, everyone!
I have the following question: I have trained a huge translation model trained on 1m sentences of news texts. Also I have several minor translation models trained on texts of different domains. Are there any tools in Moses that can enable merging translation models? Is it possible to merge models when the decoder is running?
Thanks!
Kind regards, Alex Kalinin.
------------------------------
Message: 2
Date: Wed, 4 Dec 2013 13:41:44 +0100
From: Prasanth K <prasanthk.ms09@gmail.com>
Subject: Re: [Moses-support] Regarding Transliteration
To: Thomas Meyer <Thomas.Meyer@idiap.ch>
Cc: moses-support <moses-support@mit.edu>, pranjal4456@gmail.com
Message-ID:
<CA+n+9-h=xq6aUYvXEzh=vWR48gk7_dUbRjYa6FW6sqX8x4RAXQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
@Thomas, I dont think his intention was to search for a given utf8 string;
it was more like to identify strings in a different language.
@Pranjal, when you say unicode words, understand that everything (even the
english words) are encoded in utf8. Its more like you want to pick the
Bengali (read as foreign language) fragments from your sentence so that
they are skipped when given to the decoder. I recall doing something like
this by defining a range of valid characters (which is the utf8 code points
for characters in bengali in this case) and writing a filter to mark such
characters. May be not a cool solution, but one that will work for ILs
given that they have different scripts and code-values in Unicode.
- Regards,
Prasanth
On Wed, Dec 4, 2013 at 11:12 AM, Thomas Meyer <Thomas.Meyer@idiap.ch> wrote:
> Hi,
>
> echo "???? is a vast country" | perl -pe '/(????)/; print $1."\n"'
>
> if you want to replace it with something:
>
> echo "???? is a vast country" | perl -pe 's/????/something/g'
>
> But this is normally not the list to ask perl questions to...
>
> Best,
> Thomas
>
>
> On 04/12/13 10:55, Pranjal Das wrote:
>
> can anyone help with a perl script to extract unicode words from a
> sentence. For eg. i want to extract the word ???? from the sentence "????
> is a vast country"..
> * Pranjal Das*
> Department of Information Technology,
> Institute of Science and Technology,
> Gauhati University,Guwahati,Assam
> Phone- +91-8399879454
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
--
"Theories have four stages of acceptance. i) this is worthless nonsense;
ii) this is an interesting, but perverse, point of view, iii) this is true,
but quite unimportant; iv) I always said so."
--- J.B.S. Haldane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131204/d6e398cc/attachment-0001.htm
------------------------------
Message: 3
Date: Wed, 04 Dec 2013 12:59:58 +0000
From: Christian Buck <cbuck@lantis.de>
Subject: Re: [Moses-support] regarding xml
To: moses-support@mit.edu
Message-ID: <529F274E.7070505@lantis.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hi,
To suggest transliterated words to moses where the source was not seen
in the training data (otherwise transliterating those words in the
training data would probably be the better way) you can use the XML
markup as described here:
http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc7
Please note that the language model will not think highly of unseen
words and performance might deteriorate.
cheers,
Christian
On 03/12/13 07:21, Kalyani Baruah wrote:
> hi.. everyone.. i want to give my translated output to my translation
> system so that that i get the correct sentence as output.. bt how to
> give mose translation system my translated output in an xml fomate..
>
>
>
> Regards,
>
>
> *Kalyanee Kanchan Baruah*
> Department of Information Technology,
> Institute of Science and Technology,
> Gauhati University,Guwahati,India
> Phone- +91-9706242124
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
------------------------------
Message: 4
Date: Wed, 4 Dec 2013 14:27:57 +0000 (UTC)
From: Rico Sennrich <rico.sennrich@gmx.ch>
Subject: Re: [Moses-support] merging two translation models
To: moses-support@mit.edu
Message-ID: <loom.20131204T152331-842@post.gmane.org>
Content-Type: text/plain; charset=utf-8
??????? ????????? <verbalab@...> writes:
>
> Hello, everyone!
>
> I have the following question: I have trained a huge translation model
trained on 1m sentences of news
> texts. Also I have several minor translation models trained on texts of
different domains. Are there any
> tools in Moses that can enable merging translation models? Is it possible
to merge models when the decoder
> is running?
>
> Thanks!
>
> Kind regards, Alex Kalinin.
>
Hi Alex,
there are several scripts that allow you to merge multiple translation
models: http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc52
if you're interested in merging models at decoding time, you can do a
log-linear combination of models (
http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc18 ), or a
linear-interpolation / count-based merge (
http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc55 ).
All methods allow you to weight the models in order to prioritize in-domain
models. Check the documentation/literature if you're interested in more details.
best wishes,
Rico
------------------------------
Message: 5
Date: Wed, 4 Dec 2013 15:36:29 +0100
From: Philipp Koehn <pkoehn@inf.ed.ac.uk>
Subject: Re: [Moses-support] merging two translation models
To: ??????? ????????? <verbalab@yandex.ru>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDAZER3sO=Tb__oPqSb0i41qcmAMbmntXxEs5piWtYuVqQ@mail.gmail.com>
Content-Type: text/plain; charset=KOI8-R
Hi,
it is possible to specify multiple translation tables
and train different weights for them.
Alternatively, you can merge them into a single
table. See some information about this here:
http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc52
The models cannot be updated while the decoder
is running.
-phi
On Wed, Dec 4, 2013 at 12:36 PM, ??????? ????????? <verbalab@yandex.ru> wrote:
> Hello, everyone!
>
> I have the following question: I have trained a huge translation model trained on 1m sentences of news texts. Also I have several minor translation models trained on texts of different domains. Are there any tools in Moses that can enable merging translation models? Is it possible to merge models when the decoder is running?
>
> Thanks!
>
> Kind regards, Alex Kalinin.
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
------------------------------
Message: 6
Date: Wed, 4 Dec 2013 21:28:03 +0530 (IST)
From: renubalyan <renubalyan@cdac.in>
Subject: [Moses-support] Warning: Too many arguments while IRSTLM
language model Training
To: moses-support@mit.edu
Message-ID:
<1875219553.7819.1386172683401.JavaMail.open-xchange@webmail.cdac.in>
Content-Type: text/plain; charset="utf-8"
Hi,
I am building the baseline system based on Moses manual instructions.
I have installed Moses, GIZA++ and IRSTLM as mentioned in the manual.
The corpus preparation (tokenization, ...cleaning) steps also goes well.
However when I move to Language Model Training: I have some problems
I am following these steps:
1. mkdir ~/lm
2. cd ~/lm
3. /home/renu/Desktop/irstlm/bin/add-start-end.sh <
/home/renu/Desktop/corpus/news-commentary-v8.fr-en.true.en>
news-commentary-v8.fr-en.sb.en
4. export IRSTLM=/home/renu/Desktop/irstlm;
/home/renu/Desktop/irstlm/bin/build-lm.sh -i news-commentary-v8.fr-en.sb.en -t
./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en
5. /home/renu/Desktop/irstlm/bin/compile-lm --text yes
news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
Steps 1-4 work well but step 5 gives me -------(Warning:Too many parameters)
I have searched the web for any possible solution but could not find any.
I am not able to move ahead, kindly help.
Thanks
Renu
-------------------------------------------------------------------------------------------------------------------------------
This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.
-------------------------------------------------------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131204/c888afa1/attachment.htm
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 86, Issue 10
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 86, Issue 10"
Post a Comment