Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Training with multiple text files (Sunayana Gawde)
2. Segmentation Fault on tuning phase (Alex Martinez)
3. Re: Training with multiple text files (Philipp Koehn)
----------------------------------------------------------------------
Message: 1
Date: Wed, 11 Nov 2015 15:48:30 +0530
From: Sunayana Gawde <sunayanagawde17@gmail.com>
Subject: [Moses-support] Training with multiple text files
To: moses-support@mit.edu
Message-ID:
<CANQTV3SZVUvyePuTRo+36QBAt22=ewfFM3hVbUJ+GvRqFCjy8Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hello all,
I have developed a Baseline Machine Translation system as stated on moses
website. Also i developed a MT system for a English-Hindi parallel corpus
available online with which i am getting very low BLEU score i.e.5.31. Now
i have a parallel text in English and Hindi in health n tourism corpus
which contains many text files. How to train the system with multiple text
files? I am only familiar to develop that baseline system. Is there
something else which i need specifically for Hindi?
Please help. Thanks.
--
*Thanks & Regards*
Ms. Sunayana R. Gawde.
DCST, Goa University.
* P**leas**e don't print t**his e-mail unles**s you really need to.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151111/0dd21ca7/attachment-0001.html
------------------------------
Message: 2
Date: Wed, 11 Nov 2015 12:46:26 +0000 (GMT)
From: Alex Martinez <cmxela@me.com>
Subject: [Moses-support] Segmentation Fault on tuning phase
To: moses-support@mit.edu
Message-ID: <fabbf159-1841-4d4e-9d5b-1d205c1cee41@me.com>
Content-Type: text/plain; charset="utf-8"
Hi,
I've just pulled the code and rebuild the MT system and I'm getting a segmentation fault during tuning step using EMS in a experiment that was working well with a version of the code pulled on October 2.
I have pulled and refreshed the code because I was facing some problems getting the word alignments with moses --server and I saw that the code that deals with the server params has been updated recently.
The model is a factored model and the error in the log is:
?Using SCRIPTS_ROOTDIR: /opt/moses/scripts
Asking moses for feature names and values from /mnt/a62/devel/en_es/process/model/moses.bin.ini.4
Executing: /opt/moses/bin/moses -threads all -v 0 -config /mnt/a62/devel/en_es/process/model/moses.bin.ini.4 -show-weights
exec: /opt/moses/bin/moses -threads all -v 0 -config /mnt/a62/devel/en_es/process/model/moses.bin.ini.4 -show-weights
Executing: /opt/moses/bin/moses -threads all -v 0 -config /mnt/a62/devel/en_es/process/model/moses.bin.ini.4 -show-weights > ./features.list 2> /dev/null
MERT starting values and ranges for random generation:
LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
LexicalReordering0 = 0.300 ( 0.00 .. 1.00)
Distortion0 = 0.300 ( 0.00 .. 1.00)
LM0 = 0.500 ( 0.00 .. 1.00)
LM1 = 0.500 ( 0.00 .. 1.00)
LM2 = 0.500 ( 0.00 .. 1.00)
WordPenalty0 = -1.000 ( 0.00 .. 1.00)
PhrasePenalty0 = 0.200 ( 0.00 .. 1.00)
TranslationModel0 = 0.200 ( 0.00 .. 1.00)
TranslationModel0 = 0.200 ( 0.00 .. 1.00)
TranslationModel0 = 0.200 ( 0.00 .. 1.00)
TranslationModel0 = 0.200 ( 0.00 .. 1.00)
TranslationModel1 = 0.200 ( 0.00 .. 1.00)
TranslationModel1 = 0.200 ( 0.00 .. 1.00)
TranslationModel1 = 0.200 ( 0.00 .. 1.00)
TranslationModel1 = 0.200 ( 0.00 .. 1.00)
TranslationModel2 = 0.200 ( 0.00 .. 1.00)
TranslationModel2 = 0.200 ( 0.00 .. 1.00)
TranslationModel2 = 0.200 ( 0.00 .. 1.00)
TranslationModel2 = 0.200 ( 0.00 .. 1.00)
GenerationModel0 = 0.300 ( 0.00 .. 1.00)
GenerationModel0 = 0.000 ( 0.00 .. 1.00)
GenerationModel1 = 0.300 ( 0.00 .. 1.00)
GenerationModel1 = 0.000 ( 0.00 .. 1.00)
featlist: LexicalReordering0=0.300000
featlist: LexicalReordering0=0.300000
featlist: LexicalReordering0=0.300000
featlist: LexicalReordering0=0.300000
featlist: LexicalReordering0=0.300000
featlist: LexicalReordering0=0.300000
featlist: Distortion0=0.300000
featlist: LM0=0.500000
featlist: LM1=0.500000
featlist: LM2=0.500000
featlist: WordPenalty0=-1.000000
featlist: PhrasePenalty0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel1=0.200000
featlist: TranslationModel1=0.200000
featlist: TranslationModel1=0.200000
featlist: TranslationModel1=0.200000
featlist: TranslationModel2=0.200000
featlist: TranslationModel2=0.200000
featlist: TranslationModel2=0.200000
featlist: TranslationModel2=0.200000
featlist: GenerationModel0=0.300000
featlist: GenerationModel0=0.000000
featlist: GenerationModel1=0.300000
featlist: GenerationModel1=0.000000
Saved: ./run1.moses.ini
Normalizing lambdas: 0.300000 0.300000 0.300000 0.300000 0.300000 0.300000 0.300000 0.500000 0.500000 0.500000 -1.000000 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 0.200000 0.300000 0.000000 0.300000 0.000000
DECODER_CFG = -weight-overwrite 'WordPenalty0= -0.128205 PhrasePenalty0= 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 Distortion0= 0.038462 GenerationModel0= 0.038462 0.000000 TranslationModel1= 0.025641 0.025641 0.025641 0.025641 LM1= 0.064103 TranslationModel0= 0.025641 0.025641 0.025641 0.025641 LM0= 0.064103 LM2= 0.064103 TranslationModel2= 0.025641 0.025641 0.025641 0.025641 GenerationModel1= 0.038462 0.000000'
Executing: /opt/moses/bin/moses -threads all -v 0 -config /mnt/a62/devel/en_es/process/model/moses.bin.ini.4 -weight-overwrite 'WordPenalty0= -0.128205 PhrasePenalty0= 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 Distortion0= 0.038462 GenerationModel0= 0.038462 0.000000 TranslationModel1= 0.025641 0.025641 0.025641 0.025641 LM1= 0.064103 TranslationModel0= 0.025641 0.025641 0.025641 0.025641 LM0= 0.064103 LM2= 0.064103 TranslationModel2= 0.025641 0.025641 0.025641 0.025641 GenerationModel1= 0.038462 0.000000' -n-best-list run1.best100.out 100 distinct -input-file /mnt/a62/devel/en_es/data/corpora.tuning.en > run1.out
Executing: /opt/moses/bin/moses -threads all -v 0 -config /mnt/a62/devel/en_es/process/model/moses.bin.ini.4 -weight-overwrite 'WordPenalty0= -0.128205 PhrasePenalty0= 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 Distortion0= 0.038462 GenerationModel0= 0.038462 0.000000 TranslationModel1= 0.025641 0.025641 0.025641 0.025641 LM1= 0.064103 TranslationModel0= 0.025641 0.025641 0.025641 0.025641 LM0= 0.064103 LM2= 0.064103 TranslationModel2= 0.025641 0.025641 0.025641 0.025641 GenerationModel1= 0.038462 0.000000' -n-best-list run1.best100.out 100 distinct -input-file /mnt/a62/devel/en_es/data/corpora.tuning.en > run1.out
Segmentation fault (core dumped)
Exit code: 139
The decoder died. CONFIG WAS -weight-overwrite 'WordPenalty0= -0.128205 PhrasePenalty0= 0.025641 LexicalReordering0= 0.038462 0.038462 0.038462 0.038462 0.038462 0.038462 Distortion0= 0.038462 GenerationModel0= 0.038462 0.000000 TranslationModel1= 0.025641 0.025641 0.025641 0.025641 LM1= 0.064103 TranslationModel0= 0.025641 0.025641 0.025641 0.025641 LM0= 0.064103 LM2= 0.064103 TranslationModel2= 0.025641 0.025641 0.025641 0.025641 GenerationModel1= 0.038462 0.000000'
Thanks in advance for helping...
Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151111/e111e536/attachment-0001.html
------------------------------
Message: 3
Date: Wed, 11 Nov 2015 09:25:56 -0500
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] Training with multiple text files
To: Sunayana Gawde <sunayanagawde17@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDDCvMk-FLKJEzjPU15hxEP2SXVZn5o+Qbr=FPYinb1dEA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
you have to convert your parallel text files yourself into the format that
Moses expects, i.e., two files, one for English, one for Hindi, where
line x in the English file corresponds to line x in the Hindi file.
If this is rather raw data, you may have to run sentence alignment on
your data, using tools such as Hunalign.
http://mokk.bme.hu/en/resources/hunalign/
-phi
On Wed, Nov 11, 2015 at 5:18 AM, Sunayana Gawde <sunayanagawde17@gmail.com>
wrote:
> Hello all,
> I have developed a Baseline Machine Translation system as stated on moses
> website. Also i developed a MT system for a English-Hindi parallel corpus
> available online with which i am getting very low BLEU score i.e.5.31. Now
> i have a parallel text in English and Hindi in health n tourism corpus
> which contains many text files. How to train the system with multiple text
> files? I am only familiar to develop that baseline system. Is there
> something else which i need specifically for Hindi?
> Please help. Thanks.
>
> --
> *Thanks & Regards*
>
> Ms. Sunayana R. Gawde.
>
> DCST, Goa University.
> * P**leas**e don't print t**his e-mail unles**s you really need to.*
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151111/26f63807/attachment.html
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 109, Issue 20
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 109, Issue 20"
Post a Comment