Home » Uncategories » Moses-support Digest, Vol 107, Issue 54

Moses-support Digest, Vol 107, Issue 54

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Problem in Accuracy (Philipp Koehn)
2. Re: experiment.perl, processPhraseTableMin and threads option
(Tomasz Gawryl)
3. Re: is there a way to remove a bad entry in the phrase table
? (Vincent Nguyen)


----------------------------------------------------------------------

Message: 1
Date: Wed, 23 Sep 2015 15:46:56 +0200
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] Problem in Accuracy
To: fatma elzahraa Eltaher <fatmaeltaher@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDDO5i+eiwPnGasUHX3NZHdStOhYNp98sy=fFwX-vkXv8Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

you can test the language model in isolation.

Typically, we try to minimize perplexity, so you can run
the language model on a test set and check the perplexity
score. Running the language model on a test set also
helps you with debugging, since it prints out which
n-grams are found in the language model.

-phi

On Thu, Sep 10, 2015 at 5:33 PM, fatma elzahraa Eltaher <
fatmaeltaher@gmail.com> wrote:

> Dear All,
>
> I try to test LM but the accuracy was very law.what I must do to be ensure
> that every thing is oky.
>
>
> thank you,
>
>
>
> Fatma El-Zahraa El -Taher
>
> Teaching Assistant at Computer & System department
>
> Faculty of Engineering, Azhar University
>
> Email : fatmaeltaher@gmail.com
> mobile: +201141600434
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150923/616915f7/attachment-0001.html

------------------------------

Message: 2
Date: Wed, 23 Sep 2015 16:03:35 +0200
From: "Tomasz Gawryl" <tomasz.gawryl@skrivanek.pl>
Subject: Re: [Moses-support] experiment.perl, processPhraseTableMin
and threads option
To: <moses-support@mit.edu>
Message-ID: <004901d0f608$a3c348c0$eb49da40$@gawryl@skrivanek.pl>
Content-Type: text/plain; charset="utf-8"

Hi Philipp,



Shame on me, you are right.

Thanx for help!



Regards,

TG



From: phkoehn@gmail.com [mailto:phkoehn@gmail.com] On Behalf Of Philipp Koehn
Sent: Wednesday, September 23, 2015 3:30 PM
To: Tomasz Gawryl
Cc: moses-support@mit.edu
Subject: Re: [Moses-support] experiment.perl, processPhraseTableMin and threads option



Hi,



I ran the command as you provided it, and it worked.



The "-threads all" functionality was added in August - did you

compile the latest version of the code?



-phi



On Tue, Sep 22, 2015 at 12:16 PM, Tomasz Gawryl <tomasz.gawryl@skrivanek.pl> wrote:

Hi,

I'm trying to add this line to config:

ttable-binarizer = "$moses-bin-dir/processPhraseTableMin -threads all"

This option is supported by processPhraseTableMin
command(https://github.com/moses-smt/mosesdecoder/blob/master/misc/processPh
raseTableMin.cpp line 24).

/home/moses/src/mosesdecoder/scripts/training/binarize-model.perl
/home/moses/working/experiments/model/moses.ini.1 /home/mose
s/working/experiments/model/moses.bin.ini.3 -Binarizer
/home/moses/src/mosesdecoder/bin/processPhraseTableMin -threads all

But it produces error "Unknown option: threads" in file
TRAINING_binarize-config.3.STDERR (and stops training).

I removed this option but it seems that such case it uses only one thread:

moses 1470 113 10.8 4255200 3448392 pts/13 Sl 11:05 66:03
/home/moses/src/mosesdecoder/bin/processPhraseTableMin -in
/home/moses/working/experiments/model/moses.bin.ini.5.tables/phrase-table.0-
0.1.1.gz.sorted -out
/home/moses/working/experiments/model/moses.bin.ini.5.tables/phrase-table.0-
0.1.1 -nscores 4 -threads 1

I know that my server is able to run around 16 threads (and indeed does
during former steps).

What can I do to improve this step to use more threads?

Regards,
TG



_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150923/3c579c76/attachment-0001.html

------------------------------

Message: 3
Date: Wed, 23 Sep 2015 16:50:38 +0200
From: Vincent Nguyen <vnguyen@neuf.fr>
Subject: Re: [Moses-support] is there a way to remove a bad entry in
the phrase table ?
To: moses-support@mit.edu
Message-ID: <5602BC3E.1090801@neuf.fr>
Content-Type: text/plain; charset="windows-1252"

I agree and would like to.
But this is tricky, look at the first 30 lines of my phrase table below.

and this happens a lot in the first line of tables where there are &apos
or weird codes, EN/FR pairs do not match.




! ! ! ! ||| ! ! ! ! ||| 0.103413 0.132185 0.103413 0.401758 ||| 0-0 1-1
2-2 3-3 ||| 1 1 1 ||| |||
! ! ! ) ||| ! ! ! ) ||| 0.339323 0.167884 0.508985 0.4246 ||| 0-0 1-0
2-0 2-1 2-2 3-3 ||| 3 2 2 ||| |||
! ! ! ||| ! ! ! ||| 0.501834 0.219223 0.716905 0.50463 ||| 0-0 1-1 2-2
||| 10 7 6 ||| |||
! ! ! ||| budget ! ! ! ||| 0.0517067 0.219223 0.0147733 4.50635e-05 |||
0-1 1-2 2-3 ||| 2 7 1 ||| |||
! ! ) , ||| ! ! ) - , ||| 0.103413 0.111989 0.103413 0.00192967 ||| 0-0
1-1 2-2 3-3 3-4 ||| 1 1 1 ||| |||
! ! ) ||| ! ! ) ||| 0.103413 0.278429 0.103413 0.533321 ||| 0-0 1-1 2-2
||| 1 1 1 ||| |||
! ! ||| ! ! ||| 0.625 0.363573 0.769231 0.633844 ||| 0-0 1-1 ||| 16 13
10 ||| |||
! ! ||| . ||| 4.65922e-08 6.71089e-07 0.00795487 0.140779 ||| 0-0 1-0
||| 2.21954e+06 13 1 ||| |||
! ! ||| budget ! ! ||| 0.0517067 0.363573 0.00795487 5.66022e-05 ||| 0-1
1-2 ||| 2 13 1 ||| |||
! ! ||| n?cessaire ! ! ||| 0.103413 0.363573 0.00795487 0.000130572 |||
0-1 1-2 ||| 1 13 1 ||| |||
! &#91; never again ! ||| ! ||| 6.51628e-06 5.42074e-13 0.103413
0.796143 ||| 0-0 4-0 ||| 15870 1 1 ||| |||
! &#93; this is ||| tel est ||| 7.38667e-05 9.16191e-11 0.103413
0.00147917 ||| 2-0 3-1 ||| 1400 1 1 ||| |||
! &#93; this ||| tel ||| 1.09594e-05 1.44188e-10 0.103413 0.0035893 |||
2-0 ||| 9436 1 1 ||| |||
! &#93; ||| ! &#93; ||| 0.103413 0.352335 0.103413 0.472387 ||| 0-0 1-1
||| 1 1 1 ||| |||
! &amp; quot ; ||| ! &quot; . et ||| 0.0517067 2.36396e-12 0.0517067
1.88268e-05 ||| 0-0 1-1 2-1 3-3 ||| 2 2 1 ||| |||
! &amp; quot ; ||| ! &quot; ||| 0.000222394 1.44515e-11 0.0517067
0.518419 ||| 0-0 2-1 ||| 465 2 1 ||| |||
! &amp; quot ||| ! &quot; . ||| 0.000662906 8.30626e-09 0.0344711
0.00232791 ||| 0-0 1-1 2-1 ||| 156 3 1 ||| |||
! &amp; quot ||| ! &quot; ||| 0.00218918 8.30626e-09 0.339323 0.518419
||| 0-0 2-1 ||| 465 3 2 ||| |||
! &amp; ||| ! ||| 6.51628e-06 7.21755e-05 0.103413 0.796143 ||| 0-0 |||
15870 1 1 ||| |||
! &apos; &#93; , addressed ||| ! &quot; adress? ||| 0.103413 3.70838e-07
0.103413 0.00596848 ||| 0-0 1-1 2-1 4-2 ||| 1 1 1 ||| |||
! &apos; &#93; , ||| ! &quot; ||| 0.000222394 2.49698e-06 0.103413
0.215573 ||| 0-0 1-1 2-1 ||| 465 1 1 ||| |||
! &apos; &#93; ||| ! &quot; ||| 0.000222394 3.57128e-05 0.103413
0.215573 ||| 0-0 1-1 2-1 ||| 465 1 1 ||| |||
! &apos; &apos; Alstom shares ||| l&apos; on constate un
dysfonctionnement ||| 0.0344711 5.62605e-16 0.103413 1.03361e-14 ||| 1-0
2-0 1-1 3-4 4-4 ||| 3 1 1 ||| |||
! &apos; &apos; ||| l&apos; on constate un ||| 0.0147733 1.56906e-11
0.0129267 2.2766e-12 ||| 1-0 2-0 1-1 ||| 7 8 1 ||| |||
! &apos; &apos; ||| l&apos; on constate ||| 0.000984889 1.56906e-11
0.0129267 2.36929e-10 ||| 1-0 2-0 1-1 ||| 105 8 1 ||| |||
! &apos; &apos; ||| l&apos; on ||| 6.76656e-06 1.56906e-11 0.0129267
6.18613e-06 ||| 1-0 2-0 1-1 ||| 15283 8 1 ||| |||
! &apos; &apos; ||| ou que l&apos; on constate ||| 0.0344711 1.56906e-11
0.0129267 4.69534e-15 ||| 1-2 2-2 1-3 ||| 3 8 1 ||| |||
! &apos; &apos; ||| ou que l&apos; on ||| 0.00304157 1.56906e-11
0.0129267 1.22594e-10 ||| 1-2 2-2 1-3 ||| 34 8 1 ||| |||
! &apos; &apos; ||| que l&apos; on constate un ||| 0.0344711 1.56906e-11
0.0129267 4.56092e-14 ||| 1-1 2-1 1-2 ||| 3 8 1 ||| |||
! &apos; &apos; ||| que l&apos; on constate ||| 0.00323167 1.56906e-11
0.0129267 4.74661e-12 ||| 1-1 2-1 1-2 ||| 32 8 1 ||| |||



Le 23/09/2015 15:12, Tom Hoar a ?crit :
> Vincent,
>
> If you suspect bad entries, isn't it better to address the root of the
> problem and prepare your training corpus better?
>
>
> On 9/23/2015 6:46 PM, moses-support-request@mit.edu wrote:
>> Date: Tue, 22 Sep 2015 20:24:02 +0200
>> From: Philipp Koehn<phi@jhu.edu>
>> Subject: Re: [Moses-support] is there a way to remove a bad entry in
>> the phrase table ?
>> To: Vincent Nguyen<vnguyen@neuf.fr>
>> Cc: moses-support<moses-support@mit.edu>
>>
>> Hi,
>>
>> you can remove it manually (just edit the text file), there will be no
>> negative consequences.
>>
>> However, it is not a realistic strategy to try to remove by hand every
>> offending phrase table entry.
>>
>> -phi
>>
>> On Tue, Sep 22, 2015 at 4:05 PM, Vincent Nguyen<vnguyen@neuf.fr> wrote:
>>
>>> >Hi,
>>> >
>>> >I was wondering if after an analysis of the BLEU-Annotation file we
>>> >realize that there must be a bad entry in the phrase table,
>>> >we could remove it manually or in some other ways ?
>>> >
>>> >Gracias.
>>> >V.
>>> >_______________________________________________
>>> >Moses-support mailing list
>>> >Moses-support@mit.edu
>>> >http://mailman.mit.edu/mailman/listinfo/moses-support
>>> >
>
> --
> Best regards,
>
> Tom Hoar
> Chief Executive Officer
> /*Precision Translation Tools Pte Ltd*/
> Singapore/Thailand
> Web: www.precisiontranslationtools.com
> <http://www.precisiontranslationtools.com>
> Thailand Mobile: +66 87 345-1875
> Skype: tahoar
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150923/fea2c175/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 107, Issue 54
**********************************************

0 Response to "Moses-support Digest, Vol 107, Issue 54"

Post a Comment

Subscribe to: Post Comments (Atom)