Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Regarding an error on estimating language model (Hieu Hoang)
2. Re: is there a way to remove a bad entry in the phrase table
? (Tom Hoar)
3. Re: experiment.perl, processPhraseTableMin and threads option
(Philipp Koehn)
4. Re: Kendall's tau metric (Philipp Koehn)
----------------------------------------------------------------------
Message: 1
Date: Wed, 23 Sep 2015 12:48:51 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Regarding an error on estimating language
model
To: Shubham Tripathi <stripathi1770@gmail.com>, Moses-support@mit.edu
Message-ID: <560291A3.303@gmail.com>
Content-Type: text/plain; charset="windows-1252"
you can try using the KENLM estimation tool
http://www.statmt.org/moses/?n=FactoredTraining.BuildingLanguageModel#ntoc20
On 21/09/2015 13:38, Shubham Tripathi wrote:
> I have been following your tutorials on building a language model and
> I am facing difficulties in it. The web page I am referring is
> http://www.statmt.org/moses/?n=Moses.Baseline
>
> On building the final model, the following error occurs -
>
> /*Cleaning temporary directory /tmp*/
> /*
> */
> /*Warning: some temporary files could not be removed*/
> /*
> */
> /*Extracting dictionary from training corpus*/
> /*
> */
> /*Splitting dictionary into 3 lists*/
> /*
> */
> /*Extracting n-gram statistics for each word list*/
> /*
> */
> /*Important: dictionary must be ordered according to order of
> appearance of words in data*/
> /*used to generate n-gram blocks, so that sub language model blocks
> results ordered too*/
> /*dict.**/
> /*
> */
> /*$bin/ngt -i="$inpfile" -n=$order -gooout=y -o="$gzip -c >
> $tmpdir/ngram.${sdict}.gz" -fd="$tmpdir/$sdict" $dictionary
> -iknstat="$tmpdir/ikn.stat.$sdict" >> $logfile 2>&1*/
> /*
> */
> /*Estimating language models for each word list*/
> /*
> */
> /*ls: cannot access /tmp/dict.*: No such file or directory*/
> /*
> */
> /*Merging language models into BanglaFinal.txt*/
> /*
> */
> /*Cleaning temporary directory /tmp*/
>
> The output file is not made in any of the directories. I am unable to
> understand /cannot access /tmp/dict.*: No such file or directory
> /error. Also, on checking the directory 'tmp', I find no sub directory
> named as 'dict'.
>
> Regards,
> *Shubham Tripathi*
> Pre- Final Year, Electrical Engineering Department
> National Institute of Technology, Jaipur, India - 302017
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
--
Hieu Hoang
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150923/7b1921d3/attachment-0001.html
------------------------------
Message: 2
Date: Wed, 23 Sep 2015 20:12:46 +0700
From: Tom Hoar <tahoar@precisiontranslationtools.com>
Subject: Re: [Moses-support] is there a way to remove a bad entry in
the phrase table ?
To: moses-support@mit.edu
Message-ID: <5602A54E.1030001@precisiontranslationtools.com>
Content-Type: text/plain; charset="windows-1252"
Vincent,
If you suspect bad entries, isn't it better to address the root of the
problem and prepare your training corpus better?
On 9/23/2015 6:46 PM, moses-support-request@mit.edu wrote:
> Date: Tue, 22 Sep 2015 20:24:02 +0200
> From: Philipp Koehn<phi@jhu.edu>
> Subject: Re: [Moses-support] is there a way to remove a bad entry in
> the phrase table ?
> To: Vincent Nguyen<vnguyen@neuf.fr>
> Cc: moses-support<moses-support@mit.edu>
>
> Hi,
>
> you can remove it manually (just edit the text file), there will be no
> negative consequences.
>
> However, it is not a realistic strategy to try to remove by hand every
> offending phrase table entry.
>
> -phi
>
> On Tue, Sep 22, 2015 at 4:05 PM, Vincent Nguyen<vnguyen@neuf.fr> wrote:
>
>> >Hi,
>> >
>> >I was wondering if after an analysis of the BLEU-Annotation file we
>> >realize that there must be a bad entry in the phrase table,
>> >we could remove it manually or in some other ways ?
>> >
>> >Gracias.
>> >V.
>> >_______________________________________________
>> >Moses-support mailing list
>> >Moses-support@mit.edu
>> >http://mailman.mit.edu/mailman/listinfo/moses-support
>> >
--
Best regards,
Tom Hoar
Chief Executive Officer
/*Precision Translation Tools Pte Ltd*/
Singapore/Thailand
Web: www.precisiontranslationtools.com
<http://www.precisiontranslationtools.com>
Thailand Mobile: +66 87 345-1875
Skype: tahoar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150923/4f6a2f0f/attachment-0001.html
------------------------------
Message: 3
Date: Wed, 23 Sep 2015 15:30:18 +0200
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] experiment.perl, processPhraseTableMin
and threads option
To: Tomasz Gawryl <tomasz.gawryl@skrivanek.pl>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDCoOMK2DpaZZG6SP0JOjKYVxTtUq+svLuj_qX47djZ=zg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
I ran the command as you provided it, and it worked.
The "-threads all" functionality was added in August - did you
compile the latest version of the code?
-phi
On Tue, Sep 22, 2015 at 12:16 PM, Tomasz Gawryl <tomasz.gawryl@skrivanek.pl>
wrote:
> Hi,
>
> I'm trying to add this line to config:
>
> ttable-binarizer = "$moses-bin-dir/processPhraseTableMin -threads all"
>
> This option is supported by processPhraseTableMin
> command(
> https://github.com/moses-smt/mosesdecoder/blob/master/misc/processPh
> raseTableMin.cpp line 24).
>
> /home/moses/src/mosesdecoder/scripts/training/binarize-model.perl
> /home/moses/working/experiments/model/moses.ini.1 /home/mose
> s/working/experiments/model/moses.bin.ini.3 -Binarizer
> /home/moses/src/mosesdecoder/bin/processPhraseTableMin -threads all
>
> But it produces error "Unknown option: threads" in file
> TRAINING_binarize-config.3.STDERR (and stops training).
>
> I removed this option but it seems that such case it uses only one thread:
>
> moses 1470 113 10.8 4255200 3448392 pts/13 Sl 11:05 66:03
> /home/moses/src/mosesdecoder/bin/processPhraseTableMin -in
>
> /home/moses/working/experiments/model/moses.bin.ini.5.tables/phrase-table.0-
> 0.1.1.gz.sorted -out
>
> /home/moses/working/experiments/model/moses.bin.ini.5.tables/phrase-table.0-
> 0.1.1 -nscores 4 -threads 1
>
> I know that my server is able to run around 16 threads (and indeed does
> during former steps).
>
> What can I do to improve this step to use more threads?
>
> Regards,
> TG
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150923/236bd22d/attachment-0001.html
------------------------------
Message: 4
Date: Wed, 23 Sep 2015 15:43:40 +0200
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] Kendall's tau metric
To: Arefeh Kazemi Najafabadi <arefeh.kazeminajafabadi@dcu.ie>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDBXXjSdcsBhNz-xO3EGQnYBjWo64f0c8nupRcoJocgZhQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
you can tell mert the scorer type with "--sctype":
http://www.statmt.org/moses/?n=FactoredTraining.Tuning
The KENDALL scorer is likely doing what you want.
You can specify a combination of scorers
--sctype KENDALL,BLEU --scconfig weights:0.6+0.4
-phi
2015-09-19 10:11 GMT+02:00 Arefeh Kazemi Najafabadi <
arefeh.kazeminajafabadi@dcu.ie>:
> Hi everyone
>
> I'm testing a reordering model for hierarchical moses. I want to
> interpolate bleu with kendall's tau permutation distance and use it as my
> tuning metric. can anyone help me how to do this?
>
> Regards
>
> *Email Disclaimer"This e-mail and any files transmitted with it are confidential and are intended solely for use by the addressee. Any unauthorised dissemination, distribution or copying of this message and any attachments is strictly prohibited. If you have received this e-mail in error, please notify the sender and delete the message. Any views or opinions presented in this e-mail may solely be the views of the author and cannot be relied upon as being those of Dublin City University. E-mail communications such as this cannot be guaranteed to be virus-free, timely, secure or error-free and Dublin City University does not accept liability for any such matters or their consequences. Please consider the environment before printing this e-mail."S?anadh R?omhphoist"T? an r?omhphost seo agus aon chomhad a sheoltar leis faoi r?n agus is lena ?s?id ag an seola? agus sin amh?in ?. T? cosc ioml?n ar scaipeadh, dh?ileadh n? ch?ipe?il neamh?daraithe ar an teachtaireacht seo agus ar !
aon cheangalt?n at? ag dul leis. M? t? an r?omhphost seo faighte agat tr? dhearmad cuir sin in i?l le do thoil don seolt?ir agus scrios an teachtaireacht. D?fh?adfadh s? gurb iad tuairim? an ?dair agus sin amh?in at? in aon tuairim? no dearctha? at? curtha i l?thair sa r?omhphost seo agus n?or ch?ir glacadh leo mar thuairim? n? dhearctha? Ollscoil Chathair Bhaile ?tha Cliath. N? ghlactar leis go bhfuil cumars?id r?omhphoist den s?rt seo saor ? v?reas, in am, sl?n, n? saor ? earr?id agus n? ghlacann Ollscoil Chathair Bhaile ?tha Cliath le dliteanas in aon ch?s den s?rt sin n? as aon iarmhairt a d?eascr?dh astu. Cuimhnigh ar an timpeallacht le do thoil sula gcuireann t? an r?omhphost seo i gcl?."*
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150923/eabeb616/attachment.html
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 107, Issue 53
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 107, Issue 53"
Post a Comment