Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: EMS MML IndexError: list index out of range (Barry Haddow)
2. Re: MOSES build failed (ULStudent:GIOVANNI.GALLO)
----------------------------------------------------------------------
Message: 1
Date: Sun, 26 Jan 2014 12:37:39 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] EMS MML IndexError: list index out of
range
To: jian zhang <jianzhang09@gmail.com>, moses-support@mit.edu
Message-ID: <52E50193.2010609@staffmail.ed.ac.uk>
Content-Type: text/plain; charset="iso-8859-1"
Hi Jian
The logic looks correct to me. If the domains file has been provided, we
then need to check if the sentence is in-domain. If the domains file is
not provided, then all sentences are considered out-of-domain.
The fact that all scores are 99999 means that the MML filter is seeing
all your sentences as in-domain. It could be that something went wrong
during corpus preprocessing, or during the creation of the domains file
(/home/mml/mml-test/experiment/model/domains.1). Do the lengths in the
domains file match the lengths of your in and out corpora?
cheers - Barry
On 25/01/14 03:29, jian zhang wrote:
> Hi Barry, I don't not understand line *if (defined($filter_domains) &&
> !&check_sentence_filtered($i))* at mml-score.perl, before computing
> the bilingual cross-entropy difference,
> Should it not be *if (!defined($filter_domains) &&
> !&check_sentence_filtered($i)) *?
>
> Regards,
>
> Jian Zhang
>
>
>
>
> On Fri, Jan 24, 2014 at 10:27 PM, jian zhang <jianzhang09@gmail.com
> <mailto:jianzhang09@gmail.com>> wrote:
>
> Hi Barry,
>
> All the scores are 99999 in that file.
>
> Thanks,
>
>
> Jian
>
>
> On Fri, Jan 24, 2014 at 3:51 PM, Barry Haddow
> <bhaddow@staffmail.ed.ac.uk <mailto:bhaddow@staffmail.ed.ac.uk>>
> wrote:
>
> Hi Jian
>
> This is a bit suspect:
>
>
> 2014-01-24 14:17:26,276 Retaining at least 0 entries and
> ignoring 2075137
>
> Are the scores in this file sensible (or are they all the same?)
>
> /home/mml/mml-test/experiment/training/corpus-mml-score.1
>
> cheers - Barry
>
>
> On 24/01/14 14:53, jian zhang wrote:
>
> Hi,
>
> I got error of IndexError: list index out of range at the
> TRAINING_mml-filter-before-wa step.
>
> I had read the post at
> https://www.mail-archive.com/moses-support@mit.edu/msg08767.html,
> however I still can not figure out what is wrong.
>
> The full error is
>
> general:strategy = Score
> general:source_language = fr
> general:target_language = en
> general:input_stem =
> /home/mml/mml-test/experiment/training/corpus.1
> general:output_stem =
> /home/mml/mml-test/experiment/training/corpus-mml.1
> general:domain_file =
> /home/mml/mml-test/experiment/model/domains.1
> general:domain_file_out =
> /home/mml/mml-test/experiment/training/corpus-mml.1
> score:score_file =
> /home/mml/mml-test/experiment/training/corpus-mml-score.1
> score:proportion = 0.9
>
> 2014-01-24 14:17:26,276 Retaining at least 0 entries and
> ignoring 2075137
> Traceback (most recent call last):
> File
> "/home/tools/mosesdecoder/scripts/ems/support/mml-filter.py",
> line 156, in <module>
> main()
> File
> "/home/tools/mosesdecoder/scripts/ems/support/mml-filter.py",
> line 111, in main
> strategy = strategy_class(config)
> File
> "/home/tools/mosesdecoder/scripts/ems/support/mml-filter.py",
> line 72, in __init__
> [float(line[:-1]) for line in open(self.score_file)],
> reverse=True)[ignore_count + count]
> IndexError: list index out of range
>
> And my ems configuration file has:
>
> #################################################################
> # PARALLEL CORPUS PREPARATION:
> # create a tokenized, sentence-aligned corpus, ready for
> training
>
> [CORPUS]
>
> #in-domain parallel corpus
> [CORPUS:in]
> clean-stem = $training-in-domain-corpus
>
> [CORPUS:out]
> #out-domain parallel corpus
> clean-stem = $training-out-domain-corpus
>
>
> #################################################################
> # LANGUAGE MODEL TRAINING
> [LM]
> [LM:lm]
> type = 8
> lm = $language-model
> #################################################################
> # MODIFIED MOORE LEWIS FILTERING
>
> [MML]
>
> lm-training = $srilm-dir/ngram-count
> lm-settings = "-interpolate -kndiscount -unk"
> lm-binarizer = $moses-src-dir/bin/build_binary
> lm-query = $moses-src-dir/bin/query
> order = 5
>
> ### in-/out-of-domain source/target corpora to train the 4
> language model
> #
> # in-domain parallel corpus
> indomain-stem = [CORPUS:in:clean-split-stem]
>
> # out-of-domain parallel corpus
> outdomain-stem = [CORPUS:out:clean-split-stem]
>
> # settings: number of lines sampled from the corpora to
> train each language model on
> settings = "--line-count 100000"
>
> #################################################################
> # TRANSLATION MODEL TRAINING
> [TRAINING]
> script = $moses-script-dir/training/train-model.perl
> training-options = "-mgiza -mgiza-cpus 12
> -sort-buffer-size 16G -sort-compress gzip -sort-parallel
> 12 -cores 12"
> parallel = yes
> alignment-symmetrization-method = grow-diag-final-and
> lexicalized-reordering = msd-bidirectional-fe
> score-settings = "--GoodTuring"
> include-word-alignment-in-rules = yes
>
> #space separated all out-of domain corpora to be filtered
> mml-filter-corpora = out
> mml-before-wa = "-proportion 0.9"
>
> #####################################################
>
> Thanks.
>
>
> Jian Zhang
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
> --
> Jian Zhang
> Centre for Next Generation Localisation (CNGL)
> <http://www.cngl.ie/index.html>
> Dublin City University <http://www.dcu.ie/>
>
>
>
>
>
>
> --
> Jian Zhang
> Centre for Next Generation Localisation (CNGL)
> <http://www.cngl.ie/index.html>
> Dublin City University <http://www.dcu.ie/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140126/c909332f/attachment-0001.htm
------------------------------
Message: 2
Date: Sun, 26 Jan 2014 14:24:28 +0000
From: "ULStudent:GIOVANNI.GALLO" <12064866@studentmail.ul.ie>
Subject: Re: [Moses-support] MOSES build failed
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<086e00f4c28544049a970447df38092e@DBXPR01MB078.eurprd01.prod.exchangelabs.com>
Content-Type: text/plain; charset="us-ascii"
Hi Hieu,
The BUILD-INSTRUCTIONS.txt reads at a certain point:
"Add all the above code to your .bashrc or .bash_login as appropriate. Then
you're ready to install packages in non-standard paths"
Should I add that code somewhere? Can it be the problem that is preventing me from compiling mosesdecoder correctly?
Many thanks.
GG
________________________________
Da: hieuhoang@gmail.com <hieuhoang@gmail.com> per conto di Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Inviato: domenica 26 gennaio 2014 00:20
A: ULStudent:GIOVANNI.GALLO
Cc: moses-support@mit.edu
Oggetto: Re: [Moses-support] MOSES build failed
I'm using Mavericks too and it compiles ok. These are the commands i used
Boost:
./bootstrap.sh
./b2 -j8 --prefix=$PWD --libdir=$PWD/lib64 --layout=tagged link=static threading=multi,single install
Moses:
./bjam --with-boost=/Users/hieu/workspace/boost/boost_1_55_0
I have macport installed, but not gcc or boost
On 25 January 2014 20:35, ULStudent:GIOVANNI.GALLO <12064866@studentmail.ul.ie<mailto:12064866@studentmail.ul.ie>> wrote:
Hi Hieu,
Thanks for the quick reply!
I followed the installation instructions on the manual and (apparently) git clone download all source files correctly and boost compiled ok (at least, I didn't get any error message at the end).
I'm using te last version of Mavericks (OS X 10.9.1) and gcc:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn)
Target: x86_64-apple-darwin13.0.0
Thread model: posix
Many thanks!
GG
________________________________
Da: hieuhoang@gmail.com<mailto:hieuhoang@gmail.com> <hieuhoang@gmail.com<mailto:hieuhoang@gmail.com>> per conto di Hieu Hoang <Hieu.Hoang@ed.ac.uk<mailto:Hieu.Hoang@ed.ac.uk>>
Inviato: sabato 25 gennaio 2014 21:09
A: ULStudent:GIOVANNI.GALLO
Cc: moses-support@mit.edu<mailto:moses-support@mit.edu>
Oggetto: Re: [Moses-support] MOSES build failed
strange.
Did git clone download all the source files correctly? DId boost compile ok?
What OS and gcc version are you using?
On 25 January 2014 19:54, ULStudent:GIOVANNI.GALLO <12064866@studentmail.ul.ie<mailto:12064866@studentmail.ul.ie>> wrote:
Hi all,
Please find the build log file attached. I executed the following command when I got the error:
./bjam --with-boost=/Users/Giancarlo/boost_1_55_0 -j8
Many thanks.
Best regards,
Giovanni Gallo.
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu<mailto:Moses-support@mit.edu>
http://mailman.mit.edu/mailman/listinfo/moses-support
--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140126/37095878/attachment.htm
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 87, Issue 59
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 87, Issue 59"
Post a Comment