Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: tuning not working properly in factored model (Matthias Huck)
2. Re: feature function referring headers in phrase-extract?
(Hieu Hoang)
3. Re: tuning not working properly in factored model
(Carlos Escolano)
----------------------------------------------------------------------
Message: 1
Date: Thu, 28 Apr 2016 21:14:27 +0200
From: Matthias Huck <mhuck@cis.lmu.de>
Subject: Re: [Moses-support] tuning not working properly in factored
model
To: Carlos Escolano <carlos.e.p93@gmail.com>
Cc: moses-support@mit.edu, Ondrej Bojar <bojar@ufal.mff.cuni.cz>,
Marta Ruiz <martaruizcostajussa@gmail.com>
Message-ID: <1461870867.18419.105.camel@cis.lmu.de>
Content-Type: text/plain; charset="UTF-8"
Hi Carlos,
Have you tried switching off MBR decoding during tuning? (Run Moses
without the -mbr parameter.) The exception it throws seems to suggest
that MBR doesn't work with more than a single output factor.
Cheers,
Matthias
On Thu, 2016-04-28 at 21:09 +0200, Carlos Escolano wrote:
> Hi,
>
> Thank you for your answer
>
> I've tried setting the output factors in the moses.ini before and
> mert-moses.pl throws the following error:
>
> Loading table into memory...done.
> terminate called after throwing an instance of 'util::Exception'
> what(): moses/mbr.cpp:112 in const Moses::TrellisPath doMBR(const
> Moses::TrellisPathList&, const Moses::AllOptions&) threw
> util::Exception
> because `oFactors.size() != 1'.
> Need exactly one output factor!
>
> Using the moses.ini without tuning all factors are generated, It's
> during
> the tuning process that only the forms appear.
>
> Best regards,
>
> Carlos
>
> 2016-04-28 20:14 GMT+02:00 Matthias Huck <mhuck@cis.lmu.de>:
>
> > Hi,
> >
> > Moses can be configured to output the target-side factors of your
> > choice.
> > Add something like this to your moses.ini:
> >
> > [output-factors]
> > 0
> > 1
> > 2
> >
> > Cheers,
> > Matthias
> >
> >
> > On Thu, 2016-04-28 at 18:16 +0200, Carlos Escolano wrote:
> > > Hi,
> > >
> > > Thank you for your answer.
> > >
> > > You are right. While the phrase table has all three factors in
> > > the
> > > run.X.best.out only the form appears.
> > >
> > > I'll check why this is happening.
> > >
> > > Best Regards,
> > >
> > > Carlos
> > >
> > >
> > >
> > > 2016-04-28 8:46 GMT+02:00 Ondrej Bojar <bojar@ufal.mff.cuni.cz>:
> > >
> > > > Dear Carlos,
> > > >
> > > > My frequent mistake in this respect is the match of factor
> > representation
> > > > in run.X.best.out and the reference sentences.
> > > >
> > > > Technically, both is possible: evaluating only the first factor
> > > > (form)
> > or
> > > > all factors of each token. BLEU does not care. Mismatch will
> > > > cause
> > terribly
> > > > low scores.
> > > >
> > > > O.
> > > >
> > > >
> > > > On April 27, 2016 9:48:50 PM CEST, Carlos Escolano <
> > carlos.e.p93@gmail.com>
> > > > wrote:
> > > > > Hi,
> > > > >
> > > > > I trained a chinese to spanish unfacored model and all worked
> > > > > perfectly.
> > > > > But when I try to train a factored model for the same task I
> > > > > have
> > some
> > > > > trouble while tuning. The factors I'm using are only words
> > > > > for
> > chinese
> > > > > and
> > > > > words, lemmas and POS tags for spanish.
> > > > >
> > > > > Training seems to finish correctly and the phrase tables
> > > > > shows all
> > the
> > > > > factors but when tuning t it only does 2 runs and prints a
> > > > > message
> > > > > saying
> > > > > that weights have not change in the last run. Leaving the
> > > > > original
> > > > > weights.
> > > > > Also when translating, the BLEU obtained is worse than the
> > > > > obtained
> > > > > with
> > > > > the not factored model.
> > > > >
> > > > >
> > > > > These are my calls for training and tuning the model:
> > > > >
> > > > > $SCRIPTS_ROOTDIR/training/train-model.perl \
> > > > > -external-bin-dir $GIZA_DIR/mgiza-bin -mgiza \
> > > > > --corpus $WORKING_DIR/train/train \
> > > > > --alignment grow-diag-final-and \
> > > > > --score-options '--GoodTuring' \
> > > > > --root-dir $WORKING_DIR/baseline/ \
> > > > > --f zh --e es \
> > > > > --lm 0:5:$WORKING_DIR/baseline/lm/words.lm.es:0 \
> > > > > --translation-factors 0-0,1,2 \
> > > > > --reordering msd-bidirectional-fe \
> > > > > --reordering-factors 0-0 \
> > > > >
> > > > > $MOSES_SCRIPTS/training/mert-moses.pl \
> > > > > $WORKING_DIR/dev/dev.zh \
> > > > > $WORKING_DIR/dev/dev.es \
> > > >
> > > > >
> > $MOSES_DIR/moses-cmd/bin/gcc-4.8.5/release/link-static/threading
> > -multi/moses
> > > > > \
> > > > > $WORKING_DIR/baseline/model/moses.ini \
> > > > > --nbest 100 \
> > > > > --working-dir $WORKING_DIR/baseline/tuning/ \
> > > > > --decoder-flags "-drop-unknown -mbr -threads 24 -mp -v 0" \
> > > > > --rootdir $MOSES_SCRIPTS \
> > > > > --mertdir $MOSES_DIR/bin/ \
> > > > > -threads 24 \
> > > > > --filtercmd
> > > > > '/veu4/usuaris24/xtrans/mosesdecoder/scripts/training/
> > > > > filter-model-given-input.pl'
> > > > >
> > > >
> > > > >
> > /veu4/usuaris24/smt/softlic/mosesdecoder/scripts//ems/support/reuse
> > -weights.perl
> > > > > \
> > > > > $WORKING_DIR/baseline/tuning/moses.ini <
> > > > > $WORKING_DIR/baseline/model/moses.ini >
> > > > > $WORKING_DIR/baseline/tuning/moses.weight-reused.ini
> > > > >
> > > > >
> > > > > Best regards,
> > > > >
> > > > > Carlos
> > > > >
> > > > >
> > > > >
> > -------------------------------------------------------------------
> > -----
> > > > >
> > > > > _______________________________________________
> > > > > Moses-support mailing list
> > > > > Moses-support@mit.edu
> > > > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > > >
> > > > --
> > > > Ondrej Bojar (mailto:obo@cuni.cz / bojar@ufal.mff.cuni.cz)
> > > > http://www.cuni.cz/~obo
> > > >
> > > _______________________________________________
> > > Moses-support mailing list
> > > Moses-support@mit.edu
> > > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
------------------------------
Message: 2
Date: Fri, 29 Apr 2016 08:59:22 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] feature function referring headers in
phrase-extract?
To: Shuoyang Ding <mtsding@gmail.com>, moses-support
<moses-support@mit.edu>
Message-ID: <5723145A.5080709@gmail.com>
Content-Type: text/plain; charset="windows-1252"
You should be careful of referencing files in phrase-extract/ if you are
developing code that is in moses/
The code in phrase-extract/ will not be linked.
It's probably safer to simply copy what you need from phrase-extract/
into moses/ and use the copy
On 25/04/2016 17:16, Shuoyang Ding wrote:
> Hi all,
>
> I?m developing a feature function that involves some headers in
> phrase-extract:
>
> phrase-extract/extract-ghkm/AlignmentGraph.h
> phrase-extract/extract-ghkm/Rule.h
> phrase-extract/extract-ghkm/ScfgRule.h
> phrase-extract/syntax-common/xml_tree_parser.h
>
> Now I?m having some issue with recompiling with this new feature
> function ? these headers seems not correctly referenced. I attached
> the build log with this email. Specifically:
>
> ./phrase-extract/extract-ghkm/AlignmentGraph.h:28:24: fatal error:
> SyntaxTree.h: No such file or directory
>
> I guess I may have to change moses/Jamfile to accommodate this change.
> Is there anyone who is familiar with bjam who can help me?
>
> Thanks!
>
> Regards,
> Shuoyang Ding
>
> Ph.D. Student
> Center for Language and Speech Processing
> Department of Computer Science
> Johns Hopkins University
>
> Hackerman Hall 225A
> 3400 N. Charles St.
> Baltimore, MD 21218
>
> http://cs.jhu.edu/~sding <http://cs.jhu.edu/%7Esding>
>
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160429/9da8cae5/attachment-0001.html
------------------------------
Message: 3
Date: Fri, 29 Apr 2016 14:22:03 +0200
From: Carlos Escolano <carlos.e.p93@gmail.com>
Subject: Re: [Moses-support] tuning not working properly in factored
model
To: Matthias Huck <mhuck@cis.lmu.de>
Cc: moses-support@mit.edu, Ondrej Bojar <bojar@ufal.mff.cuni.cz>,
Marta Ruiz <martaruizcostajussa@gmail.com>
Message-ID:
<CACvrzNTxZmQDqOkpQ8uL1iVJWvyzn_b8wY-ZYcWDtkCE2tNEFA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi Matthias,
Thank you, I really apreciate your help.
It seems that was the problem. I'm checking the tuning results and it
did 8 runs and returned all factors.
Hope the BLEU would be better this time.
Best regards,
Carlos
El 28/4/2016 9:14 p. m., "Matthias Huck" <mhuck@cis.lmu.de> escribi?:
> Hi Carlos,
>
> Have you tried switching off MBR decoding during tuning? (Run Moses
> without the -mbr parameter.) The exception it throws seems to suggest
> that MBR doesn't work with more than a single output factor.
>
> Cheers,
> Matthias
>
> On Thu, 2016-04-28 at 21:09 +0200, Carlos Escolano wrote:
> > Hi,
> >
> > Thank you for your answer
> >
> > I've tried setting the output factors in the moses.ini before and
> > mert-moses.pl throws the following error:
> >
> > Loading table into memory...done.
> > terminate called after throwing an instance of 'util::Exception'
> > what(): moses/mbr.cpp:112 in const Moses::TrellisPath doMBR(const
> > Moses::TrellisPathList&, const Moses::AllOptions&) threw
> > util::Exception
> > because `oFactors.size() != 1'.
> > Need exactly one output factor!
> >
> > Using the moses.ini without tuning all factors are generated, It's
> > during
> > the tuning process that only the forms appear.
> >
> > Best regards,
> >
> > Carlos
> >
> > 2016-04-28 20:14 GMT+02:00 Matthias Huck <mhuck@cis.lmu.de>:
> >
> > > Hi,
> > >
> > > Moses can be configured to output the target-side factors of your
> > > choice.
> > > Add something like this to your moses.ini:
> > >
> > > [output-factors]
> > > 0
> > > 1
> > > 2
> > >
> > > Cheers,
> > > Matthias
> > >
> > >
> > > On Thu, 2016-04-28 at 18:16 +0200, Carlos Escolano wrote:
> > > > Hi,
> > > >
> > > > Thank you for your answer.
> > > >
> > > > You are right. While the phrase table has all three factors in
> > > > the
> > > > run.X.best.out only the form appears.
> > > >
> > > > I'll check why this is happening.
> > > >
> > > > Best Regards,
> > > >
> > > > Carlos
> > > >
> > > >
> > > >
> > > > 2016-04-28 8:46 GMT+02:00 Ondrej Bojar <bojar@ufal.mff.cuni.cz>:
> > > >
> > > > > Dear Carlos,
> > > > >
> > > > > My frequent mistake in this respect is the match of factor
> > > representation
> > > > > in run.X.best.out and the reference sentences.
> > > > >
> > > > > Technically, both is possible: evaluating only the first factor
> > > > > (form)
> > > or
> > > > > all factors of each token. BLEU does not care. Mismatch will
> > > > > cause
> > > terribly
> > > > > low scores.
> > > > >
> > > > > O.
> > > > >
> > > > >
> > > > > On April 27, 2016 9:48:50 PM CEST, Carlos Escolano <
> > > carlos.e.p93@gmail.com>
> > > > > wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I trained a chinese to spanish unfacored model and all worked
> > > > > > perfectly.
> > > > > > But when I try to train a factored model for the same task I
> > > > > > have
> > > some
> > > > > > trouble while tuning. The factors I'm using are only words
> > > > > > for
> > > chinese
> > > > > > and
> > > > > > words, lemmas and POS tags for spanish.
> > > > > >
> > > > > > Training seems to finish correctly and the phrase tables
> > > > > > shows all
> > > the
> > > > > > factors but when tuning t it only does 2 runs and prints a
> > > > > > message
> > > > > > saying
> > > > > > that weights have not change in the last run. Leaving the
> > > > > > original
> > > > > > weights.
> > > > > > Also when translating, the BLEU obtained is worse than the
> > > > > > obtained
> > > > > > with
> > > > > > the not factored model.
> > > > > >
> > > > > >
> > > > > > These are my calls for training and tuning the model:
> > > > > >
> > > > > > $SCRIPTS_ROOTDIR/training/train-model.perl \
> > > > > > -external-bin-dir $GIZA_DIR/mgiza-bin -mgiza \
> > > > > > --corpus $WORKING_DIR/train/train \
> > > > > > --alignment grow-diag-final-and \
> > > > > > --score-options '--GoodTuring' \
> > > > > > --root-dir $WORKING_DIR/baseline/ \
> > > > > > --f zh --e es \
> > > > > > --lm 0:5:$WORKING_DIR/baseline/lm/words.lm.es:0 \
> > > > > > --translation-factors 0-0,1,2 \
> > > > > > --reordering msd-bidirectional-fe \
> > > > > > --reordering-factors 0-0 \
> > > > > >
> > > > > > $MOSES_SCRIPTS/training/mert-moses.pl \
> > > > > > $WORKING_DIR/dev/dev.zh \
> > > > > > $WORKING_DIR/dev/dev.es \
> > > > >
> > > > > >
> > > $MOSES_DIR/moses-cmd/bin/gcc-4.8.5/release/link-static/threading
> > > -multi/moses
> > > > > > \
> > > > > > $WORKING_DIR/baseline/model/moses.ini \
> > > > > > --nbest 100 \
> > > > > > --working-dir $WORKING_DIR/baseline/tuning/ \
> > > > > > --decoder-flags "-drop-unknown -mbr -threads 24 -mp -v 0" \
> > > > > > --rootdir $MOSES_SCRIPTS \
> > > > > > --mertdir $MOSES_DIR/bin/ \
> > > > > > -threads 24 \
> > > > > > --filtercmd
> > > > > > '/veu4/usuaris24/xtrans/mosesdecoder/scripts/training/
> > > > > > filter-model-given-input.pl'
> > > > > >
> > > > >
> > > > > >
> > > /veu4/usuaris24/smt/softlic/mosesdecoder/scripts//ems/support/reuse
> > > -weights.perl
> > > > > > \
> > > > > > $WORKING_DIR/baseline/tuning/moses.ini <
> > > > > > $WORKING_DIR/baseline/model/moses.ini >
> > > > > > $WORKING_DIR/baseline/tuning/moses.weight-reused.ini
> > > > > >
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > Carlos
> > > > > >
> > > > > >
> > > > > >
> > > -------------------------------------------------------------------
> > > -----
> > > > > >
> > > > > > _______________________________________________
> > > > > > Moses-support mailing list
> > > > > > Moses-support@mit.edu
> > > > > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > > > >
> > > > > --
> > > > > Ondrej Bojar (mailto:obo@cuni.cz / bojar@ufal.mff.cuni.cz)
> > > > > http://www.cuni.cz/~obo
> > > > >
> > > > _______________________________________________
> > > > Moses-support mailing list
> > > > Moses-support@mit.edu
> > > > http://mailman.mit.edu/mailman/listinfo/moses-support
> > >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160429/7f1018ce/attachment.html
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 114, Issue 63
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 114, Issue 63"
Post a Comment