Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Tuning for factored phrase based systems (Sa?o Kuntaric)
2. Biconcor: Permission denied (Fathima Farhath Farook)
3. too few factors error in mert (Hasan Sait ARSLAN)
----------------------------------------------------------------------
Message: 1
Date: Tue, 6 Dec 2016 09:55:26 +0100
From: Sa?o Kuntaric <saso.kuntaric@gmail.com>
Subject: Re: [Moses-support] Tuning for factored phrase based systems
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CANsquDroWevThOrFFCSas8TGwd1n059p0T_JNX96u=ZxBcjbGQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi Angli,
Here is an excerpt of Hieu's answers regarding this topic when I was doing
research in factored models, might be of some help:
On 30/06/2016 21:44, Sa?o Kuntaric wrote:
Hi all,
I would like to ask one more question. When you say that my reference only
has the surface form, are you talking about the "tuning corpus", which in
the case of my command
~/mosesdecoder/scripts/training/mert-moses.pl ~/working/IT_corpus/TMX/txt/
factored_corpus/singles/tuning_corpus.tagged.clean.en
~/working/IT_corpus/TMX/txt/factored_corpus/singles/tuning
_corpus.tagged.clean.sl ~/mosesdecoder/bin/moses
~/working/IT_corpus/TMX/txt/factored_corpus/singles/test/model/moses.ini
--mertdir ~/mosesdecoder/bin/ --decoder-flags="-threads all"
are tuning_corpus.tagged.clean.en and tuning_corpus.tagged.clean.sl? Can
tuning be done with files that only contains surface forms?
it's usual that the reference tuning data does not have factors, even if
there are factors in the phrase table. After all, you don't care if the
output surface form is correct but the other factors are wrong.
Will the results be compatible with tuning done with a factored tuning
corpus?
yes
Best regards,
Sa?o
2016-12-04 1:37 GMT+01:00 Hieu Hoang <hieuhoang@gmail.com>:
>
>
> Hieu
> Sent while bumping into things
>
> On 1 Dec 2016 07:01, "Angli Liu" <anglil@cs.washington.edu> wrote:
>
> Hi, what's the major difference between the tuning process for a factored
> phrase based system (i.e., surface+pos data) and a simple baseline phrase
> based system?
>
>
> Nothing, the tuning just optimise weights for feature functions.
>
> If you decompose your translation so that it has multiple phrase tables
> and generation models, then they are just extra feature functions with
> weights to be tuned
>
> Do I need to organize the dev set the same way as the training set (i.e.,
> surface|pos)?
>
> Yes
>
> Is there a tutorial on the moses website on this topic?
>
> Maybe this
> http://www.statmt.org/moses/?n=FactoredTraining.FactoredTraining
>
>
> Thanks!
>
> -Angli
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
--
lp,
Sa?o
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161206/b50b479a/attachment-0001.html
------------------------------
Message: 2
Date: Tue, 6 Dec 2016 14:45:13 +0530
From: Fathima Farhath Farook <fathimafarhath@gmail.com>
Subject: [Moses-support] Biconcor: Permission denied
To: moses-support@mit.edu
Message-ID:
<CAP6fBzBqq0imdTOfyCyoXmUTN_Od06hVx=NCTigDEQ5T=fVrUg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Dear all,
I have been doing some experiments with EMS for some time and i was able to
check the statistics over 'biconcor'.
For very recent, I couldn't get the statistic out of the concorder in the
EMS test result page and it returned permission denied.
So to check if something went wrong with the permission, i tried the
following command in the terminal. But I work fine and gave methe dfesired
result.
sudo -u apache LD_LIBRARY_PATH=/home/farhath/moses/mosesdecoder/opt/lib/
/var/www/html/EMS/biconcor -html -l
/var/www/html/test/ems/ST2/model/biconcor.13 -Q 4La74LeP4Lai4LeK4oCN4La6
Can anyone guide me in this regard.
--
Regards,
Farhath Farook
<http://farhathforever.blogspot.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161206/2afd0fdd/attachment-0001.html
------------------------------
Message: 3
Date: Tue, 6 Dec 2016 11:18:38 +0200
From: Hasan Sait ARSLAN <hasan.sait.arslan@gmail.com>
Subject: [Moses-support] too few factors error in mert
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAJj7HD3f5HggUN25rP8UB_eRVyNSVoVLhG_YKUp3ytLnTXT2HA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
I have a factored dataset. It involves 4 factors,
factor1|factor2|factor3|factor4. I have trained my model with such a
dataset.
Now when I want to tune my model, I encounter with the following error:
*Exception: moses/Word.cpp:159 in void
Moses::Word::CreateFromString(Moses::FactorDirection, const
std::vector<long unsigned int>&, const StringPiece&, bool, bool) threw
util::Exception because `!isNonTerminal && i < factorOrder.size()'.Too few
factors in string '-|-|Punc|Punc*
The details of the error is in mert.txt file, which is attached to this
e-mail.
Thanks,
Kind Regards,
Hasan Sait Arslan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161206/03a4163b/attachment.html
-------------- next part --------------
Using SCRIPTS_ROOTDIR: /export/students/sait/build/mosesdecoder/scripts
filtering the phrase tables... T dets 6 10:59:05 EET 2016
exec: /export/students/sait/build/mosesdecoder/scripts/training/filter-model-given-input.pl ./filtered /export/students/sait/factored_translation/working/unfactored/model/moses.ini /export/students/sait/factored_translation/dataset/dev.tr
Executing: /export/students/sait/build/mosesdecoder/scripts/training/filter-model-given-input.pl ./filtered /export/students/sait/factored_translation/working/unfactored/model/moses.ini /export/students/sait/factored_translation/dataset/dev.tr > filterphrases.out 2> filterphrases.err
Asking moses for feature names and values from filtered/moses.ini
Executing: /export/students/sait/build/mosesdecoder/bin/moses -threads all -config filtered/moses.ini -show-weights
exec: /export/students/sait/build/mosesdecoder/bin/moses -threads all -config filtered/moses.ini -show-weights
Executing: /export/students/sait/build/mosesdecoder/bin/moses -threads all -config filtered/moses.ini -show-weights > ./features.list 2> /dev/null
MERT starting values and ranges for random generation:
Distortion0 = 0.300 ( 0.00 .. 1.00)
LM0 = 0.500 ( 0.00 .. 1.00)
WordPenalty0 = -1.000 ( 0.00 .. 1.00)
PhrasePenalty0 = 0.200 ( 0.00 .. 1.00)
TranslationModel0 = 0.200 ( 0.00 .. 1.00)
TranslationModel0 = 0.200 ( 0.00 .. 1.00)
TranslationModel0 = 0.200 ( 0.00 .. 1.00)
TranslationModel0 = 0.200 ( 0.00 .. 1.00)
featlist: Distortion0=0.300000
featlist: LM0=0.500000
featlist: WordPenalty0=-1.000000
featlist: PhrasePenalty0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel0=0.200000
featlist: TranslationModel0=0.200000
run 1 start at T dets 6 11:01:28 EET 2016
Parsing --decoder-flags: |-threads all|
Saving new config to: ./run1.moses.ini
Saved: ./run1.moses.ini
Normalizing lambdas: 0.300000 0.500000 -1.000000 0.200000 0.200000 0.200000 0.200000 0.200000
DECODER_CFG = -weight-overwrite 'TranslationModel0= 0.071429 0.071429 0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 0.107143 WordPenalty0= -0.357143'
Executing: /export/students/sait/build/mosesdecoder/bin/moses -threads all -config filtered/moses.ini -weight-overwrite 'TranslationModel0= 0.071429 0.071429 0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 0.107143 WordPenalty0= -0.357143' -n-best-list run1.best100.out 100 distinct -input-file /export/students/sait/factored_translation/dataset/dev.tr > run1.out
Executing: /export/students/sait/build/mosesdecoder/bin/moses -threads all -config filtered/moses.ini -weight-overwrite 'TranslationModel0= 0.071429 0.071429 0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 0.107143 WordPenalty0= -0.357143' -n-best-list run1.best100.out 100 distinct -input-file /export/students/sait/factored_translation/dataset/dev.tr > run1.out
1-10.20.2 0.2 0.2 0.20.30.5(1) run decoder to produce n-best lists
params = -threads all
decoder_config = -weight-overwrite 'TranslationModel0= 0.071429 0.071429 0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 0.107143 WordPenalty0= -0.357143'
Defined parameters (per moses.ini or switch):
config: filtered/moses.ini
distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/export/students/sait/factored_translation/working/mert-work/filtered/phrase-table.0-0.1.1.gz input-factor=0 output-factor=0 Distortion KENLM name=LM0 factor=0 path=/export/students/sait/factored_translation/lm/surface.lm order=3
input-factors: 0 1 2 3 4
input-file: /export/students/sait/factored_translation/dataset/dev.tr
mapping: 0 T 0
n-best-list: run1.best100.out 100 distinct
threads: all
weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 Distortion0= 0.3 LM0= 0.5
weight-overwrite: TranslationModel0= 0.071429 0.071429 0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 0.107143 WordPenalty0= -0.357143
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhrasePenalty
FeatureFunction: PhrasePenalty0 start: 2 end: 2
line=PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/export/students/sait/factored_translation/working/mert-work/filtered/phrase-table.0-0.1.1.gz input-factor=0 output-factor=0
FeatureFunction: TranslationModel0 start: 3 end: 6
line=Distortion
FeatureFunction: Distortion0 start: 7 end: 7
line=KENLM name=LM0 factor=0 path=/export/students/sait/factored_translation/lm/surface.lm order=3
Loading the LM will be faster if you build a binary file.
Reading /export/students/sait/factored_translation/lm/surface.lm
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
FeatureFunction: LM0 start: 8 end: 8
Loading UnknownWordPenalty0
Loading WordPenalty0
Loading PhrasePenalty0
Loading Distortion0
Loading LM0
Loading TranslationModel0
Start loading text phrase table. Moses format : [27.627] seconds
Reading /export/students/sait/factored_translation/working/mert-work/filtered/phrase-table.0-0.1.1.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
Created input-output object : [56.159] seconds
Exception: moses/Word.cpp:159 in void Moses::Word::CreateFromString(Moses::FactorDirection, const std::vector<long unsigned int>&, const StringPiece&, bool, bool) threw util::Exception because `!isNonTerminal && i < factorOrder.size()'.
Too few factors in string '-|-|Punc|Punc'.
Exit code: 1
The decoder died. CONFIG WAS -weight-overwrite 'TranslationModel0= 0.071429 0.071429 0.071429 0.071429 LM0= 0.178571 PhrasePenalty0= 0.071429 Distortion0= 0.107143 WordPenalty0= -0.357143'
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 122, Issue 7
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 122, Issue 7"
Post a Comment