Moses-support Digest, Vol 107, Issue 8

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: [Moses-developers] Generation models with Mmsapt (Hieu Hoang)
2. Translation Model binarizing step in EMS - multicore ?
(Vincent Nguyen)
3. Re: Translation Model binarizing step in EMS - multicore ?
(Marcin Junczys-Dowmunt)
4. Re: Several Issues with Baseline and EMS (Jeroen Vermeulen)


----------------------------------------------------------------------

Message: 1
Date: Wed, 2 Sep 2015 09:56:08 +0300
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] [Moses-developers] Generation models with
Mmsapt
To: ugermann@inf.ed.ac.uk, Michael Denkowski
<michael.j.denkowski@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <55E69D88.9050107@gmail.com>
Content-Type: text/plain; charset="windows-1252"

It should work. The function
EvaluateInIsolation()
in the LM is for optimisation reason. eg. if the target phrase is 'a b c
d' and the LM is a trigram, the trigrams 'a b c' and 'b c d' can be
precalculated in EvaluateInIsolation().

Implementing a pt for factors requires setting up some variables, which
may not have happen yet in mmsapt. if you can send me a small example
model, i'll see what i can do

On 01/09/2015 02:11, Ulrich Germann wrote:
> Hi Michael,
>
> I have no experience with factored models, so I'm speculating here to
> some degree. The reason the phrase table calls EvaluateInIsolation is
> because all "isolated" phrase scores are considered when pruning. In
> my opinion pruning should not happen within the phrase tables (for
> exactly the reason that it does not allow feature functions to be
> agnostic about other feature functions) but by whatever object calls
> all the phrase tables and does the generation. However, for software
> legacy reasons, that's the way it is right now, and I'm not likely to
> address this issue any time soon myself. The most reasonable fix for
> this in my opinion is to move pruning where it belongs --- post all
> the factor generation stuff.
>
> Hieu is probably still the person with the best understanding of how
> factored phrase table entry generation works, so maybe he can chime in
> on this ...
>
> Cheers - Uli
>
>
> On Mon, Aug 31, 2015 at 11:29 PM, Michael Denkowski
> <michael.j.denkowski@gmail.com <mailto:michael.j.denkowski@gmail.com>>
> wrote:
>
> Hi Ulrich,
>
> I was looking into using a class-based LM with your dynamic phrase
> table via generation models. I translate factor 0 to 0 with the
> Mmsapt, then generate target factor 1 (word class) with a GM. The
> class-based LM operates on factor 1.
>
> I'm hitting a segfault on what appears to be an
> order-of-operations issue with the PT and LM. In mmsapt.cpp:578,
> Mmsapt::mkTPhrase makes a call to tp->EvaluateInIsolation. This
> calls all of the models, including the LMs. The class LM tries to
> score factor 1, which doesn't exist yet (since generation happens
> after translation), and it dies. By nature, other phrase tables
> don't have this issue since they can just pull up pre-computed scores.
>
> Is scoring with all of the models here a strategic choice to get
> better performance or would it be sufficient to just score with
> the PT features? Thanks!
>
> --Michael
>
>
>
>
> --
> Ulrich Germann
> Senior Researcher
> School of Informatics
> University of Edinburgh
>
>
> _______________________________________________
> Moses-developers mailing list
> Moses-developers@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-developers

--
Hieu Hoang
Researcher
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150902/ed1eec13/attachment-0001.html

------------------------------

Message: 2
Date: Wed, 2 Sep 2015 10:43:11 +0200
From: Vincent Nguyen <vnguyen@neuf.fr>
Subject: [Moses-support] Translation Model binarizing step in EMS -
multicore ?
To: moses-support <moses-support@mit.edu>
Message-ID: <55E6B69F.7000408@neuf.fr>
Content-Type: text/plain; charset=utf-8; format=flowed

Hi,

Unless I am mistaken, it seems that binarizing the TM step in EMS in not
multi core.

ttable-binarizer = "$moses-bin-dir/processPhraseTableMin"

[training]

training-options = "-mgiza -mgiza-cpus 8 -sort-compress gzip
-sort-parallel 4 -cores 4"

binarize-all = $moses-script-dir/training/binarize-model.perl


the last line does not take into account the -cores 4

should we put -nscores 4 somewhere so that binarizing goes multicore ?


Vincent


------------------------------

Message: 3
Date: Wed, 02 Sep 2015 10:52:54 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Translation Model binarizing step in EMS
- multicore ?
To: Vincent Nguyen <vnguyen@neuf.fr>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <1ba4b989380de1cd5c2e597a513793f4@amu.edu.pl>
Content-Type: text/plain; charset="utf-8"



-nscores means "number of scores". For multi-threading "-threads 4" can
be used. Using more than 12 threads is not recommended.

Best,

Marcin

W dniu 2015-09-02 10:43, Vincent Nguyen napisa?(a):

> Hi,
>
> Unless I am mistaken, it seems that binarizing the TM step in EMS in not
> multi core.
>
> ttable-binarizer = "$moses-bin-dir/processPhraseTableMin"
>
> [training]
>
> training-options = "-mgiza -mgiza-cpus 8 -sort-compress gzip
> -sort-parallel 4 -cores 4"
>
> binarize-all = $moses-script-dir/training/binarize-model.perl
>
> the last line does not take into account the -cores 4
>
> should we put -nscores 4 somewhere so that binarizing goes multicore ?
>
> Vincent
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support [1]



Links:
------
[1] http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150902/f6106e17/attachment-0001.html

------------------------------

Message: 4
Date: Wed, 02 Sep 2015 16:59:53 +0700
From: Jeroen Vermeulen <jtv@precisiontranslationtools.com>
Subject: Re: [Moses-support] Several Issues with Baseline and EMS
To: Anita Pal <fujitahibaraki@gmail.com>, moses-support@mit.edu
Message-ID:
<C5C478FF-C180-420C-BD3A-9AFC62BC1096@precisiontranslationtools.com>
Content-Type: text/plain; charset="utf-8"

Yes, the semicolon ends one command and starts another.

Don't set HOME. The shell does that for you. If you want a different setting than your actual home directory, give it a different name or you might confuse programs you run from that shell.


Jeroen

On September 2, 2015 1:26:43 AM GMT+07:00, Anita Pal <fujitahibaraki@gmail.com> wrote:
>Hey!
>
>I'm really, really new to Linux and have no idea how to set /change the
>config file when it comes to running the experiments as described here
>(
>http://www.statmt.org/moses/?n=moses.baseline)
>
>home-dir = /home/liam/
>
>working-dir =/home/liam/working/experiments
>moses-src-dir = /home/liam/mosesdecoder
>moses-script-dir = home/liam/working/experiments (this is where config
>is
>located?)
>moses-bin-dir = /home/liam/mosesdecoder/bin
>external-bin-dir =/home/liam/mosesdecoder/tools
>data-dir =/home/liam/corpus
>train-dir =/home/liam/corpus/training
>dev-dir = /home/liam/corpus/dev
>irstlm-dir =/home/liam/irstlm/bin
>
>Is this correct? I really have no idea ):
>
>Because I just keep getting errors no matter what I do. I have the same
>problem when it comes to training the language model via IRSTLM. For
>example:
>
>export IRSTLM=$HOME/irstlm; ~/irstlm/bin/build-lm.sh \
> -i news-commentary-v8.fr-en.sb.en \
> -t ./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en
> ~/irstlm/bin/compile-lm \
> --text=yes \
> news-commentary-v8.fr-en.lm.en.gz \
> news-commentary-v8.fr-en.arpa.en
>
>I assume this is a separate command. I keep getting errors though I do
>set HOME=/home/liam/irstlm.
>
>What am I doing wrong?
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Moses-support mailing list
>Moses-support@mit.edu
>http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150902/7d31a447/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 107, Issue 8
*********************************************

Tweet