Moses-support Digest, Vol 108, Issue 83

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Multiple generation factors in factored model
(Rajen Chatterjee)
2. Re: At least fileset to keep lm, tm and tuning result
(Philipp Koehn)
3. Error while training the translation system (Sunayana Gawde)
4. Re: Error while training the translation system (Saumitra Yadav)


----------------------------------------------------------------------

Message: 1
Date: Fri, 30 Oct 2015 06:06:04 +0100
From: Rajen Chatterjee <rajen.k.chatterjee@gmail.com>
Subject: Re: [Moses-support] Multiple generation factors in factored
model
To: Anoop (?????) <anoop.kunchukuttan@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAC4-+NzSnSTL9AsJvUHt2oHF1h7UbyDge9K8Z_54K3or2zhVoQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Anoop,

One of the possible reason could be that you model (with 3 generation
factors) is too specific (the more factors you add, the more specific your
model will be) and your data might be limited and sparse to cover all
possible combination of factors.

You can try multiple decoding path in moses (for each decoding path you can
specify one combination of factors)
http://www.statmt.org/moses/?n=FactoredTraining.FactoredTraining
Also you can find example in moses manual.

On Thu, Oct 29, 2015 at 2:12 PM, Anoop (?????) <anoop.kunchukuttan@gmail.com
> wrote:

> Hi,
>
> I have a training corpus with multiple factors on the target side. I
> experimented with various factor configurations - one for generation of
> each target factor from the surface form and then use an LM over the factor
> as a feature. The surface form to factor mappings are fairly deterministic,
> so the LM over factors is where I hope to see benefits. Indeed I do obtain
> significant improvements in output with each individual factor over a PBSMT
> system. However, when I put together multiple factors along with multiple
> language models, the results actually don't as much improvement as using
> some of the individual factors. The performance generally better the
> baseline PBSMT though in most cases.
>
> Do you have some suggestions regarding why this would be so, and if this
> could be rectified?
> I have attached the moses.ini for the factored system with three
> generation factors.
>
> Regards
> Anoop.
>
> --
> I claim to be a simple individual liable to err like any other fellow
> mortal. I own, however, that I have humility enough to confess my errors
> and to retrace my steps.
>
> http://flightsofthought.blogspot.com
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
-Regards,
Rajen Chatterjee.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151030/306bf393/attachment-0001.html

------------------------------

Message: 2
Date: Fri, 30 Oct 2015 01:20:58 -0400
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] At least fileset to keep lm, tm and
tuning result
To: Masa Taka <grantaka36@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDDwgbsSdF=T29fdGSSqQi+H2exY3E9UetvYq8H5y8YPDw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

to run a system, you will only need the moses.ini with the tuned weights
and all the files that are referenced in there. This is minimally the
language model and the phrase table, but may also include the reordering
table and other fancy feature functions that have models (like OSM).

-phi

On Thu, Oct 29, 2015 at 9:24 PM, Masa Taka <grantaka36@gmail.com> wrote:

> Question:
> What files should I keep "at least" as the language model, translation
> model and tuning result?
>
> For example, on English(f) to Japanese(e) translation, should the
> following files be enough?
> - model/ja/lm.ja.gz
> - model/en-ja/phrase-table.gz
> - tuning/en-ja/moses.ini
>
> Detail:
> When performing Moses SMT by default commands on the same PC, the
> directories model/ and tuning/ are overwritten. I would like to keep the
> minimum fileset to compare different training conditions (-o etc.) and
> translation conditions (-dl, -s etc.)
>
> I know it is preferable all files should be stored, but for this time, it
> is enough to have the same output 'e.txt' when inputting the same 'f.txt.'
>
> moses -f moses.ini < f.txt 1> e.txt 2> output.err
>
> Regards,
> Masataka
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151030/f5816159/attachment-0001.html

------------------------------

Message: 3
Date: Fri, 30 Oct 2015 12:21:28 +0530
From: Sunayana Gawde <sunayanagawde17@gmail.com>
Subject: [Moses-support] Error while training the translation system
To: moses-support@mit.edu
Message-ID:
<CANQTV3QuEZookCjSjYDh8srFSx4j84usFpUTVt4CpE4k_z_htQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi all,

i am trying to develop a baseline translation system with the help of steps
on the site http://www.statmt.org/moses/?n=Moses.Baseline

while training the translation system,

I tried the command
nohup nice ~/mosesdecoder/scripts/training/train-model.perl -root-dir train
-corpus ~/corpus/news-commentary-v8.fr-en.clean -f fr -e en -alignment
grow-diag-final-and -reordering msd-bidirectional-fe -lm
0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8
-external-bin-dir ~/mosesdecoder/tools >& training.out &
and i got the result as expected
But when i tried checking the status by using
tail -f training.out

i got this error:

Executing: mkdir -p /home/sunayana/working/train/corpus
(1.0) selecting factors @ Fri Oct 30 10:56:40 IST 2015
(1.1) running mkcls @ Fri Oct 30 10:56:40 IST 2015
/home/sunayana/mosesdecoder/tools/mkcls -c50 -n2 -p/home/sunayana/corpus/
news-commentary-v8.fr-en.clean.fr
-V/home/sunayana/working/train/corpus/fr.vcb.classes opt
/home/sunayana/working/train/corpus/fr.vcb.classes already in place,
reusing
(1.1) running mkcls @ Fri Oct 30 10:56:40 IST 2015
/home/sunayana/mosesdecoder/tools/mkcls -c50 -n2
-p/home/sunayana/corpus/news-commentary-v8.fr-en.clean.en
-V/home/sunayana/working/train/corpus/en.vcb.classes opt
Executing: /home/sunayana/mosesdecoder/tools/mkcls -c50 -n2
-p/home/sunayana/corpus/news-commentary-v8.fr-en.clean.en
-V/home/sunayana/working/train/corpus/en.vcb.classes opt
ERROR: Execution of: /home/sunayana/mosesdecoder/tools/mkcls -c50 -n2
-p/home/sunayana/corpus/news-commentary-v8.fr-en.clean.en
-V/home/sunayana/working/train/corpus/en.vcb.classes opt
died with signal 11, with coredump



Please help
--
*Thanks & Regards*

Ms. Sunayana R. Gawde.

DCST, Goa University.
* P**leas**e don't print t**his e-mail unles**s you really need to.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151030/cb58c59f/attachment-0001.html

------------------------------

Message: 4
Date: Fri, 30 Oct 2015 15:58:17 +0530
From: Saumitra Yadav <yadav.saumitra07@gmail.com>
Subject: Re: [Moses-support] Error while training the translation
system
To: Sunayana Gawde <sunayanagawde17@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAPeFGXagCwBetC5hMPgqQgDYHCnycrCRv1ExNZTTbO_KTRi9Jg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

hi
whats the output of
/home/sunayana/mosesdecoder/tools/mkcls -c50 -n2
-p/home/sunayana/corpus/news-commentary-v8.fr-en.clean.en
-V/home/sunayana/working/train/corpus/en.vcb.classes opt


Regards,
Saumitra Yadav
Intern, LTRC
IIIT-Hyderabad

On Fri, Oct 30, 2015 at 12:21 PM, Sunayana Gawde <sunayanagawde17@gmail.com>
wrote:

> Hi all,
>
> i am trying to develop a baseline translation system with the help of
> steps on the site http://www.statmt.org/moses/?n=Moses.Baseline
>
> while training the translation system,
>
> I tried the command
> nohup nice ~/mosesdecoder/scripts/training/train-model.perl -root-dir
> train -corpus ~/corpus/news-commentary-v8.fr-en.clean -f fr -e en
> -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm
> 0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8
> -external-bin-dir ~/mosesdecoder/tools >& training.out &
> and i got the result as expected
> But when i tried checking the status by using
> tail -f training.out
>
> i got this error:
>
> Executing: mkdir -p /home/sunayana/working/train/corpus
> (1.0) selecting factors @ Fri Oct 30 10:56:40 IST 2015
> (1.1) running mkcls @ Fri Oct 30 10:56:40 IST 2015
> /home/sunayana/mosesdecoder/tools/mkcls -c50 -n2 -p/home/sunayana/corpus/
> news-commentary-v8.fr-en.clean.fr
> -V/home/sunayana/working/train/corpus/fr.vcb.classes opt
> /home/sunayana/working/train/corpus/fr.vcb.classes already in place,
> reusing
> (1.1) running mkcls @ Fri Oct 30 10:56:40 IST 2015
> /home/sunayana/mosesdecoder/tools/mkcls -c50 -n2
> -p/home/sunayana/corpus/news-commentary-v8.fr-en.clean.en
> -V/home/sunayana/working/train/corpus/en.vcb.classes opt
> Executing: /home/sunayana/mosesdecoder/tools/mkcls -c50 -n2
> -p/home/sunayana/corpus/news-commentary-v8.fr-en.clean.en
> -V/home/sunayana/working/train/corpus/en.vcb.classes opt
> ERROR: Execution of: /home/sunayana/mosesdecoder/tools/mkcls -c50 -n2
> -p/home/sunayana/corpus/news-commentary-v8.fr-en.clean.en
> -V/home/sunayana/working/train/corpus/en.vcb.classes opt
> died with signal 11, with coredump
>
>
>
> Please help
> --
> *Thanks & Regards*
>
> Ms. Sunayana R. Gawde.
>
> DCST, Goa University.
> * P**leas**e don't print t**his e-mail unles**s you really need to.*
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151030/35fe39c7/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 108, Issue 83
**********************************************

Tweet