Moses-support Digest, Vol 122, Issue 10

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Run fast_align with multithreading (Lane Schwartz)
2. Re: Run fast_align with multithreading (Michael Denkowski)
3. Re: Run fast_align with multithreading (Lane Schwartz)
4. Re: Tuning for factored phrase based systems (Angli Liu)


----------------------------------------------------------------------

Message: 1
Date: Tue, 6 Dec 2016 11:06:53 -0600
From: Lane Schwartz <dowobeha@gmail.com>
Subject: [Moses-support] Run fast_align with multithreading
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZkVmFbUym-rqE+uC+SK5--EsY0PLRySVqtTJmXxPFq6vw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

Looking at the change log in the fast_align github repo, it appears that
there is now multithreading in fast_align.

This should be simple, but I haven't been able to find it documented
anywhere. Can anyone verify that fast_align does indeed support
multithreading, and if it does, how do you enable it?

Thanks,
Lane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161206/ba93b0d8/attachment-0001.html

------------------------------

Message: 2
Date: Tue, 6 Dec 2016 12:18:49 -0500
From: Michael Denkowski <michael.j.denkowski@gmail.com>
Subject: Re: [Moses-support] Run fast_align with multithreading
To: Lane Schwartz <dowobeha@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CA+-GegK5YUAk4ht0Kqb6GbOGGv+rY3tMVZ+eht_cHs4v4hD6vg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi Lane,

I had to track this one down too. The multithreading uses OpenMP, so if
you install OpenMP on your machine and build fast_align, multithreading
should be baked in. By default, it should use all available CPUs. You can
override this by setting the OMP_NUM_THREADS environment variable (export
OMP_NUM_THREADS=8).

Best,
Michael

On Tue, Dec 6, 2016 at 12:06 PM, Lane Schwartz <dowobeha@gmail.com> wrote:

> Hi,
>
> Looking at the change log in the fast_align github repo, it appears that
> there is now multithreading in fast_align.
>
> This should be simple, but I haven't been able to find it documented
> anywhere. Can anyone verify that fast_align does indeed support
> multithreading, and if it does, how do you enable it?
>
> Thanks,
> Lane
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161206/300992d2/attachment-0001.html

------------------------------

Message: 3
Date: Tue, 6 Dec 2016 11:23:32 -0600
From: Lane Schwartz <dowobeha@gmail.com>
Subject: Re: [Moses-support] Run fast_align with multithreading
To: Michael Denkowski <michael.j.denkowski@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZ=5t_8T0TT-gcYhX0fLcBAAp_0wn6Wr0fhNGry9+XcrhA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

OK. Thanks!

On Tue, Dec 6, 2016 at 11:18 AM, Michael Denkowski <
michael.j.denkowski@gmail.com> wrote:

> Hi Lane,
>
> I had to track this one down too. The multithreading uses OpenMP, so if
> you install OpenMP on your machine and build fast_align, multithreading
> should be baked in. By default, it should use all available CPUs. You can
> override this by setting the OMP_NUM_THREADS environment variable (export
> OMP_NUM_THREADS=8).
>
> Best,
> Michael
>
> On Tue, Dec 6, 2016 at 12:06 PM, Lane Schwartz <dowobeha@gmail.com> wrote:
>
>> Hi,
>>
>> Looking at the change log in the fast_align github repo, it appears that
>> there is now multithreading in fast_align.
>>
>> This should be simple, but I haven't been able to find it documented
>> anywhere. Can anyone verify that fast_align does indeed support
>> multithreading, and if it does, how do you enable it?
>>
>> Thanks,
>> Lane
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>


--
When a place gets crowded enough to require ID's, social collapse is not
far away. It is time to go elsewhere. The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161206/f386ea1e/attachment-0001.html

------------------------------

Message: 4
Date: Tue, 06 Dec 2016 17:39:22 +0000
From: Angli Liu <anglil@cs.washington.edu>
Subject: Re: [Moses-support] Tuning for factored phrase based systems
To: Hieu Hoang <hieuhoang@gmail.com>, Sa?o Kuntaric
<saso.kuntaric@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAJXA_eMAg6c7U5SigyqVEbx+dz3LrDRXPHEbu6Uckf4YOiwhKA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Thank you!

On Tue, Dec 6, 2016 at 12:55 AM Sa?o Kuntaric <saso.kuntaric@gmail.com>
wrote:

> Hi Angli,
>
> Here is an excerpt of Hieu's answers regarding this topic when I was doing
> research in factored models, might be of some help:
>
> On 30/06/2016 21:44, Sa?o Kuntaric wrote:
>
> Hi all,
>
> I would like to ask one more question. When you say that my reference only
> has the surface form, are you talking about the "tuning corpus", which in
> the case of my command
>
> ~/mosesdecoder/scripts/training/mert-moses.pl
> ~/working/IT_corpus/TMX/txt/factored_corpus/singles/tuning_corpus.tagged.clean.en
> ~/working/IT_corpus/TMX/txt/factored_corpus/singles/
> tuning_corpus.tagged.clean.sl ~/mosesdecoder/bin/moses
> ~/working/IT_corpus/TMX/txt/factored_corpus/singles/test/model/moses.ini
> --mertdir ~/mosesdecoder/bin/ --decoder-flags="-threads all"
>
> are tuning_corpus.tagged.clean.en and tuning_corpus.tagged.clean.sl? Can
> tuning be done with files that only contains surface forms?
>
> it's usual that the reference tuning data does not have factors, even if
> there are factors in the phrase table. After all, you don't care if the
> output surface form is correct but the other factors are wrong.
>
> Will the results be compatible with tuning done with a factored tuning
> corpus?
>
> yes
>
> Best regards,
>
> Sa?o
>
> 2016-12-04 1:37 GMT+01:00 Hieu Hoang <hieuhoang@gmail.com>:
>
>
>
> Hieu
> Sent while bumping into things
>
> On 1 Dec 2016 07:01, "Angli Liu" <anglil@cs.washington.edu> wrote:
>
> Hi, what's the major difference between the tuning process for a factored
> phrase based system (i.e., surface+pos data) and a simple baseline phrase
> based system?
>
>
> Nothing, the tuning just optimise weights for feature functions.
>
> If you decompose your translation so that it has multiple phrase tables
> and generation models, then they are just extra feature functions with
> weights to be tuned
>
> Do I need to organize the dev set the same way as the training set (i.e.,
> surface|pos)?
>
> Yes
>
> Is there a tutorial on the moses website on this topic?
>
> Maybe this
> http://www.statmt.org/moses/?n=FactoredTraining.FactoredTraining
>
>
> Thanks!
>
> -Angli
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> lp,
>
> Sa?o
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161206/f0a40983/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 122, Issue 10
**********************************************

Tweet