Moses-support Digest, Vol 108, Issue 79

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: lm integration in mosesdecoder (Hieu Hoang)
2. Re: Moses on SGE clarification (Philipp Koehn)
3. Multiple generation factors in factored model (Anoop (?????))


----------------------------------------------------------------------

Message: 1
Date: Thu, 29 Oct 2015 11:50:22 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] lm integration in mosesdecoder
To: koormoosh <koormoosh@gmail.com>, moses-support
<moses-support@mit.edu>
Message-ID:
<CAEKMkbj3+tK6wkznLkdZ_a6f7hCccD=hmv-ajxGxSbfdyYNAaQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

There's no tutorial but it should be easy to understand.

FactorCollection is the vocabulary for the decoder, there is only ever 1
FactorCollection object, which you can get by calling
FactorCollection &fc = FactorCollection::Instance()
A Factor is an entry in the vocab. You can get the Factor object by doing
const Factor *factor = fc.AddFactor("whatever", false);
The factor pointer is guaranteed to be unique for different strings, and
the same for the same string. Ie.
const Factor *fc1 = fc.AddFactor("whatever", false);
const Factor *fc2 = fc.AddFactor("whatever", false);
fc1 and fc2 will point to the same object.

Hieu Hoang
http://www.hoang.co.uk/hieu

On 29 October 2015 at 11:40, koormoosh <koormoosh@gmail.com> wrote:

> Is there any tutorial to introduce concepts "Factor", "FactorCollection"
> used in the SkeletonLM.cpp, SkeletonLM.h?
>
> On Fri, Oct 23, 2015 at 1:43 AM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
>> i don't think you do. Try it without. Once you've compiled with MYOWN, run
>> ./bin/moses
>> this should give you a list of feature functions. It should also have
>> MYOWN
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 22 October 2015 at 14:59, koormoosh <koormoosh@gmail.com> wrote:
>>
>>> and also add it to the LM/Jamfile and the Jamroot?
>>>
>>> On Fri, Oct 23, 2015 at 12:45 AM, Hieu Hoang <hieuhoang@gmail.com>
>>> wrote:
>>>
>>>> you need to add your lm into
>>>> FF/Factory.cpp
>>>>
>>>> Hieu Hoang
>>>> http://www.hoang.co.uk/hieu
>>>>
>>>> On 22 October 2015 at 14:44, koormoosh <koormoosh@gmail.com> wrote:
>>>>
>>>>> So based on SkeletonLM I can create my own MYOWN_LM.h and MYOWN_LM.cpp
>>>>> and that's it? Is there anywhere else in the mosesdecoder that I need to
>>>>> change?
>>>>>
>>>>> On Fri, Oct 23, 2015 at 12:13 AM, Hieu Hoang <hieuhoang@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> make a copy of
>>>>>> LM/SkeletonLM.*
>>>>>> Do whatever you need to do.
>>>>>>
>>>>>> Look at IRSTLM.*, DALM.*, Ken.* for examples
>>>>>>
>>>>>> Hieu Hoang
>>>>>> http://www.hoang.co.uk/hieu
>>>>>>
>>>>>> On 22 October 2015 at 14:09, koormoosh <koormoosh@gmail.com> wrote:
>>>>>>
>>>>>>> Hello, Are there any guidelines for integrating language models in
>>>>>>> mosesdecoder? Thanks.
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Moses-support mailing list
>>>>>>> Moses-support@mit.edu
>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151029/defa0c98/attachment-0001.html

------------------------------

Message: 2
Date: Thu, 29 Oct 2015 08:26:31 -0400
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] Moses on SGE clarification
To: Vincent Nguyen <vnguyen@neuf.fr>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAAFADDDwKs=z+76cM3sR-X7gp8S-RS87yjzz=MK6U6CnZkgBqg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

these machine names are just there for convenience.

If you want experiment.perl to submit jobs per qsub,
all you have to do is to run experiment.perl with the
additional switch "-cluster".

You can also put the head node's name into the
experiment.machines file, then you do not need to
use the switch anymore.

-phi

On Wed, Oct 28, 2015 at 10:20 AM, Vincent Nguyen <vnguyen@neuf.fr> wrote:

> Hi there,
>
> I need some clarification before screwing up some files.
> I just setup a SGE cluster with a Master + 2 Nodes.
>
> to make it clear let say my cluster name is "default", my master
> headnode is "master", my 2 other nodes are "node1" and "node2"
>
>
> for EMS :
>
> I opened the default experiment.machines file and I see :
>
> cluster: townhill seville hermes lion seville sannox lutzow frontend
> multicore-4: freddie
> multicore-8: tyr thor odin crom
> multicore-16: saxnot vali vili freyja bragi hoenir
> multicore-24: syn hel skaol saga buri loki sif magni
> multicore-32: gna snotra lofn thrud
>
> townhill and others are what ? name machines / nodes ? name of several
> clusters ?
> should I just put "default" or "master node1 node2" ?
>
> multicore-X: should I put machine names here
> if my 3 machines are 8 cores each
> multicore-8: master node1 node2
> right ?
>
>
> then in the config file for EMS:
>
> #generic-parallelizer =
> $moses-script-dir/ems/support/generic-parallelizer.perl
> #generic-parallelizer =
> $moses-script-dir/ems/support/generic-multicore-parallelizer.perl
>
> which one should take if my nodes are multicore ? still the first one ?
>
>
> ### cluster settings (if run on a cluster machine)
> # number of jobs to be submitted in parallel
> #
> #jobs = 10
> should I count approx 1 job per core on the total cores of my 3 machines ?
>
> # arguments to qsub when scheduling a job
> #qsub-settings = ""
> can this stay empty ?
>
> # project for priviledges and usage accounting
> #qsub-project = iccs_smt
> standard value ?
>
> # memory and time
> #qsub-memory = 4
> #qsub-hours = 48
> 4 what ? GB ?
>
> ### multi-core settings
> # when the generic parallelizer is used, the number of cores
> # specified here
> cores = 4
> is this ignored if generic-parallelizer.perl is chosen ?
>
>
> is there a way to put more load on one specific node ?
>
> Many thanks,
> V.
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151029/c3da02f6/attachment-0001.html

------------------------------

Message: 3
Date: Thu, 29 Oct 2015 18:42:35 +0530
From: Anoop (?????) <anoop.kunchukuttan@gmail.com>
Subject: [Moses-support] Multiple generation factors in factored model
To: moses-support <moses-support@mit.edu>
Message-ID:
<CADXxMYeRedFgthxUJHn1oQJkLmMG8an4L-veS4XLO5z_nTXPtg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

I have a training corpus with multiple factors on the target side. I
experimented with various factor configurations - one for generation of
each target factor from the surface form and then use an LM over the factor
as a feature. The surface form to factor mappings are fairly deterministic,
so the LM over factors is where I hope to see benefits. Indeed I do obtain
significant improvements in output with each individual factor over a PBSMT
system. However, when I put together multiple factors along with multiple
language models, the results actually don't as much improvement as using
some of the individual factors. The performance generally better the
baseline PBSMT though in most cases.

Do you have some suggestions regarding why this would be so, and if this
could be rectified?
I have attached the moses.ini for the factored system with three generation
factors.

Regards
Anoop.

--
I claim to be a simple individual liable to err like any other fellow
mortal. I own, however, that I have humility enough to confess my errors
and to retrace my steps.

http://flightsofthought.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151029/c2b90cfd/attachment.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: moses.ini
Type: application/octet-stream
Size: 1674 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20151029/c2b90cfd/attachment.obj

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 108, Issue 79
**********************************************

Tweet