Moses-support Digest, Vol 112, Issue 25

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Dumb question: Environment variables in Moses config
(Hieu Hoang)
2. Re: Access to set of possible target words from within
feature function (Hieu Hoang)
3. Re: Access to set of possible target words from within
feature function (Marcin Junczys-Dowmunt)
4. Re: same language for input and output (Hieu Hoang)
5. Re: Segmentation Fault (Jasneet Sabharwal)


----------------------------------------------------------------------

Message: 1
Date: Fri, 12 Feb 2016 22:45:20 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Dumb question: Environment variables in
Moses config
To: Jeremy Gwinnup <jeremy@gwinnup.org>, moses-support@mit.edu
Message-ID: <56BE6080.1010600@gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed

hmm. doubt it. The moses.ini file is parsed in
moses/Parameter.cpp
there's no calls to any OS functions that expands environment variables.
It would be useful

On 12/02/16 14:33, Jeremy Gwinnup wrote:
> Hi,
>
> Dumb question - can I reference environment variables in a moses.ini ?
>
> Thanks!
> -Jeremy
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>

--
Hieu Hoang
http://www.hoang.co.uk/hieu



------------------------------

Message: 2
Date: Fri, 12 Feb 2016 22:47:40 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Access to set of possible target words
from within feature function
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <56BE610C.4060507@gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed

i think this happens just before expansion, once all the translation
options have been collected

On 12/02/16 22:36, Marcin Junczys-Dowmunt wrote:
> But this happens during hypothesis expansion? I think I need it before.
>
> I have a classifier for which I want to restrict softmaxing over all
> available target words for the complete sentence instead over all
> known vocabulary items.
>
> I will take a look at your second suggestion.
>
> On 12.02.2016 23:32, Hieu Hoang wrote:
>> sounds useful.There's
>> FeatureFunction::EvaluateTranslationOptionListWithSourceContext()
>> which is called for for every subphrase. It passes the feature
>> function a list of translation option for that subphrase. The method
>> is called in
>> TranslationOptionCollection::EvaluateTranslationOptionListWithSourceContext
>>
>>
>> Or you add yet another Evaluate() method that passes the feature
>> function the InputPathList, which really has every translation
>> options in 1 go
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 12 February 2016 at 22:19, Marcin Junczys-Dowmunt
>> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>>
>> Hi everybody,
>> Is there a nice way to access the complete set of translation
>> options
>> for a sentence from within a feature function? I just need the
>> bag of
>> possible words before any processing. Let's say from within
>> "EmptyHypothesisState", there is InputType, but I cannot really
>> find any
>> hook to get that easily.
>>
>> Thanks,
>> Marcin
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
>

--
Hieu Hoang
http://www.hoang.co.uk/hieu



------------------------------

Message: 3
Date: Fri, 12 Feb 2016 23:54:26 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Access to set of possible target words
from within feature function
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <56BE62A2.3020605@amu.edu.pl>
Content-Type: text/plain; charset=UTF-8; format=flowed

Indeed. Thank you very much, dear sir.

On 12.02.2016 23:47, Hieu Hoang wrote:
> i think this happens just before expansion, once all the translation
> options have been collected
>
> On 12/02/16 22:36, Marcin Junczys-Dowmunt wrote:
>> But this happens during hypothesis expansion? I think I need it before.
>>
>> I have a classifier for which I want to restrict softmaxing over all
>> available target words for the complete sentence instead over all
>> known vocabulary items.
>>
>> I will take a look at your second suggestion.
>>
>> On 12.02.2016 23:32, Hieu Hoang wrote:
>>> sounds useful.There's
>>> FeatureFunction::EvaluateTranslationOptionListWithSourceContext()
>>> which is called for for every subphrase. It passes the feature
>>> function a list of translation option for that subphrase. The method
>>> is called in
>>> TranslationOptionCollection::EvaluateTranslationOptionListWithSourceContext
>>>
>>>
>>> Or you add yet another Evaluate() method that passes the feature
>>> function the InputPathList, which really has every translation
>>> options in 1 go
>>>
>>> Hieu Hoang
>>> http://www.hoang.co.uk/hieu
>>>
>>> On 12 February 2016 at 22:19, Marcin Junczys-Dowmunt
>>> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>>>
>>> Hi everybody,
>>> Is there a nice way to access the complete set of translation
>>> options
>>> for a sentence from within a feature function? I just need the
>>> bag of
>>> possible words before any processing. Let's say from within
>>> "EmptyHypothesisState", there is InputType, but I cannot really
>>> find any
>>> hook to get that easily.
>>>
>>> Thanks,
>>> Marcin
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>>
>



------------------------------

Message: 4
Date: Fri, 12 Feb 2016 23:05:23 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] same language for input and output
To: Vincent Nguyen <vnguyen@neuf.fr>, moses-support
<moses-support@mit.edu>
Message-ID: <56BE6533.4090300@gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed

oh, you mean English -> English? yes, of course. eg. the recaser
'translates'
lowercased english -> cased English

On 12/02/16 15:56, Vincent Nguyen wrote:
> Hey Hieu,
>
> This may sound crazy but do you know if someone has ever run a training
> and testing with same language for the pair ?
>
> I read a paper on summarization mentioning Moses+
>
> what is this ?
>
> Cheers,
>
> Vincent
>

--
Hieu Hoang
http://www.hoang.co.uk/hieu



------------------------------

Message: 5
Date: Fri, 12 Feb 2016 15:34:04 -0800
From: Jasneet Sabharwal <jasneet.sabharwal@sfu.ca>
Subject: Re: [Moses-support] Segmentation Fault
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <E72C7620-09AA-4407-910B-434D55D8107B@sfu.ca>
Content-Type: text/plain; charset="windows-1252"

Thanks Hieu for your reply.

Is it possible to do a verbose output of what?s happening, so that I can identify when it?s going out of memory? I?m only running it for 1928 sentences. I have almost 170gb of free memory and additional 400gb memory in buffer.

Thanks,
Jasneet

> On Feb 12, 2016, at 2:36 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
> looks like it's run out of memory.
>
> On 11/02/16 23:23, Jasneet Sabharwal wrote:
>> Hi,
>>
>> I was adding a new feature function in Moses ( <https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp>https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp <https://github.com/KonceptGeek/mosesdecoder/blob/master/moses/FF/CoarseBiLM.cpp>). It works fine when I test it for 1-2 sentences, but when I?m trying to tune my parameters, I?m getting segmentation faults or sometimes it is bad_alloc. Following was one of the commands that was executed during the tuning process which caused the Segmentation Fault or bad_alloc:
>>
>> moses -threads 40 -v 0 -config filtered/moses.ini -weight-overwrite 'CoarseLM100= 0.075758 LM0= 0.075758 CoarseBiLMNotClustered= 0.075758 WordPenalty0= -0.151515 PhrasePenalty0= 0.030303 CoarseBiLMClustered= 0.075758 TranslationModel0= 0.030303 0.030303 0.030303 0.030303 Distortion0= 0.045455 CoarseLM1600= 0.075758 LexicalReordering0= 0.045455 0.045455 0.045455 0.045455 0.045455 0.045455' -n-best-list run1.best100.out 100 distinct -input-file tune.word.lc.cn <http://tune.word.lc.cn/>
>>
>> The log is enclosed in this email.
>>
>> Any pointers would be very useful.
>>
>> Thanks,
>> Jasneet
>>
>>
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>
>
> --
> Hieu Hoang
> http://www.hoang.co.uk/hieu <http://www.hoang.co.uk/hieu>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160212/7efb0d68/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 112, Issue 25
**********************************************

Tweet