Moses-support Digest, Vol 97, Issue 68

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Phrase-based moses segfaults on [...] input tokens
(Marcin Junczys-Dowmunt)
2. Re: Error in EMS for factor based SMT (Hieu Hoang)
3. Re: Phrase-based moses segfaults on [...] input tokens
(Marcin Junczys-Dowmunt)


----------------------------------------------------------------------

Message: 1
Date: Sat, 22 Nov 2014 11:42:42 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Phrase-based moses segfaults on [...]
input tokens
To: moses-support@mit.edu
Message-ID: <547068A2.1080707@amu.edu.pl>
Content-Type: text/plain; charset="windows-1252"

er, with normal moses, not with moses_chart.

W dniu 22.11.2014 o 11:39, Marcin Junczys-Dowmunt pisze:
> Nope, we at WIPO had no problems with tokens like that with a moses
> version from two years ago and moses_chart. We merged with master
> recently and stuff started happening. And we have LOTS of tokens like
> that, we would have noticed earlier :)
>
> W dniu 22.11.2014 o 11:36, Hieu Hoang pisze:
>> i think it's been like that since the introduction of syntax models
>> into Moses more than 5 years ago. It's not safe to have non-terms in
>> the phrase-based model, there might be something new that triggered
>> the segfault.
>>
>> On 22 November 2014 10:32, Marcin Junczys-Dowmunt <junczys@amu.edu.pl
>> <mailto:junczys@amu.edu.pl>> wrote:
>>
>> OK then. Is this new? We did not have this problem some time ago.
>>
>> W dniu 22.11.2014 o 11:31, Hieu Hoang pisze:
>>> escape your input properly. It's 1 pipeline for all models
>>>
>>>
>>> On 22 November 2014 10:27, Marcin Junczys-Dowmunt
>>> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>>>
>>> Hi,
>>> phrase-based moses segfaults when it encounters tokens of
>>> the form
>>> [...], e.g. [7] or [test], I understand that for hiero or
>>> syntax-based
>>> models those are non-terminals and they should rather by
>>> escaped.
>>> However, for phrase-based this should not be needed. Is that
>>> something
>>> you would bother to fix or rather say "we don't care, escape
>>> your input
>>> properly"?.
>>>
>>> Cheers,
>>> Marcin
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>
>>>
>>> --
>>> Hieu Hoang
>>> Research Associate
>>> University of Edinburgh
>>> http://www.hoang.co.uk/hieu
>>>
>>
>>
>>
>>
>>
>>
>> --
>> Hieu Hoang
>> Research Associate
>> University of Edinburgh
>> http://www.hoang.co.uk/hieu
>>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141122/bf2d7a07/attachment-0001.htm

------------------------------

Message: 2
Date: Sat, 22 Nov 2014 10:45:02 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Error in EMS for factor based SMT
To: Rajen Chatterjee <rajen.k.chatterjee@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAEKMkbjuRTHASWLPd6fWYT9dG2UYqYZvgBQhSjA4KAdAC2-Uhg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

what does your source corpus file look like?
(a) word1 word2 word3
or
(b) word1|word1-STEM|word1-POS word2|word2-STEM|word2-POS
word3|word3-STEM|word3-POS

the EMS expects (a), it then requires the scripts to create (b). If you
already have (b), then let me know. You have to set some variable to bypass
the scripts


On 21 November 2014 13:08, Rajen Chatterjee <rajen.k.chatterjee@gmail.com>
wrote:

> Hi Hieu Hoang,
>
> I am using EMS for factored based SMT.
> I already have a factored corpus and I have used
> factorized-stem=/path/to/corpus, but still I am getting this error "ERROR:
> you need to define INPUT-FACTOR:lemma:factor-script"
>
> *Training Option:*
> input-factors = word lemma pos
> output-factors = word
> alignment-factors = "lemma -> word"
> translation-factors = "word -> word"
> reordering-factors = "word -> word"
> #generation-factors = "word -> pos"
> decoding-steps = "t0"
>
> *Factor Definition:*
> [INPUT-FACTOR]
> temp-dir = $working-dir/training/factor
>
> #[OUTPUT-FACTOR]
> #temp-dir = $working-dir/training/factor
>
> As I am already using factorized-stem then why it is asking for a
> factor-script ?
>
>
> --
> -Regards,
> Rajen Chatterjee.
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141122/4316a3ee/attachment-0001.htm

------------------------------

Message: 3
Date: Sat, 22 Nov 2014 11:48:48 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Phrase-based moses segfaults on [...]
input tokens
To: moses-support@mit.edu
Message-ID: <54706A10.3020909@amu.edu.pl>
Content-Type: text/plain; charset="windows-1252"

Is there a canonical way to ask Moses during runtime "Am I
phrase-based?/Am I chart?" ? I am going to hack around that in our
branch. That's cheaper than changing the whole training pipeline.

W dniu 22.11.2014 o 11:42, Marcin Junczys-Dowmunt pisze:
> er, with normal moses, not with moses_chart.
>
> W dniu 22.11.2014 o 11:39, Marcin Junczys-Dowmunt pisze:
>> Nope, we at WIPO had no problems with tokens like that with a moses
>> version from two years ago and moses_chart. We merged with master
>> recently and stuff started happening. And we have LOTS of tokens like
>> that, we would have noticed earlier :)
>>
>> W dniu 22.11.2014 o 11:36, Hieu Hoang pisze:
>>> i think it's been like that since the introduction of syntax models
>>> into Moses more than 5 years ago. It's not safe to have non-terms in
>>> the phrase-based model, there might be something new that triggered
>>> the segfault.
>>>
>>> On 22 November 2014 10:32, Marcin Junczys-Dowmunt
>>> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>>>
>>> OK then. Is this new? We did not have this problem some time ago.
>>>
>>> W dniu 22.11.2014 o 11:31, Hieu Hoang pisze:
>>>> escape your input properly. It's 1 pipeline for all models
>>>>
>>>>
>>>> On 22 November 2014 10:27, Marcin Junczys-Dowmunt
>>>> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>>>>
>>>> Hi,
>>>> phrase-based moses segfaults when it encounters tokens of
>>>> the form
>>>> [...], e.g. [7] or [test], I understand that for hiero or
>>>> syntax-based
>>>> models those are non-terminals and they should rather by
>>>> escaped.
>>>> However, for phrase-based this should not be needed. Is
>>>> that something
>>>> you would bother to fix or rather say "we don't care,
>>>> escape your input
>>>> properly"?.
>>>>
>>>> Cheers,
>>>> Marcin
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Hieu Hoang
>>>> Research Associate
>>>> University of Edinburgh
>>>> http://www.hoang.co.uk/hieu
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Hieu Hoang
>>> Research Associate
>>> University of Edinburgh
>>> http://www.hoang.co.uk/hieu
>>>
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141122/e3d3b72d/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 97, Issue 68
*********************************************

Tweet