Moses-support Digest, Vol 97, Issue 70

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Phrase-based moses segfaults on [...] input tokens
(Hieu Hoang)
2. Re: Phrase-based moses segfaults on [...] input tokens
(Marcin Junczys-Dowmunt)


----------------------------------------------------------------------

Message: 1
Date: Sat, 22 Nov 2014 11:13:20 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Phrase-based moses segfaults on [...]
input tokens
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbgWBbaLu6DPb94TUqsVSOUhJaMoYmGv26oQmV0iPqVyxQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

it may be you can change
Phrase::CreateFromString
so that it never creates non-terminals

On 22 November 2014 10:48, Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
wrote:

> Is there a canonical way to ask Moses during runtime "Am I
> phrase-based?/Am I chart?" ? I am going to hack around that in our branch.
> That's cheaper than changing the whole training pipeline.
>
> W dniu 22.11.2014 o 11:42, Marcin Junczys-Dowmunt pisze:
>
> er, with normal moses, not with moses_chart.
>
> W dniu 22.11.2014 o 11:39, Marcin Junczys-Dowmunt pisze:
>
> Nope, we at WIPO had no problems with tokens like that with a moses
> version from two years ago and moses_chart. We merged with master recently
> and stuff started happening. And we have LOTS of tokens like that, we would
> have noticed earlier :)
>
> W dniu 22.11.2014 o 11:36, Hieu Hoang pisze:
>
> i think it's been like that since the introduction of syntax models into
> Moses more than 5 years ago. It's not safe to have non-terms in the
> phrase-based model, there might be something new that triggered the
> segfault.
>
> On 22 November 2014 10:32, Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
> wrote:
>
>> OK then. Is this new? We did not have this problem some time ago.
>>
>> W dniu 22.11.2014 o 11:31, Hieu Hoang pisze:
>>
>> escape your input properly. It's 1 pipeline for all models
>>
>>
>> On 22 November 2014 10:27, Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
>> wrote:
>>
>>> Hi,
>>> phrase-based moses segfaults when it encounters tokens of the form
>>> [...], e.g. [7] or [test], I understand that for hiero or syntax-based
>>> models those are non-terminals and they should rather by escaped.
>>> However, for phrase-based this should not be needed. Is that something
>>> you would bother to fix or rather say "we don't care, escape your input
>>> properly"?.
>>>
>>> Cheers,
>>> Marcin
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>
>>
>>
>> --
>> Hieu Hoang
>> Research Associate
>> University of Edinburgh
>> http://www.hoang.co.uk/hieu
>>
>>
>>
>>
>>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141122/394b043d/attachment-0001.htm

------------------------------

Message: 2
Date: Sat, 22 Nov 2014 13:10:37 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Phrase-based moses segfaults on [...]
input tokens
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <54707D3D.4020302@amu.edu.pl>
Content-Type: text/plain; charset="utf-8"

Done that. I would however prefer to switch that on/off depending which
version is calling the code. Otherwise this will probably come back at
one moment and bite me in my behind when I have long forgotten I put it
there.

W dniu 22.11.2014 o 12:13, Hieu Hoang pisze:
> it may be you can change
> Phrase::CreateFromString
> so that it never creates non-terminals
>
> On 22 November 2014 10:48, Marcin Junczys-Dowmunt <junczys@amu.edu.pl
> <mailto:junczys@amu.edu.pl>> wrote:
>
> Is there a canonical way to ask Moses during runtime "Am I
> phrase-based?/Am I chart?" ? I am going to hack around that in our
> branch. That's cheaper than changing the whole training pipeline.
>
> W dniu 22.11.2014 o 11:42, Marcin Junczys-Dowmunt pisze:
>> er, with normal moses, not with moses_chart.
>>
>> W dniu 22.11.2014 o 11:39, Marcin Junczys-Dowmunt pisze:
>>> Nope, we at WIPO had no problems with tokens like that with a
>>> moses version from two years ago and moses_chart. We merged with
>>> master recently and stuff started happening. And we have LOTS of
>>> tokens like that, we would have noticed earlier :)
>>>
>>> W dniu 22.11.2014 o 11:36, Hieu Hoang pisze:
>>>> i think it's been like that since the introduction of syntax
>>>> models into Moses more than 5 years ago. It's not safe to have
>>>> non-terms in the phrase-based model, there might be something
>>>> new that triggered the segfault.
>>>>
>>>> On 22 November 2014 10:32, Marcin Junczys-Dowmunt
>>>> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>>>>
>>>> OK then. Is this new? We did not have this problem some
>>>> time ago.
>>>>
>>>> W dniu 22.11.2014 o 11:31, Hieu Hoang pisze:
>>>>> escape your input properly. It's 1 pipeline for all models
>>>>>
>>>>>
>>>>> On 22 November 2014 10:27, Marcin Junczys-Dowmunt
>>>>> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>>>>>
>>>>> Hi,
>>>>> phrase-based moses segfaults when it encounters tokens
>>>>> of the form
>>>>> [...], e.g. [7] or [test], I understand that for hiero
>>>>> or syntax-based
>>>>> models those are non-terminals and they should rather
>>>>> by escaped.
>>>>> However, for phrase-based this should not be needed.
>>>>> Is that something
>>>>> you would bother to fix or rather say "we don't care,
>>>>> escape your input
>>>>> properly"?.
>>>>>
>>>>> Cheers,
>>>>> Marcin
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Hieu Hoang
>>>>> Research Associate
>>>>> University of Edinburgh
>>>>> http://www.hoang.co.uk/hieu
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Hieu Hoang
>>>> Research Associate
>>>> University of Edinburgh
>>>> http://www.hoang.co.uk/hieu
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141122/bd615330/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 97, Issue 70
*********************************************

Tweet