Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Polysynthetic languages? (Michael Joyner)
2. Re: Segmentation fault on hierarchical model with moses in
server mode (Barry Haddow)
----------------------------------------------------------------------
Message: 1
Date: Mon, 1 Feb 2016 08:12:26 -0500
From: Michael Joyner <mjoyner@vbservices.net>
Subject: [Moses-support] Polysynthetic languages?
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAAdxTGg7XutdtPZZe32LzfVcSEX-sk4h+eWaEYOEA_utMf0A=w@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
I am trying to use Moses with Cherokee using the New Testament and Genesis
as primary corpus. I am feeding it the WEB, BBE as source English texts at
the moment.
As Cherokee uses bound pronouns and no articles and has almost nil
preposition analogues, (these features are mostly verb infixes), is there a
technique for corpus adjustment that can be done to improve the phrase
mapping between Cherokee and English?
I am currently doing Cherokee => English.
Thanks, Mike
--
WEB: World English Bible (Public Domain)
BBE: Basic English Bible (Public Domain)
- Learn to the Cherokee language: http://jalagigawoni.gnomio.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160201/97460a5d/attachment-0001.html
------------------------------
Message: 2
Date: Mon, 1 Feb 2016 13:16:14 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Segmentation fault on hierarchical model
with moses in server mode
To: Martin Baumg?rtner <martin.baumgaertner@star-group.net>, Matthias
Huck <mhuck@inf.ed.ac.uk>, Hieu Hoang <hieuhoang@gmail.com>, Ulrich
Germann <ugermann@inf.ed.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <56AF5A9E.1080500@staffmail.ed.ac.uk>
Content-Type: text/plain; charset="utf-8"
Hi Martin
In your original mail, you passed different inputs to the server and
command-line version. The following are extracted from your debug:
command-line:
Translating: <s> das ist ein haus </s>
server:
Translating: <s> dies ist ein haus . </s>
(Although you specify the command-line in the first case as "echo 'dies
ist ein haus' | moses -f string-to-tree/moses.ini", which contradicts
the debug. I assume you spliced together different runs.)
Could you show an example where you pass the *same* string to the two
different Moses versions and get different outputs?
cheers - Barry
On 01/02/16 12:47, Martin Baumg?rtner wrote:
> With the current fix the crash is gone, but there is still an
> unexpected difference between server mode and command line moses: the
> first produces an empty string, now, whereas the latter produces "this
> is a house". I tried another engine (english->chinese) and got the
> same behaviour - empty string vs. correct translation.
>
> Cheers,
> Martin
>
> Am 29.01.2016 um 22:37 schrieb Matthias Huck:
>> On Fri, 2016-01-29 at 21:26 +0000, Hieu Hoang wrote:
>>> The decoder should handle no translation without falling over. But
>>> yes, the model is too toy
>> Normally the decoder would always produce some translation. (The translation could be an empty sentence, of course.) If it's misconfigured, it should tell you about it. But maybe not with a segmentation fault. :-)
>>
>>
>>> On 29 Jan 2016 9:15 pm, "Matthias Huck"<mhuck@inf.ed.ac.uk> wrote:
>>>> Hi,
>>>>
>>>> It seems to me that this toy string-to-tree setup is either
>>>> outdated,
>>>> or it always had issues. It should be replaced.
>>>>
>>>> Under real-world conditions, the decoder should always be able to
>>>> produce some hypothesis. We would therefore usually extract a whole
>>>> set
>>>> of glue rules. And we would typically also add an [unknown-lhs]
>>>> section
>>>> to the moses.ini that would tell the decoder which left-hand side
>>>> non
>>>> -terminal labels to use for out-of-vocabulary words. To my
>>>> knowledge,
>>>> these two techniques are crucial for being able to parse any input
>>>> sentence provided to the chart decoder in syntax-based translation.
>>>>
>>>> So, in my opinion, the problem is most likely neither the server
>>>> implementation nor the syntax-based decoder, but a problematic
>>>> setup.
>>>> I would consider it okay for the server to crash (or at least print
>>>> a
>>>> warning) under such circumstances. You don't want it to silently
>>>> not
>>>> translate complete sentences.
>>>>
>>>> (I must admit that I didn't look into it in too much detail, but it
>>>> sho
>>>> uld be easy to confirm.)
>>>>
>>>> Cheers,
>>>> Matthias
>>>>
>>>>
>>>> On Fri, 2016-01-29 at 20:28 +0000, Barry Haddow wrote:
>>>>> Hi All
>>>>>
>>>>> I think I see what happened now.
>>>>>
>>>>> When you give the input "dies ist ein haus" to the sample model,
>>>> the
>>>>> "dies" is unknown, and there is no translation. The server did
>>>> not check
>>>>> for this condition, and got a seg fault. I have added a check, so
>>>> if you
>>>>> pull and try again it should not crash.
>>>>>
>>>>> In the log pasted by Martin, he passed "das ist ein haus" to
>>>>> command-line Moses, which works, and gives a translation.
>>>>>
>>>>> I think ideally the sample models should handle unknown words,
>>>> and give
>>>>> a translation. Maybe adding a glue rule would be sufficient?
>>>>>
>>>>> cheers - Barry
>>>>>
>>>>> On 29/01/16 11:13, Barry Haddow wrote:
>>>>>> Hi
>>>>>>
>>>>>> When I run command-line Moses, I get the output below - i.e. no
>>>> best
>>>>>> translation. The server crashes for me since it does not check
>>>> for the
>>>>>> null pointer, but the command-line version does.
>>>>>>
>>>>>> I think there should be a translation for this example.
>>>>>>
>>>>>> cheers - Barry
>>>>>>
>>>>>> [gna]bhaddow: echo 'dies ist ein haus' | ~/moses.new/bin/moses
>>>> -f
>>>>>> string-to-tree/moses.ini
>>>>>> Defined parameters (per moses.ini or switch):
>>>>>> config: string-to-tree/moses.ini
>>>>>> cube-pruning-pop-limit: 1000
>>>>>> feature: KENLM name=LM factor=0 order=3 num
>>>> -features=1
>>>>>> path=lm/europarl.srilm.gz WordPenalty UnknownWordPenalty
>>>>>> PhraseDictionaryMemory input-factor=0 output-factor=0
>>>>>> path=string-to-tree/rule-table num-features=1 table-limit=20
>>>>>> input-factors: 0
>>>>>> inputtype: 3
>>>>>> mapping: 0 T 0
>>>>>> max-chart-span: 20 1000
>>>>>> non-terminals: X S
>>>>>> search-algorithm: 3
>>>>>> translation-details: translation-details.log
>>>>>> weight: WordPenalty0= 0 LM= 0.5
>>>> PhraseDictionaryMemory0= 0.5
>>>>>> line=KENLM name=LM factor=0 order=3 num-features=1
>>>> path=lm/europarl.srilm.gz
>>>>>> Loading the LM will be faster if you build a binary file.
>>>>>> Reading lm/europarl.srilm.gz
>>>>>> ----5---10---15---20---25---30---35---40---45---50---55---60--
>>>> -65---70---75---80---85---90---95--100
>>>>>> **The ARPA file is missing . Substituting log10 probability
>>>> -100.000.
>>>> *******************************************************************
>>>> *******************************
>>>>>> FeatureFunction: LM start: 0 end: 0
>>>>>> line=WordPenalty
>>>>>> FeatureFunction: WordPenalty0 start: 1 end: 1
>>>>>> line=UnknownWordPenalty
>>>>>> FeatureFunction: UnknownWordPenalty0 start: 2 end: 2
>>>>>> line=PhraseDictionaryMemory input-factor=0 output-factor=0
>>>>>> path=string-to-tree/rule-table num-features=1 table-limit=20
>>>>>> FeatureFunction: PhraseDictionaryMemory0 start: 3 end: 3
>>>>>> Loading LM
>>>>>> Loading WordPenalty0
>>>>>> Loading UnknownWordPenalty0
>>>>>> Loading PhraseDictionaryMemory0
>>>>>> Start loading text phrase table. Moses format : [3.038] seconds
>>>>>> Reading string-to-tree/rule-table
>>>>>> ----5---10---15---20---25---30---35---40---45---50---55---60--
>>>> -65---70---75---80---85---90---95--100
>>>> *******************************************************************
>>>> *********************************
>>>>>> max-chart-span: 20
>>>>>> Created input-output object : [3.041] seconds
>>>>>> Line 0: Initialize search took 0.000 seconds total
>>>>>> Translating: dies ist ein haus ||| [0,0]=X (1) [0,1]=X (1)
>>>>>> [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1)
>>>> [1,2]=X (1)
>>>>>> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1)
>>>> [2,4]=X (1)
>>>>>> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1)
>>>> [4,5]=X (1)
>>>>>> [5,5]=X (1)
>>>>>>
>>>>>> 0 1 2 3 4 5
>>>>>> 0 1 2 2 1 0
>>>>>> 0 0 0 2 0
>>>>>> 0 0 4 0
>>>>>> 0 0 0
>>>>>> 0 0
>>>>>> 0
>>>>>> Line 0: Additional reporting took 0.000 seconds total
>>>>>> Line 0: Translation took 0.002 seconds total
>>>>>> Translation took 0.000 seconds
>>>>>> Name:moses VmPeak:74024 kB VmRSS:11084 kB RSSMax:36832 kB
>>>>>> user:2.972 sys:0.048 CPU:3.020 real:3.058
>>>>>>
>>>>>>
>>>>>> On 29/01/16 00:40, Hieu Hoang wrote:
>>>>>>> If it works ok on the command line but crashes when using the
>>>> server,
>>>>>>> then that suggest a server issue.
>>>>>>>
>>>>>>> I don't know much about the server code, to be honest.
>>>>>>>
>>>> --
>>>> The University of Edinburgh is a charitable body, registered in
>>>> Scotland, with registration number SC005336.
>>>>
>>>>
>
> --
>
> *STAR Group* <http://www.star-group.net>
> <http://www.star-group.net/>
>
> *Martin Baumg?rtner*
>
> STAR Language Technology & Solutions GmbH
> Umberto-Nobile-Stra?e 19 | 71063 Sindelfingen | Germany
> Tel. +49 70 31-4 10 92-0
> <mailto:martin.baumgaertner@star-group.net>martin.baumgaertner@star-group.net
>
> Fax +49 70 31-4 10 92-70 www.star-group.net <http://www.star-group.net/>
> Gesch?ftsf?hrer: Oliver Rau, Bernd Barth
> Handelsregister Stuttgart HRB 245654 | St.-Nr. 56098/11677
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160201/27d4262d/attachment.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/gif
Size: 8030 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20160201/27d4262d/attachment.gif
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: not available
Url: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160201/27d4262d/attachment.pl
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 112, Issue 2
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 112, Issue 2"
Post a Comment