Moses-support Digest, Vol 108, Issue 47

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Compact lex reordering table on OSX/clang (Jeroen Vermeulen)
2. Re: Compact lex reordering table on OSX/clang (Hieu Hoang)
3. Re: Compact lex reordering table on OSX/clang (Jeroen Vermeulen)
4. Re: Compact lex reordering table on OSX/clang
(Marcin Junczys-Dowmunt)
5. Re: decoding-graph-backoff (Saumitra Yadav)


----------------------------------------------------------------------

Message: 1
Date: Tue, 13 Oct 2015 13:03:41 +0700
From: Jeroen Vermeulen <jtv@precisiontranslationtools.com>
Subject: Re: [Moses-support] Compact lex reordering table on OSX/clang
To: moses-support@mit.edu
Message-ID: <561C9EBD.4090100@precisiontranslationtools.com>
Content-Type: text/plain; charset=windows-1252

On 10/12/2015 11:15 PM, Hieu Hoang wrote:
> I'm not sure if anyone else encounters it but the compact lexical
> reordering table crashes for me on OSX/clang during loading.
>
> The stack trace i have for this is
> LexicalReorderingTableCompact::LexicalReorderingTableCompact
> LexicalReorderingTableCompact::Load line 180
> StringVector::load line 2808
> StringVector::loadCharArray line 247

Could the file simply not be open? It's opened in
LexicalReorderingTableCompact::Load, but as far as I can see, *nothing*
ever checks that this actually works. Code just keeps reading from the
file and assuming success, and using the possibly invalid result. Maybe
this just happens to be the first point where it causes a crash.


Jeroen


------------------------------

Message: 2
Date: Tue, 13 Oct 2015 10:59:16 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Compact lex reordering table on OSX/clang
To: Jeroen Vermeulen <jtv@precisiontranslationtools.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbh63O5-TE-5rvAnKmF_N4mqMoM7j3eX916kT8FZ3+Ro8A@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

you're quite right, i've added a check

https://github.com/moses-smt/mosesdecoder/commit/982d52e5b657f4c1fa7369e577cfd75a8af16543
However, that the the problem I'm having on OSX. It opens but it crashes on
loading.

I suspect one of the datatypes has slightly different size on clang/OSX
compared to gcc/Linux

Hieu Hoang
http://www.hoang.co.uk/hieu

On 13 October 2015 at 07:03, Jeroen Vermeulen <
jtv@precisiontranslationtools.com> wrote:

> On 10/12/2015 11:15 PM, Hieu Hoang wrote:
> > I'm not sure if anyone else encounters it but the compact lexical
> > reordering table crashes for me on OSX/clang during loading.
> >
> > The stack trace i have for this is
> > LexicalReorderingTableCompact::LexicalReorderingTableCompact
> > LexicalReorderingTableCompact::Load line 180
> > StringVector::load line 2808
> > StringVector::loadCharArray line 247
>
> Could the file simply not be open? It's opened in
> LexicalReorderingTableCompact::Load, but as far as I can see, *nothing*
> ever checks that this actually works. Code just keeps reading from the
> file and assuming success, and using the possibly invalid result. Maybe
> this just happens to be the first point where it causes a crash.
>
>
> Jeroen
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151013/27db6084/attachment-0001.html

------------------------------

Message: 3
Date: Tue, 13 Oct 2015 17:50:18 +0700
From: Jeroen Vermeulen <jtv@precisiontranslationtools.com>
Subject: Re: [Moses-support] Compact lex reordering table on OSX/clang
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <561CE1EA.6010204@precisiontranslationtools.com>
Content-Type: text/plain; charset=utf-8

On 10/13/2015 04:59 PM, Hieu Hoang wrote:
> you're quite right, i've added a check
>
> https://github.com/moses-smt/mosesdecoder/commit/982d52e5b657f4c1fa7369e577cfd75a8af16543
> However, that the the problem I'm having on OSX. It opens but it crashes
> on loading.
>
> I suspect one of the datatypes has slightly different size on clang/OSX
> compared to gcc/Linux

Before the loading gets to this point, CanonicalHuffman.Load() does
something that intrigues me, as a reader who doesn't really grok the
code: it fread()s an array of Data.

If Data is the class I find in mert/Data.h, then AFAICT the compiler
would be well within its rights to break this. Not only is it not a
POD, it contains pointers, including in strings and vectors. You
wouldn't expect that to work. Did I take a wrong turn somewhere?


Jeroen


------------------------------

Message: 4
Date: Tue, 13 Oct 2015 13:05:41 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Compact lex reordering table on OSX/clang
To: Jeroen Vermeulen <jtv@precisiontranslationtools.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <446b02bd2abaae890e715bbcf9dc75c1@amu.edu.pl>
Content-Type: text/plain; charset="utf-8"



Hi,

yes, definitely wrong turn, all code should be in CompactPT.
I am not sure this is actually a code bug, is it working with g++ on
macOS?

W dniu 2015-10-13 12:50, Jeroen Vermeulen napisa?(a):

> On 10/13/2015 04:59 PM, Hieu Hoang wrote:
>
>> you're quite right, i've added a check https://github.com/moses-smt/mosesdecoder/commit/982d52e5b657f4c1fa7369e577cfd75a8af16543 [1] However, that the the problem I'm having on OSX. It opens but it crashes on loading. I suspect one of the datatypes has slightly different size on clang/OSX compared to gcc/Linux
>
> Before the loading gets to this point, CanonicalHuffman.Load() does
> something that intrigues me, as a reader who doesn't really grok the
> code: it fread()s an array of Data.
>
> If Data is the class I find in mert/Data.h, then AFAICT the compiler
> would be well within its rights to break this. Not only is it not a
> POD, it contains pointers, including in strings and vectors. You
> wouldn't expect that to work. Did I take a wrong turn somewhere?
>
> Jeroen
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support [2]



Links:
------
[1]
https://github.com/moses-smt/mosesdecoder/commit/982d52e5b657f4c1fa7369e577cfd75a8af16543
[2] http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151013/44d677a1/attachment-0001.html

------------------------------

Message: 5
Date: Tue, 13 Oct 2015 16:58:15 +0530
From: Saumitra Yadav <yadav.saumitra07@gmail.com>
Subject: Re: [Moses-support] decoding-graph-backoff
To: Philipp Koehn <phi@jhu.edu>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAPeFGXZut4U6gqcMXRHyRk1oZz59JrkS4UNjOWp-_ddJtdNofA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Sir,
As you mentioned adding "-tt" while running decoder for input gives us
feature score for each pair, also a non-zero value for a translation model
(given that there are multiple phrase-table) gives us indication from which
phrase-table translation candidate was used. Sir I'm making a system for
Hindi to Urdu , and after running decoder with "-tt" im getting (small
sample):

|lm=(2:-8.9735,1:-17.0476,2:-0.639163,2:-6.23737,3:-4.51238,3:-1.19594,3:-6.55882,3:-3.23557,3:-2.26149,2:-14.4246,2:-2.78677,2:-7.39963,3:-1.62681,2:-13.4158,1:-13.6408,2:-1.33196,3:-1.25049)|
???? |0-0,wa=0-0 ,total=-0.28041, LexicalReordering0= -0.0316486 0 0 0 0 0
Distortion0= 0 LM0= -8.9735 WordPenalty0= -1 PhrasePenalty0= 1
TranslationModel0=
-0.167537 -0.313897 -0.239363 -0.242946 TranslationModel1= 0 0 0 0| ????
|1-1,wa=0-0 ,total=-0.463503, LexicalReordering0= -0.510826 0 0 -0.0511291
0 0 Distortion0= 0 LM0= -17.0476 WordPenalty0= -1 PhrasePenalty0= 1
*TranslationModel0=
0 0 0 0 TranslationModel1= 0 0 0 0*|


Red colored TranslationModel0 feature tells us that translation model 0 was
used to get translated phrase in red color
but sir blue color one says that none of the two translation models were
used? Is my assumption correct?
also phrase was translated (blue font)
can you please tell why this happened?


Regards,
Saumitra Yadav
Intern, LTRC
IIIT-Hyderabad

On Thu, Jul 30, 2015 at 7:13 PM, Philipp Koehn <phi@jhu.edu> wrote:

> Hi,
>
> yes, that is correct. If there are non-zero valued scores listed with a
> translation model feature, then this translation model was used for the
> phrase pair.
>
> -phi
>
> On Wed, Jul 29, 2015 at 7:57 PM, Saumitra Yadav <
> yadav.saumitra07@gmail.com> wrote:
>
>> Sir,
>> Thank you for that option , it really helped. I just wanted to know if
>> I'm analysing it correctly
>> For initial analysis m just finding how many times which phrase tables
>> were called , so in attached file (formatted just for easy readability )
>> please find the output of one sentence , is it correct to say that
>> TranslationModel0 was used 2 times and TranslationModel1 was used 5 times
>> for given input?
>>
>> Regards,
>> Saumitra Yadav
>> M.Tech.
>> Department Of Computer Science And Technology
>> Goa University
>>
>>
>> On Wed, Jul 29, 2015 at 9:22 PM, Philipp Koehn <phi@jhu.edu> wrote:
>>
>>> Hi,
>>>
>>> when you call the decoder with the option "-tt" then you get for
>>> each phrase pairs a list of all feature scores. You can use this
>>> to track down which phrase table was used for each phrase
>>> translation.
>>>
>>> -phi
>>>
>>> On Wed, Jul 29, 2015 at 10:59 AM, Hieu Hoang <hieuhoang@gmail.com>
>>> wrote:
>>>
>>>> good question. no. You can try & write it yourself.
>>>>
>>>> in the TargetPhrase class, there is a method
>>>> GetContainer()
>>>> which points to the phrase-table that a particular rule comes from. You
>>>> can use this.
>>>>
>>>> On 29/07/2015 18:51, Saumitra Yadav wrote:
>>>>
>>>> Sir,
>>>> Is there a command or argument which can tell, which phrase in output
>>>> is taken from which phrase-table (incase we have multiple phrase-tables )?
>>>>
>>>> Regards,
>>>> Saumitra Yadav
>>>> M.Tech.
>>>> Department Of Computer Science And Technology
>>>> Goa University
>>>>
>>>>
>>>> On Sun, Jul 26, 2015 at 11:49 AM, Hieu Hoang <hieuhoang@gmail.com>
>>>> wrote:
>>>>
>>>>> since you have 3 phrase-tables, you may have to have 3 entries in the
>>>>> [decoding-graph-backoff] section, eg
>>>>> [decoding-graph-backoff]
>>>>> 0
>>>>> 3
>>>>> 3
>>>>>
>>>>>
>>>>> Hieu Hoang
>>>>> Researcher
>>>>> New York University, Abu Dhabi
>>>>> http://www.hoang.co.uk/hieu
>>>>>
>>>>> On 25 July 2015 at 20:23, Saumitra Yadav <
>>>>> <yadav.saumitra07@gmail.com>yadav.saumitra07@gmail.com> wrote:
>>>>>
>>>>>> Sir,
>>>>>> Please find attached , moses.ini file i used and command used
>>>>>> was ~/Decoder/mosesdecoder/bin/moses -f moses.ini
>>>>>>
>>>>>> Regards,
>>>>>> Saumitra Yadav
>>>>>> M.Tech.
>>>>>> Department Of Computer Science And Technology
>>>>>> Goa University
>>>>>>
>>>>>>
>>>>>> On Sat, Jul 25, 2015 at 9:21 PM, Hieu Hoang <hieuhoang@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> can you please send me the moses.ini file that you used, that cause
>>>>>>> the segfault. And send me the exact command you typed
>>>>>>>
>>>>>>>
>>>>>>> On 24/07/2015 14:40, Saumitra Yadav wrote:
>>>>>>>
>>>>>>> But sir when i did that there was * segmentation fault* while
>>>>>>> loading first phrase-table, one walk around i got was giving phrase-table
>>>>>>> uncompressed to decoder.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Saumitra Yadav
>>>>>>> M.Tech.
>>>>>>> Department Of Computer Science And Technology
>>>>>>> Goa University
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 23, 2015 at 8:06 PM, Hieu Hoang < <hieuhoang@gmail.com>
>>>>>>> hieuhoang@gmail.com> wrote:
>>>>>>>
>>>>>>>> i think you have to swap the phrase tables around. The decoder
>>>>>>>> always looks at the 1st phrase-table, then backoff to the 2nd if nothing is
>>>>>>>> found
>>>>>>>>
>>>>>>>>
>>>>>>>> On 22/07/2015 16:59, Saumitra Yadav wrote:
>>>>>>>>
>>>>>>>> Sir/Ma'am,
>>>>>>>> I'm trying to use multiple phrase tables for translation in Moses
>>>>>>>> decoder, with preference to 1st phrase-table, but was getting a
>>>>>>>> segmentation fault while loading 1st phrase table, so just switched the
>>>>>>>> positions of phrase-tables in moses configuration file and it was working ,
>>>>>>>> now the table i want to give preference is 2nd in list , can i use
>>>>>>>>
>>>>>>>> [decoding-graph-backoff]
>>>>>>>> 1
>>>>>>>> 3
>>>>>>>> in configuration file for backoff so that moses uses 2nd table and
>>>>>>>> uses 1st table only for words it couldn't find in 2nd phrase-table?
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Saumitra Yadav
>>>>>>>> M.Tech.
>>>>>>>> Department Of Computer Science And Technology
>>>>>>>> Goa University
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Hieu Hoang
>>>>>>>> Researcher
>>>>>>>> New York University, Abu Dhabihttp://www.hoang.co.uk/hieu
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Hieu Hoang
>>>>>>> Researcher
>>>>>>> New York University, Abu Dhabihttp://www.hoang.co.uk/hieu
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Hieu Hoang
>>>> Researcher
>>>> New York University, Abu Dhabihttp://www.hoang.co.uk/hieu
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151013/60ad9549/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 108, Issue 47
**********************************************

Tweet