Moses-support Digest, Vol 108, Issue 34

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Moses vocabulary code (Hieu Hoang)
2. Re: Error while Decoding using Chart Decoder (Hieu Hoang)
3. Re: Moses vocabulary code (Marcin Junczys-Dowmunt)


----------------------------------------------------------------------

Message: 1
Date: Fri, 9 Oct 2015 23:54:31 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Moses vocabulary code
To: Lane Schwartz <dowobeha@gmail.com>, Marcin Junczys-Dowmunt
<junczys@amu.edu.pl>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <561845A7.3080100@gmail.com>
Content-Type: text/plain; charset="windows-1252"

const Factor* is the vocab id. It's guaranteed to be unique for each
unique string. You can map directly to the string using
factor->GetString()


On 09/10/2015 22:55, Lane Schwartz wrote:
> Thanks, Marcin.
>
> So when the various components of Moses pass words back and forth,
> what do they send each other? std::string? StringPiece?
>
> On Fri, Oct 9, 2015 at 4:28 PM, Marcin Junczys-Dowmunt
> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>
> For instance in my phrase table that would be
>
> mosesdecoder/moses/TranslationModel/CompactPT/PhraseDecoder.h
>
> StringVector<unsigned char, unsigned, std::allocator>
> m_sourceSymbols;
> StringVector<unsigned char, unsigned, std::allocator>
> m_targetSymbols;
>
> That's a memory-mapped vector of strings.
>
> W dniu 09.10.2015 o 23:22, Lane Schwartz pisze:
>> Seriously? That sounds inefficient.
>>
>> I've found code in KenLM that maps from strings to integers, but
>> not the other way around.
>>
>> Marcin, do you know, for example, where any Moses code is for
>> doing the mapping for any data structure?
>>
>>
>> On Fri, Oct 9, 2015 at 4:14 PM, Marcin Junczys-Dowmunt
>> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>>
>> Hi,
>> This would only be a simple thing if there was a common
>> framework for that, but there isn't. Each datastructure
>> implements its own vocabularies and look-up tables. There is
>> no common set of integers.
>> Best,
>> Marcin
>>
>> W dniu 09.10.2015 o 23:11, Lane Schwartz pisze:
>>> Hey,
>>>
>>> I know this should be a simple thing to find, but what code
>>> in Moses is responsible for mapping back and forth between
>>> strings and integers?
>>>
>>> Thanks,
>>> Lane
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>>
>> --
>> When a place gets crowded enough to require ID's, social collapse
>> is not
>> far away. It is time to go elsewhere. The best thing about
>> space travel
>> is that it made it possible to go elsewhere.
>> -- R.A. Heinlein, "Time Enough For Love"
>
>
>
>
> --
> When a place gets crowded enough to require ID's, social collapse is not
> far away. It is time to go elsewhere. The best thing about space travel
> is that it made it possible to go elsewhere.
> -- R.A. Heinlein, "Time Enough For Love"
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

--
Hieu Hoang
http://www.hoang.co.uk/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151009/651a901d/attachment-0001.html

------------------------------

Message: 2
Date: Fri, 9 Oct 2015 23:55:21 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Error while Decoding using Chart Decoder
To: surya <surya@cse.iitb.ac.in>, Moses Support
<moses-support@mit.edu>
Message-ID: <561845D9.70505@gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed

You can't use LexicalReordering with hierarchical model

On 09/10/2015 22:40, surya wrote:
> Hi,
>
> I have trained a parallel corpus using hierarchical model providing my
> own set of alignments with the command:
>
> ~/moses/mosesdecoder-master/scripts/training/train-model.perl
> -hierarchical -glue-grammar -max-phrase-length 5 -root-dir train -corpus
> ~/moses/surya/en-fa -f en -e fa -alignment alignment-x -reordering
> msd-bidirectional-fe -lm
> 0:3:/home/vishwajit/moses/corpus/lm/farsi.blm.en -external-bin-dir
> ~/moses/mosesdecoder-master/tools/ --first-step 4 >& training.out &
>
>
> When, I am trying to decode using chart-decoder with the following
> command (without tuning)
>
> ~/moses/mosesdecoder-master/bin/moses_chart -f ~/train/model/moses.ini
> -max-chart-span 1000 < ~/train/model/en-fa.en
>
> I am getting the following error:
>
> "Translating: <s> this finding led to more explorations . </s> |||
> [0,0]=X (1) [0,1]=X (1) [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1)
> [0,6]=X (1) [0,7]=X (1) [0,8]=X (1) [1,1]=X (1) [1,2]=X (1) [1,3]=X (1)
> [1,4]=X (1) [1,5]=X (1) [1,6]=X (1) [1,7]=X (1) [1,8]=X (1) [2,2]=X (1)
> [2,3]=X (1) [2,4]=X (1) [2,5]=X (1) [2,6]=X (1) [2,7]=X (1) [2,8]=X (1)
> [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [3,6]=X (1) [3,7]=X (1) [3,8]=X (1)
> [4,4]=X (1) [4,5]=X (1) [4,6]=X (1) [4,7]=X (1) [4,8]=X (1) [5,5]=X (1)
> [5,6]=X (1) [5,7]=X (1) [5,8]=X (1) [6,6]=X (1) [6,7]=X (1) [6,8]=X (1)
> [7,7]=X (1) [7,8]=X (1) [8,8]=X (1)
>
> Check 0 failed in moses/LexicalReordering.h:51
> Aborted (core dumped)"
>
> I have trained the system using English as both target and the source
> language. The target side(English) is pre-ordered form for Farsi
> language.
>
> Can someone help me with the issue.
>
> Also I am not getting the expected output while training the
> hierarchical model using the "--extract-options". I used the following
> command:
>
> ~/moses/mosesdecoder-master/scripts/training/train-model.perl
> -hierarchical -glue-grammar -max-phrase-length 8
> --extract-options="--DisallowNonTermConsecSource" -root-dir train
> -corpus ~/moses/surya/en-fa -f en -e fa -alignment alignment-x
> -reordering msd-bidirectional-fe -lm
> 0:3:/home/vishwajit/moses/corpus/lm/farsi.blm.en --first-step 4 >&
> training.out &
>
>
> Thanks,
> Surya Kant Agarwal
> IIT Bombay
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>

--
Hieu Hoang
http://www.hoang.co.uk/hieu



------------------------------

Message: 3
Date: Sat, 10 Oct 2015 00:59:02 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Moses vocabulary code
To: Hieu Hoang <hieuhoang@gmail.com>, Lane Schwartz
<dowobeha@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <561846B6.40204@amu.edu.pl>
Content-Type: text/plain; charset="windows-1252"

oh, I didn't know that. Is any feature function actually using that?

W dniu 10.10.2015 o 00:54, Hieu Hoang pisze:
> const Factor* is the vocab id. It's guaranteed to be unique for each
> unique string. You can map directly to the string using
> factor->GetString()
>
>
> On 09/10/2015 22:55, Lane Schwartz wrote:
>> Thanks, Marcin.
>>
>> So when the various components of Moses pass words back and forth,
>> what do they send each other? std::string? StringPiece?
>>
>> On Fri, Oct 9, 2015 at 4:28 PM, Marcin Junczys-Dowmunt
>> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>>
>> For instance in my phrase table that would be
>>
>> mosesdecoder/moses/TranslationModel/CompactPT/PhraseDecoder.h
>>
>> StringVector<unsigned char, unsigned, std::allocator>
>> m_sourceSymbols;
>> StringVector<unsigned char, unsigned, std::allocator>
>> m_targetSymbols;
>>
>> That's a memory-mapped vector of strings.
>>
>> W dniu 09.10.2015 o 23:22, Lane Schwartz pisze:
>>> Seriously? That sounds inefficient.
>>>
>>> I've found code in KenLM that maps from strings to integers, but
>>> not the other way around.
>>>
>>> Marcin, do you know, for example, where any Moses code is for
>>> doing the mapping for any data structure?
>>>
>>>
>>> On Fri, Oct 9, 2015 at 4:14 PM, Marcin Junczys-Dowmunt
>>> <junczys@amu.edu.pl> wrote:
>>>
>>> Hi,
>>> This would only be a simple thing if there was a common
>>> framework for that, but there isn't. Each datastructure
>>> implements its own vocabularies and look-up tables. There is
>>> no common set of integers.
>>> Best,
>>> Marcin
>>>
>>> W dniu 09.10.2015 o 23:11, Lane Schwartz pisze:
>>>> Hey,
>>>>
>>>> I know this should be a simple thing to find, but what code
>>>> in Moses is responsible for mapping back and forth between
>>>> strings and integers?
>>>>
>>>> Thanks,
>>>> Lane
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>>
>>>
>>> --
>>> When a place gets crowded enough to require ID's, social
>>> collapse is not
>>> far away. It is time to go elsewhere. The best thing about
>>> space travel
>>> is that it made it possible to go elsewhere.
>>> -- R.A. Heinlein, "Time Enough For Love"
>>
>>
>>
>>
>> --
>> When a place gets crowded enough to require ID's, social collapse is not
>> far away. It is time to go elsewhere. The best thing about space travel
>> is that it made it possible to go elsewhere.
>> -- R.A. Heinlein, "Time Enough For Love"
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
> --
> Hieu Hoang
> http://www.hoang.co.uk/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151009/8c85cfd9/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 108, Issue 34
**********************************************

Tweet