Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Moses vocabulary code (Lane Schwartz)
2. Re: Moses vocabulary code (Marcin Junczys-Dowmunt)
3. Re: Moses vocabulary code (Lane Schwartz)
4. Re: Moses vocabulary code (Marcin Junczys-Dowmunt)
5. Error while Decoding using Chart Decoder (surya)
----------------------------------------------------------------------
Message: 1
Date: Fri, 9 Oct 2015 16:11:11 -0500
From: Lane Schwartz <dowobeha@gmail.com>
Subject: [Moses-support] Moses vocabulary code
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZn=xxwcTsqzfdj-N-4LdkjwzMBonwjZGSv79d8gvkBpaQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hey,
I know this should be a simple thing to find, but what code in Moses is
responsible for mapping back and forth between strings and integers?
Thanks,
Lane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151009/0f0a4df2/attachment-0001.html
------------------------------
Message: 2
Date: Fri, 9 Oct 2015 23:14:52 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Moses vocabulary code
To: moses-support@mit.edu
Message-ID: <56182E4C.30303@amu.edu.pl>
Content-Type: text/plain; charset="windows-1252"
Hi,
This would only be a simple thing if there was a common framework for
that, but there isn't. Each datastructure implements its own
vocabularies and look-up tables. There is no common set of integers.
Best,
Marcin
W dniu 09.10.2015 o 23:11, Lane Schwartz pisze:
> Hey,
>
> I know this should be a simple thing to find, but what code in Moses
> is responsible for mapping back and forth between strings and integers?
>
> Thanks,
> Lane
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151009/7f25f6cc/attachment-0001.html
------------------------------
Message: 3
Date: Fri, 9 Oct 2015 16:22:14 -0500
From: Lane Schwartz <dowobeha@gmail.com>
Subject: Re: [Moses-support] Moses vocabulary code
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZkFwiVnsasHz98rSbvpm+YxGWmKrwwo65G_+Xp9koBnfQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Seriously? That sounds inefficient.
I've found code in KenLM that maps from strings to integers, but not the
other way around.
Marcin, do you know, for example, where any Moses code is for doing the
mapping for any data structure?
On Fri, Oct 9, 2015 at 4:14 PM, Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
wrote:
> Hi,
> This would only be a simple thing if there was a common framework for
> that, but there isn't. Each datastructure implements its own vocabularies
> and look-up tables. There is no common set of integers.
> Best,
> Marcin
>
> W dniu 09.10.2015 o 23:11, Lane Schwartz pisze:
>
> Hey,
>
> I know this should be a simple thing to find, but what code in Moses is
> responsible for mapping back and forth between strings and integers?
>
> Thanks,
> Lane
>
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
--
When a place gets crowded enough to require ID's, social collapse is not
far away. It is time to go elsewhere. The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151009/e8606377/attachment-0001.html
------------------------------
Message: 4
Date: Fri, 9 Oct 2015 23:28:54 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Moses vocabulary code
To: Lane Schwartz <dowobeha@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <56183196.5000006@amu.edu.pl>
Content-Type: text/plain; charset="utf-8"
For instance in my phrase table that would be
mosesdecoder/moses/TranslationModel/CompactPT/PhraseDecoder.h
StringVector<unsigned char, unsigned, std::allocator> m_sourceSymbols;
StringVector<unsigned char, unsigned, std::allocator> m_targetSymbols;
That's a memory-mapped vector of strings.
W dniu 09.10.2015 o 23:22, Lane Schwartz pisze:
> Seriously? That sounds inefficient.
>
> I've found code in KenLM that maps from strings to integers, but not
> the other way around.
>
> Marcin, do you know, for example, where any Moses code is for doing
> the mapping for any data structure?
>
>
> On Fri, Oct 9, 2015 at 4:14 PM, Marcin Junczys-Dowmunt
> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>
> Hi,
> This would only be a simple thing if there was a common framework
> for that, but there isn't. Each datastructure implements its own
> vocabularies and look-up tables. There is no common set of integers.
> Best,
> Marcin
>
> W dniu 09.10.2015 o 23:11, Lane Schwartz pisze:
>> Hey,
>>
>> I know this should be a simple thing to find, but what code in
>> Moses is responsible for mapping back and forth between strings
>> and integers?
>>
>> Thanks,
>> Lane
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> When a place gets crowded enough to require ID's, social collapse is not
> far away. It is time to go elsewhere. The best thing about space travel
> is that it made it possible to go elsewhere.
> -- R.A. Heinlein, "Time Enough For Love"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151009/d4deac92/attachment-0001.html
------------------------------
Message: 5
Date: Sat, 10 Oct 2015 03:10:53 +0530
From: surya <surya@cse.iitb.ac.in>
Subject: [Moses-support] Error while Decoding using Chart Decoder
To: Moses Support <moses-support@mit.edu>
Message-ID: <ab53b87ee269a29c73e5690d75621c92@cse.iitb.ac.in>
Content-Type: text/plain; charset=UTF-8; format=flowed
Hi,
I have trained a parallel corpus using hierarchical model providing my
own set of alignments with the command:
~/moses/mosesdecoder-master/scripts/training/train-model.perl
-hierarchical -glue-grammar -max-phrase-length 5 -root-dir train -corpus
~/moses/surya/en-fa -f en -e fa -alignment alignment-x -reordering
msd-bidirectional-fe -lm
0:3:/home/vishwajit/moses/corpus/lm/farsi.blm.en -external-bin-dir
~/moses/mosesdecoder-master/tools/ --first-step 4 >& training.out &
When, I am trying to decode using chart-decoder with the following
command (without tuning)
~/moses/mosesdecoder-master/bin/moses_chart -f ~/train/model/moses.ini
-max-chart-span 1000 < ~/train/model/en-fa.en
I am getting the following error:
"Translating: <s> this finding led to more explorations . </s> |||
[0,0]=X (1) [0,1]=X (1) [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1)
[0,6]=X (1) [0,7]=X (1) [0,8]=X (1) [1,1]=X (1) [1,2]=X (1) [1,3]=X (1)
[1,4]=X (1) [1,5]=X (1) [1,6]=X (1) [1,7]=X (1) [1,8]=X (1) [2,2]=X (1)
[2,3]=X (1) [2,4]=X (1) [2,5]=X (1) [2,6]=X (1) [2,7]=X (1) [2,8]=X (1)
[3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [3,6]=X (1) [3,7]=X (1) [3,8]=X (1)
[4,4]=X (1) [4,5]=X (1) [4,6]=X (1) [4,7]=X (1) [4,8]=X (1) [5,5]=X (1)
[5,6]=X (1) [5,7]=X (1) [5,8]=X (1) [6,6]=X (1) [6,7]=X (1) [6,8]=X (1)
[7,7]=X (1) [7,8]=X (1) [8,8]=X (1)
Check 0 failed in moses/LexicalReordering.h:51
Aborted (core dumped)"
I have trained the system using English as both target and the source
language. The target side(English) is pre-ordered form for Farsi
language.
Can someone help me with the issue.
Also I am not getting the expected output while training the
hierarchical model using the "--extract-options". I used the following
command:
~/moses/mosesdecoder-master/scripts/training/train-model.perl
-hierarchical -glue-grammar -max-phrase-length 8
--extract-options="--DisallowNonTermConsecSource" -root-dir train
-corpus ~/moses/surya/en-fa -f en -e fa -alignment alignment-x
-reordering msd-bidirectional-fe -lm
0:3:/home/vishwajit/moses/corpus/lm/farsi.blm.en --first-step 4 >&
training.out &
Thanks,
Surya Kant Agarwal
IIT Bombay
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 108, Issue 32
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 108, Issue 32"
Post a Comment