Moses-support Digest, Vol 108, Issue 33

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Moses vocabulary code (Lane Schwartz)
2. Re: Moses vocabulary code (Marcin Junczys-Dowmunt)
3. Re: Moses vocabulary code (Kenneth Heafield)


----------------------------------------------------------------------

Message: 1
Date: Fri, 9 Oct 2015 16:55:45 -0500
From: Lane Schwartz <dowobeha@gmail.com>
Subject: Re: [Moses-support] Moses vocabulary code
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZ=4K=DDgx_=Xi0au9GJxp_HUc80LbzHXyDZr=CxUafV4g@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Thanks, Marcin.

So when the various components of Moses pass words back and forth, what do
they send each other? std::string? StringPiece?

On Fri, Oct 9, 2015 at 4:28 PM, Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
wrote:

> For instance in my phrase table that would be
>
> mosesdecoder/moses/TranslationModel/CompactPT/PhraseDecoder.h
>
> StringVector<unsigned char, unsigned, std::allocator>
> m_sourceSymbols;
> StringVector<unsigned char, unsigned, std::allocator> m_targetSymbols;
>
> That's a memory-mapped vector of strings.
>
> W dniu 09.10.2015 o 23:22, Lane Schwartz pisze:
>
> Seriously? That sounds inefficient.
>
> I've found code in KenLM that maps from strings to integers, but not the
> other way around.
>
> Marcin, do you know, for example, where any Moses code is for doing the
> mapping for any data structure?
>
>
> On Fri, Oct 9, 2015 at 4:14 PM, Marcin Junczys-Dowmunt <junczys@amu.edu.pl
> > wrote:
>
>> Hi,
>> This would only be a simple thing if there was a common framework for
>> that, but there isn't. Each datastructure implements its own vocabularies
>> and look-up tables. There is no common set of integers.
>> Best,
>> Marcin
>>
>> W dniu 09.10.2015 o 23:11, Lane Schwartz pisze:
>>
>> Hey,
>>
>> I know this should be a simple thing to find, but what code in Moses is
>> responsible for mapping back and forth between strings and integers?
>>
>> Thanks,
>> Lane
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
>
> --
> When a place gets crowded enough to require ID's, social collapse is not
> far away. It is time to go elsewhere. The best thing about space travel
> is that it made it possible to go elsewhere.
> -- R.A. Heinlein, "Time Enough For Love"
>
>
>


--
When a place gets crowded enough to require ID's, social collapse is not
far away. It is time to go elsewhere. The best thing about space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151009/b3567587/attachment-0001.html

------------------------------

Message: 2
Date: Sat, 10 Oct 2015 00:01:43 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Moses vocabulary code
To: Lane Schwartz <dowobeha@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <56183947.5080005@amu.edu.pl>
Content-Type: text/plain; charset="utf-8"

Hopefully StringPiece if it's newer code.
In my own code I wasn't yet using StringPiece and I did not rewrite it
after Moses switched mostly to StringPiece. Something to fix.

W dniu 09.10.2015 o 23:55, Lane Schwartz pisze:
> Thanks, Marcin.
>
> So when the various components of Moses pass words back and forth,
> what do they send each other? std::string? StringPiece?
>
> On Fri, Oct 9, 2015 at 4:28 PM, Marcin Junczys-Dowmunt
> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>
> For instance in my phrase table that would be
>
> mosesdecoder/moses/TranslationModel/CompactPT/PhraseDecoder.h
>
> StringVector<unsigned char, unsigned, std::allocator>
> m_sourceSymbols;
> StringVector<unsigned char, unsigned, std::allocator>
> m_targetSymbols;
>
> That's a memory-mapped vector of strings.
>
> W dniu 09.10.2015 o 23:22, Lane Schwartz pisze:
>> Seriously? That sounds inefficient.
>>
>> I've found code in KenLM that maps from strings to integers, but
>> not the other way around.
>>
>> Marcin, do you know, for example, where any Moses code is for
>> doing the mapping for any data structure?
>>
>>
>> On Fri, Oct 9, 2015 at 4:14 PM, Marcin Junczys-Dowmunt
>> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>>
>> Hi,
>> This would only be a simple thing if there was a common
>> framework for that, but there isn't. Each datastructure
>> implements its own vocabularies and look-up tables. There is
>> no common set of integers.
>> Best,
>> Marcin
>>
>> W dniu 09.10.2015 o 23:11, Lane Schwartz pisze:
>>> Hey,
>>>
>>> I know this should be a simple thing to find, but what code
>>> in Moses is responsible for mapping back and forth between
>>> strings and integers?
>>>
>>> Thanks,
>>> Lane
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>>
>> --
>> When a place gets crowded enough to require ID's, social collapse
>> is not
>> far away. It is time to go elsewhere. The best thing about
>> space travel
>> is that it made it possible to go elsewhere.
>> -- R.A. Heinlein, "Time Enough For Love"
>
>
>
>
> --
> When a place gets crowded enough to require ID's, social collapse is not
> far away. It is time to go elsewhere. The best thing about space travel
> is that it made it possible to go elsewhere.
> -- R.A. Heinlein, "Time Enough For Love"

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151009/31c63a1f/attachment-0001.html

------------------------------

Message: 3
Date: Fri, 9 Oct 2015 23:15:06 +0100
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] Moses vocabulary code
To: moses-support@mit.edu
Message-ID: <56183C6A.1000201@kheafield.com>
Content-Type: text/plain; charset=utf-8

The Moses common vocabulary is moses/FactorCollection.h. Common
practice in core Moses code is to pass around a const Factor * (which
can be resolved to a StringPiece or a consecutive ID).

If a feature/phrase table has its own ids because e.g. it's baked into
the binary file, then there's a std::vector to map from Moses ID to
feature function ID. See moses/LM/Ken.h:99 for an example.

std::string (or even StringPiece) conversion at decode time is a bug. A
sadly common one.

On 10/09/2015 10:22 PM, Lane Schwartz wrote:
> Seriously? That sounds inefficient.
>
> I've found code in KenLM that maps from strings to integers, but not the
> other way around.
>
> Marcin, do you know, for example, where any Moses code is for doing the
> mapping for any data structure?
>
>
> On Fri, Oct 9, 2015 at 4:14 PM, Marcin Junczys-Dowmunt
> <junczys@amu.edu.pl <mailto:junczys@amu.edu.pl>> wrote:
>
> Hi,
> This would only be a simple thing if there was a common framework
> for that, but there isn't. Each datastructure implements its own
> vocabularies and look-up tables. There is no common set of integers.
> Best,
> Marcin
>
> W dniu 09.10.2015 o 23:11, Lane Schwartz pisze:
>> Hey,
>>
>> I know this should be a simple thing to find, but what code in
>> Moses is responsible for mapping back and forth between strings
>> and integers?
>>
>> Thanks,
>> Lane
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> When a place gets crowded enough to require ID's, social collapse is not
> far away. It is time to go elsewhere. The best thing about space travel
> is that it made it possible to go elsewhere.
> -- R.A. Heinlein, "Time Enough For Love"
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 108, Issue 33
**********************************************

Tweet