Moses-support Digest, Vol 108, Issue 35

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Moses vocabulary code (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Sat, 10 Oct 2015 00:40:56 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Moses vocabulary code
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>, Lane Schwartz
<dowobeha@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <56185088.5010109@gmail.com>
Content-Type: text/plain; charset="windows-1252"

err, i thought every ff uses (const Factor*), but perhaps not now that i
look at the code. Probing pt uses it
ProbingPt.h line 43-47
probing pt doesn't support multiple factors.

Most other pt supprts supports multiple factors but does it
inefficiently by getting tokenizing the string into multiple factors.

Factor class also has a unique id which KenLM and IRSTLM uses to map to
their internal id. I would prefer everyone (const Factor*) as the unique
id, but it's no big deal

On 09/10/2015 23:59, Marcin Junczys-Dowmunt wrote:
> oh, I didn't know that. Is any feature function actually using that?
>
> W dniu 10.10.2015 o 00:54, Hieu Hoang pisze:
>> const Factor* is the vocab id. It's guaranteed to be unique for each
>> unique string. You can map directly to the string using
>> factor->GetString()
>>
>>
>> On 09/10/2015 22:55, Lane Schwartz wrote:
>>> Thanks, Marcin.
>>>
>>> So when the various components of Moses pass words back and forth,
>>> what do they send each other? std::string? StringPiece?
>>>
>>> On Fri, Oct 9, 2015 at 4:28 PM, Marcin Junczys-Dowmunt
>>> <junczys@amu.edu.pl> wrote:
>>>
>>> For instance in my phrase table that would be
>>>
>>> mosesdecoder/moses/TranslationModel/CompactPT/PhraseDecoder.h
>>>
>>> StringVector<unsigned char, unsigned, std::allocator>
>>> m_sourceSymbols;
>>> StringVector<unsigned char, unsigned, std::allocator>
>>> m_targetSymbols;
>>>
>>> That's a memory-mapped vector of strings.
>>>
>>> W dniu 09.10.2015 o 23:22, Lane Schwartz pisze:
>>>> Seriously? That sounds inefficient.
>>>>
>>>> I've found code in KenLM that maps from strings to integers,
>>>> but not the other way around.
>>>>
>>>> Marcin, do you know, for example, where any Moses code is for
>>>> doing the mapping for any data structure?
>>>>
>>>>
>>>> On Fri, Oct 9, 2015 at 4:14 PM, Marcin Junczys-Dowmunt
>>>> <junczys@amu.edu.pl> wrote:
>>>>
>>>> Hi,
>>>> This would only be a simple thing if there was a common
>>>> framework for that, but there isn't. Each datastructure
>>>> implements its own vocabularies and look-up tables. There
>>>> is no common set of integers.
>>>> Best,
>>>> Marcin
>>>>
>>>> W dniu 09.10.2015 o 23:11, Lane Schwartz pisze:
>>>>> Hey,
>>>>>
>>>>> I know this should be a simple thing to find, but what
>>>>> code in Moses is responsible for mapping back and forth
>>>>> between strings and integers?
>>>>>
>>>>> Thanks,
>>>>> Lane
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> When a place gets crowded enough to require ID's, social
>>>> collapse is not
>>>> far away. It is time to go elsewhere. The best thing about
>>>> space travel
>>>> is that it made it possible to go elsewhere.
>>>> -- R.A. Heinlein, "Time Enough For Love"
>>>
>>>
>>>
>>>
>>> --
>>> When a place gets crowded enough to require ID's, social collapse is not
>>> far away. It is time to go elsewhere. The best thing about space
>>> travel
>>> is that it made it possible to go elsewhere.
>>> -- R.A. Heinlein, "Time Enough For Love"
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> --
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>

--
Hieu Hoang
http://www.hoang.co.uk/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20151009/86ed3435/attachment-0001.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 108, Issue 35
**********************************************

Tweet