Moses-support Digest, Vol 122, Issue 17

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Conversion of phrase model to PhraseDictionaryCompact
(Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Sat, 10 Dec 2016 13:23:16 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Conversion of phrase model to
PhraseDictionaryCompact
To: Shubham Khandelwal <skhlnmiit@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbhsv19f_i-5RB_g5QiFtc-z9faN-=WD=wXO7pgoRUHFGg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

strange, did the CreateOnDiskPt command execute ok, ie. with no error?

Does this file exist:
/home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat
If you do
cat Misc.dat
what does it say?

Hieu Hoang
http://www.hoang.co.uk/hieu

On 10 December 2016 at 11:30, Shubham Khandelwal <skhlnmiit@gmail.com>
wrote:

> Hi,
>
> Thanks Hieu for your reply.
> I have used *CreateOnDiskPt* to binarize the model and stored in
> *phrase-table.3.folder *using following command:
>
>
> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2 phrase-table.3.gz
> phrase-table.3.folder*
>
> Also I have made changes in *moses.ini.3 (*i.e. I have converted
> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*) *
>
> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
> path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0
> output-factor=0
>
> Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * ,
> it gave following error after *Created input-output object*:
>
> *terminate called after throwing an instance of 'util::Exception'*
> * what(): OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t
> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw util::Exception
> because `iter == m_miscInfo.end()'.*
> *Couldn't find value for key NumSourceFactors*
> *Aborted (core dumped)*
>
> Here, I do not know that what key value should I pass and how ? Can you
> please help me in this regard.
>
> Thank you so much for your help.
>
> Regards,
> Shubham
>
> On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
>> This is a hierarchical model. You must binarize with CreateOnDiskPt for
>> this model
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 9 December 2016 at 08:18, Shubham Khandelwal <skhlnmiit@gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> Thanks. It worked now. I have created compact phrase table.
>>> Now, when I am running it using following command:
>>>
>>> *~/mosesdecoder/bin/moses
>>> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all*
>>>
>>> Then, after creating input-output object, it gave following segmentation
>>> fault error:
>>>
>>> Created input-output object : [14.796] seconds
>>> Ich bin ein Student
>>> Line 0: Initialize search took 0.000 seconds total
>>> Translating: <s> Ich bin ein Student </s> ||| [0,0]=X (1) [0,1]=X (1)
>>> [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1)
>>> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1)
>>> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1)
>>> [5,5]=X (1)
>>>
>>> Segmentation fault (core dumped)
>>>
>>> In my machine, I have 40GB RAM but still I am confused why it gave this
>>> error.
>>> Can you please help me in this regard. I have attached moses.ini.2 for
>>> your reference.
>>>
>>> Thanks.
>>>
>>> Regards,
>>> Shubham
>>>
>>> On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>>>
>>>> maybe try
>>>>
>>>> -encoding None
>>>>
>>>> On 08/12/2016 19:44, Shubham Khandelwal wrote:
>>>>
>>>> Hi Hieu,
>>>>
>>>> Thanks for your reply.
>>>> Yes, I have used the absolute path and also I tried with -T but it did
>>>> not work.
>>>> Is there any other solution to this problem.
>>>>
>>>> Btw, Can anybody please upload the compact model of all pre-made models
>>>> as this will take less space and also it will be very fast during decoding.
>>>>
>>>> Thanks.
>>>>
>>>> On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang <hieuhoang@gmail.com>
>>>> wrote:
>>>>
>>>>> the previous email you referred to says that the directory
>>>>>
>>>>> * binarised-model/ *
>>>>>
>>>>> *must exist before you run it, otherwise it will segfault. I would
>>>>> also use absolute path to make sure, ie. not *
>>>>> *binarised-model/phrase-table *
>>>>>
>>>>> *but *
>>>>>
>>>>>
>>>>> * /home/shubham/moses/binarised-model/phrase-table *
>>>>>
>>>>> *The previous email exchange also says you should try to add the
>>>>> argument *
>>>>>
>>>>>
>>>>> * -T . *
>>>>>
>>>>> Hieu Hoang
>>>>> http://www.hoang.co.uk/hieu
>>>>>
>>>>> On 8 December 2016 at 15:52, Shubham Khandelwal <skhlnmiit@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> This is just the reminder of my previous email.
>>>>>>
>>>>>> Thanking You.
>>>>>>
>>>>>> Regards,
>>>>>> Shubham
>>>>>>
>>>>>> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal <
>>>>>> skhlnmiit@gmail.com> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I have just downloaded phrase-table.2.gz (18GB) de-en model
>>>>>>> and phrase-table.3.gz (22GB) fr-en model from the available pre-made
>>>>>>> models.
>>>>>>> Now, I am converting them to PhraseDictionaryCompact using following
>>>>>>> command (for exmaple):
>>>>>>>
>>>>>>>
>>>>>>> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in
>>>>>>> ~/model/phrase-table.3.gz -nscores 4 -out binarised-model/phrase-table *
>>>>>>>
>>>>>>> But after passing 1/3, it gave following segementation fault error:
>>>>>>>
>>>>>>> *Pass 1/3: Creating hash function for rank assignment*
>>>>>>> *Segmentation fault (core dumped)*
>>>>>>>
>>>>>>> I have found almost same issue on this thread:
>>>>>>> http://comments.gmane.org/gmane.comp.nlp.moses.user/13033
>>>>>>> However, I have provided the existing *binarised-model *folder in
>>>>>>> the command. Also, I have the write-access in /tmp but still it
>>>>>>> gave sementation fault.
>>>>>>>
>>>>>>> Can you please tell me what could be wrong here ?
>>>>>>>
>>>>>>> Thanking You.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Shubham
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Moses-support mailing list
>>>>>> Moses-support@mit.edu
>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161210/94f58db2/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 122, Issue 17
**********************************************

Tweet