Moses-support Digest, Vol 122, Issue 19

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. List of Neural Machine Translation Implementations (Nat Gillin)
2. Re: Conversion of phrase model to PhraseDictionaryCompact
(Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Mon, 12 Dec 2016 10:00:23 +0800
From: Nat Gillin <nat.gillin@gmail.com>
Subject: [Moses-support] List of Neural Machine Translation
Implementations
To: corpora@uib.no, moses-support <moses-support@mit.edu>
Message-ID:
<CAD2EOZjYQraRH4P3vDdzW8sRv93WPv41bD1WeoteM+mT2eO-=g@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Dear Corpora and Moses Community,

Sorry for cross-posting.

I have came across this awesome list of NMT implementations:
https://github.com/jonsafari/nmt-list

Are there other NMT implementations out there in the wild that are not
captured by this list?

Other than the list of NMTs, there are these:

- Under the "Querying" section of the KenLM website: there's a shortlist
of them (https://kheafield.com/code/kenlm/)
- And on the Moses "External Tools" page too:
http://www.statmt.org/moses/?n=Moses.ExternalTools#ntoc19


Are there other lists like this for NMT or any sort of MT tools/suite?


Regards,
Nat
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161211/330e2629/attachment-0001.html

------------------------------

Message: 2
Date: Mon, 12 Dec 2016 12:39:17 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Conversion of phrase model to
PhraseDictionaryCompact
To: Shubham Khandelwal <skhlnmiit@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbjt4=dppwge4haimnzgGzcLwrUMHt_hvsJ5bCya5WjEdw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hieu Hoang
http://www.hoang.co.uk/hieu

On 10 December 2016 at 14:06, Shubham Khandelwal <skhlnmiit@gmail.com>
wrote:

> Yes, CreateOnDiskPt command executed without any error.
>
> There are 5 files in this phrase-table.3.folder: Misc.dat , Source.dat,
> TargetColl.dat, TargetInd.dat, Vocab.dat
> *Misc.dat and Vocab.dat files are empty. *
> I just checked that my hard-disk memory is full as this folder took 165G
> already. So may be, due to this reason those 2 files are empty. But CreateOnDiskPt
> command should throw an error of *No space left on machine *when it
> stopped.
> Let me know if no space on my machine, is the issue or not so that I can
> go for better device having more hard-disk memory.
>
Good idea. Not sure who's going to do it but if you do it, please send me a
patch & I'll check it in

>
> Also May I know that How much memory phrase-table.3.folder has in general
> when CreateOnDiskPt command executes completely as phrase-table.3.gz size
> is only 23GB.
>
I'm not too sure. Try it on a disk with 1TB and please report back what you
find for future reference

>
> Thanking You.
>
>
> On Sat, Dec 10, 2016 at 6:53 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
>> strange, did the CreateOnDiskPt command execute ok, ie. with no error?
>>
>> Does this file exist:
>> /home/shubham/models/fr-en/phrase-table.3.folder/Misc.dat
>> If you do
>> cat Misc.dat
>> what does it say?
>>
>> Hieu Hoang
>> http://www.hoang.co.uk/hieu
>>
>> On 10 December 2016 at 11:30, Shubham Khandelwal <skhlnmiit@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Thanks Hieu for your reply.
>>> I have used *CreateOnDiskPt* to binarize the model and stored in
>>> *phrase-table.3.folder *using following command:
>>>
>>>
>>> *~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2 phrase-table.3.gz
>>> phrase-table.3.folder*
>>>
>>> Also I have made changes in *moses.ini.3 (*i.e. I have converted
>>> *PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*) *
>>>
>>> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
>>> path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0
>>> output-factor=0
>>>
>>> Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * ,
>>> it gave following error after *Created input-output object*:
>>>
>>> *terminate called after throwing an instance of 'util::Exception'*
>>> * what(): OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t
>>> OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw util::Exception
>>> because `iter == m_miscInfo.end()'.*
>>> *Couldn't find value for key NumSourceFactors*
>>> *Aborted (core dumped)*
>>>
>>> Here, I do not know that what key value should I pass and how ? Can you
>>> please help me in this regard.
>>>
>>> Thank you so much for your help.
>>>
>>> Regards,
>>> Shubham
>>>
>>> On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>>>
>>>> This is a hierarchical model. You must binarize with CreateOnDiskPt for
>>>> this model
>>>>
>>>> Hieu Hoang
>>>> http://www.hoang.co.uk/hieu
>>>>
>>>> On 9 December 2016 at 08:18, Shubham Khandelwal <skhlnmiit@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> Thanks. It worked now. I have created compact phrase table.
>>>>> Now, when I am running it using following command:
>>>>>
>>>>> *~/mosesdecoder/bin/moses
>>>>> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all*
>>>>>
>>>>> Then, after creating input-output object, it gave following
>>>>> segmentation fault error:
>>>>>
>>>>> Created input-output object : [14.796] seconds
>>>>> Ich bin ein Student
>>>>> Line 0: Initialize search took 0.000 seconds total
>>>>> Translating: <s> Ich bin ein Student </s> ||| [0,0]=X (1) [0,1]=X (1)
>>>>> [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1)
>>>>> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1)
>>>>> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1)
>>>>> [5,5]=X (1)
>>>>>
>>>>> Segmentation fault (core dumped)
>>>>>
>>>>> In my machine, I have 40GB RAM but still I am confused why it gave
>>>>> this error.
>>>>> Can you please help me in this regard. I have attached moses.ini.2 for
>>>>> your reference.
>>>>>
>>>>> Thanks.
>>>>>
>>>>> Regards,
>>>>> Shubham
>>>>>
>>>>> On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang <hieuhoang@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> maybe try
>>>>>>
>>>>>> -encoding None
>>>>>>
>>>>>> On 08/12/2016 19:44, Shubham Khandelwal wrote:
>>>>>>
>>>>>> Hi Hieu,
>>>>>>
>>>>>> Thanks for your reply.
>>>>>> Yes, I have used the absolute path and also I tried with -T but it
>>>>>> did not work.
>>>>>> Is there any other solution to this problem.
>>>>>>
>>>>>> Btw, Can anybody please upload the compact model of all pre-made
>>>>>> models as this will take less space and also it will be very fast during
>>>>>> decoding.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>> On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang <hieuhoang@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> the previous email you referred to says that the directory
>>>>>>>
>>>>>>> * binarised-model/ *
>>>>>>>
>>>>>>> *must exist before you run it, otherwise it will segfault. I would
>>>>>>> also use absolute path to make sure, ie. not *
>>>>>>> *binarised-model/phrase-table *
>>>>>>>
>>>>>>> *but *
>>>>>>>
>>>>>>>
>>>>>>> * /home/shubham/moses/binarised-model/phrase-table *
>>>>>>>
>>>>>>> *The previous email exchange also says you should try to add the
>>>>>>> argument *
>>>>>>>
>>>>>>>
>>>>>>> * -T . *
>>>>>>>
>>>>>>> Hieu Hoang
>>>>>>> http://www.hoang.co.uk/hieu
>>>>>>>
>>>>>>> On 8 December 2016 at 15:52, Shubham Khandelwal <skhlnmiit@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> This is just the reminder of my previous email.
>>>>>>>>
>>>>>>>> Thanking You.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Shubham
>>>>>>>>
>>>>>>>> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal <
>>>>>>>> skhlnmiit@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I have just downloaded phrase-table.2.gz (18GB) de-en model
>>>>>>>>> and phrase-table.3.gz (22GB) fr-en model from the available pre-made
>>>>>>>>> models.
>>>>>>>>> Now, I am converting them to PhraseDictionaryCompact using
>>>>>>>>> following command (for exmaple):
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in
>>>>>>>>> ~/model/phrase-table.3.gz -nscores 4 -out binarised-model/phrase-table *
>>>>>>>>>
>>>>>>>>> But after passing 1/3, it gave following segementation fault error:
>>>>>>>>>
>>>>>>>>> *Pass 1/3: Creating hash function for rank assignment*
>>>>>>>>> *Segmentation fault (core dumped)*
>>>>>>>>>
>>>>>>>>> I have found almost same issue on this thread:
>>>>>>>>> http://comments.gmane.org/gmane.comp.nlp.moses.user/13033
>>>>>>>>> However, I have provided the existing *binarised-model *folder in
>>>>>>>>> the command. Also, I have the write-access in /tmp but still it
>>>>>>>>> gave sementation fault.
>>>>>>>>>
>>>>>>>>> Can you please tell me what could be wrong here ?
>>>>>>>>>
>>>>>>>>> Thanking You.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Shubham
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Moses-support mailing list
>>>>>>>> Moses-support@mit.edu
>>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161212/a911fbca/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 122, Issue 19
**********************************************

Tweet