Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Fwd: (Philipp Koehn)
2. Is 0,5 BLEU normal with my training parameters?
(Hasan Sait ARSLAN)
3. Re: RNN-based features in Moses (Shafagh Fadaee)
4. Re: Conversion of phrase model to PhraseDictionaryCompact
(Shubham Khandelwal)
----------------------------------------------------------------------
Message: 1
Date: Fri, 9 Dec 2016 17:14:34 -0500
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] Fwd:
To: "G/her G/libanos" <gerizaba@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDA-gAm1DUWjhDh1md=95b+1Aor2-z+BC5jamQutjs5tgQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
to change the non-breaking prefix files, or to add a file for your
language, see
the directory $moses/scripts/share/nonbreaking_prefixes
Unfortunately, there is no Moses GUI that makes it user friendly.
The closest to that is a web-based interface to Moses.
See the documentation here:
http://www.statmt.org/moses/?n=Moses.WebTranslation
-phi
On Fri, Dec 9, 2016 at 9:17 AM, G/her G/libanos <gerizaba@gmail.com> wrote:
>
> hello there....
> 1. we done our work using baseline and the system work for the train data
> but we need to localized the source code in the tokinization to leave
> abrivated word ?/? as in english Adm. as Notbreaking_prefix.en that is use
> the dot(.)
>
> 2. where we to change the code in c++ or in the perl part of the codes
>
>
>
> 3. we need to uses the translation system in window, since the end users
> are not expert in ubuntu and we see the how to change the window see the
> Moses GUI and which part of the our file will be load to the model only or
> all the train including the train data and can we modify the GUI of the
> Moses.
>
>
>
> 10qs for every thing
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161209/b68d0dd5/attachment-0001.html
------------------------------
Message: 2
Date: Sat, 10 Dec 2016 10:35:51 +0200
From: Hasan Sait ARSLAN <hasan.sait.arslan@gmail.com>
Subject: [Moses-support] Is 0,5 BLEU normal with my training
parameters?
To: moses-support <moses-support@mit.edu>
Message-ID:
<CAJj7HD2kPab3ayXsV5SuZD17560duWMJQjY1OJr-b6c=yD9cNQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
I have trained my dataset using the following script;
~/build/mosesdecoder/scripts/training/train-model.perl -root-dir
train \
-corpus ~/factored_translation/dataset/pos/train
\
--f tr --e en -alignment tr-en.align -reordering
msd-bidirectional-fe \
-lm 0:3:/export/students/sait/factored_translation/lm/surface.lm:8
\
-lm 2:3:/export/students/sait/factored_translation/lm/pos.lm:8
\
--translation-factors 0-0,2
\
--external-bin-dir ~/build/mosesdecoder/bin
My input token factors are;
- surface form,
- lemma,
- part of speech, and
- part of speech with additional morphological information
My output token factors are;
- surface form,
- lemma, and
- part of speech
I have created two different language models. surface.lm is the model of
words in the target corpus, and pos.lm is the model of pos tags in the
target corpus.
When I translate, I obtain such sentences:
*-->They had to is cash from some bootleg DVDs can see are hiding her *
*-->cat ! did you get any Yoon is the one I BRANDON BRANDON I think I am
ever let me fires his songs *
*-->constant by idolise They have it invcible a sister . that it contender
They had to have ever happened . her into the in my heart it Mum *
*-->is tired without leaving francis recently aII-out he ca n't and thrusts
out the words ever put either of famous case long , oh do got some *
*-->BROWN : You dead father carry 's manager dead parents . BRENNER
Youbetterwatchout 's manager They had to have just moved my he 's going to
lose high volume ever had an STD ? mulling*
Is 0.5 BLEU score normal, what do you think? I mean, of course, when we
look at sentences, it is quite normal. But is it normal with the parameters
I have given for training?
Kind Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161210/72d0f15b/attachment-0001.html
------------------------------
Message: 3
Date: Sat, 10 Dec 2016 12:05:58 +0330
From: Shafagh Fadaee <shafagh@gmail.com>
Subject: Re: [Moses-support] RNN-based features in Moses
To: moses-support@mit.edu
Message-ID:
<CAKqZtrRne8oX2wATJ_a0o6KH+_rOVRbYpCpTUUWuj7ubd5z5CQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Dear Rico,
Thanks for the information.
I guess this is exactly what I was searching for. It would help me a lot.
Regards,
Hakimeh
> Date: Thu, 8 Dec 2016 17:39:39 +0000
> From: Rico Sennrich <rico.sennrich@gmx.ch>
> Subject: Re: [Moses-support] RNN-based features in Moses
> To: moses-support@mit.edu
> Message-ID: <1cf42d05-359f-fed8-32b3-0c71306d761e@gmx.ch>
> Content-Type: text/plain; charset="windows-1252"
>
> Hello Hakimeh,
>
> the branch https://github.com/moses-smt/mosesdecoder/tree/nmt-hybrid
> supports NMT (which is basically an RNN conditioned on the source text)
> as a feature function in Moses. It is described in this paper:
>
> @inproceedings{junczysdowmunt-dwojak-sennrich:2016:WMT,
> address = "Berlin, Germany",
> author = "Junczys-Dowmunt, Marcin and Dwojak, Tomasz and Sennrich,
> Rico",
> booktitle = "{Proceedings of the First Conference on Machine
> Translation}",
> month = "August",
> pages = "319--325",
> publisher = "Association for Computational Linguistics",
> title = "{The AMU-UEDIN Submission to the WMT16 News Translation
> Task: Attention-based NMT Models as Feature Functions in Phrase-based
> SMT}",
> url = "http://www.aclweb.org/anthology/W/W16/W16-2316",
> year = "2016"
> }
>
> I'm afraid that documentation is currently lacking though.
>
> best wishes,
> Rico
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161210/88828b7c/attachment-0001.html
------------------------------
Message: 4
Date: Sat, 10 Dec 2016 17:00:23 +0530
From: Shubham Khandelwal <skhlnmiit@gmail.com>
Subject: Re: [Moses-support] Conversion of phrase model to
PhraseDictionaryCompact
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAHweNTtuU-5z-MLhkaQ8u=4cN5wG3R+sQzhTBBSQv9m4N+6JSg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hi,
Thanks Hieu for your reply.
I have used *CreateOnDiskPt* to binarize the model and stored in
*phrase-table.3.folder *using following command:
*~/mosesdecoder/bin/CreateOnDiskPt 1 1 4 100 2 phrase-table.3.gz
phrase-table.3.folder*
Also I have made changes in *moses.ini.3 (*i.e. I have converted
*PhraseDictionaryMemory* to *PhraseDictionaryCompact* as follows*) *
PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
path=/home/shubham/models/fr-en/phrase-table.3.folder input-factor=0
output-factor=0
Now, when I run it using :* ~/mosesdecoder/bin/moses -f moses.ini.3 * , it
gave following error after *Created input-output object*:
*terminate called after throwing an instance of 'util::Exception'*
* what(): OnDiskPt/OnDiskWrapper.cpp:217 in uint64_t
OnDiskPt::OnDiskWrapper::GetMisc(const string&) const threw util::Exception
because `iter == m_miscInfo.end()'.*
*Couldn't find value for key NumSourceFactors*
*Aborted (core dumped)*
Here, I do not know that what key value should I pass and how ? Can you
please help me in this regard.
Thank you so much for your help.
Regards,
Shubham
On Fri, Dec 9, 2016 at 4:27 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
> This is a hierarchical model. You must binarize with CreateOnDiskPt for
> this model
>
> Hieu Hoang
> http://www.hoang.co.uk/hieu
>
> On 9 December 2016 at 08:18, Shubham Khandelwal <skhlnmiit@gmail.com>
> wrote:
>
>> Hello,
>>
>> Thanks. It worked now. I have created compact phrase table.
>> Now, when I am running it using following command:
>>
>> *~/mosesdecoder/bin/moses
>> -f ~/Translate/models/de-en/model/moses.ini.2 -threads all*
>>
>> Then, after creating input-output object, it gave following segmentation
>> fault error:
>>
>> Created input-output object : [14.796] seconds
>> Ich bin ein Student
>> Line 0: Initialize search took 0.000 seconds total
>> Translating: <s> Ich bin ein Student </s> ||| [0,0]=X (1) [0,1]=X (1)
>> [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1) [1,2]=X (1)
>> [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1) [2,4]=X (1)
>> [2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1) [4,5]=X (1)
>> [5,5]=X (1)
>>
>> Segmentation fault (core dumped)
>>
>> In my machine, I have 40GB RAM but still I am confused why it gave this
>> error.
>> Can you please help me in this regard. I have attached moses.ini.2 for
>> your reference.
>>
>> Thanks.
>>
>> Regards,
>> Shubham
>>
>> On Fri, Dec 9, 2016 at 2:02 AM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>>
>>> maybe try
>>>
>>> -encoding None
>>>
>>> On 08/12/2016 19:44, Shubham Khandelwal wrote:
>>>
>>> Hi Hieu,
>>>
>>> Thanks for your reply.
>>> Yes, I have used the absolute path and also I tried with -T but it did
>>> not work.
>>> Is there any other solution to this problem.
>>>
>>> Btw, Can anybody please upload the compact model of all pre-made models
>>> as this will take less space and also it will be very fast during decoding.
>>>
>>> Thanks.
>>>
>>> On Fri, Dec 9, 2016 at 12:50 AM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>>>
>>>> the previous email you referred to says that the directory
>>>>
>>>> * binarised-model/ *
>>>>
>>>> *must exist before you run it, otherwise it will segfault. I would also
>>>> use absolute path to make sure, ie. not *
>>>> *binarised-model/phrase-table *
>>>>
>>>> *but *
>>>>
>>>>
>>>> * /home/shubham/moses/binarised-model/phrase-table *
>>>>
>>>> *The previous email exchange also says you should try to add the
>>>> argument *
>>>>
>>>>
>>>> * -T . *
>>>>
>>>> Hieu Hoang
>>>> http://www.hoang.co.uk/hieu
>>>>
>>>> On 8 December 2016 at 15:52, Shubham Khandelwal <skhlnmiit@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> This is just the reminder of my previous email.
>>>>>
>>>>> Thanking You.
>>>>>
>>>>> Regards,
>>>>> Shubham
>>>>>
>>>>> On Thu, Dec 8, 2016 at 9:04 AM, Shubham Khandelwal <
>>>>> skhlnmiit@gmail.com> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I have just downloaded phrase-table.2.gz (18GB) de-en model
>>>>>> and phrase-table.3.gz (22GB) fr-en model from the available pre-made
>>>>>> models.
>>>>>> Now, I am converting them to PhraseDictionaryCompact using following
>>>>>> command (for exmaple):
>>>>>>
>>>>>>
>>>>>> *~/mosesdecoder/bin/processPhraseTableMin -threads all -in
>>>>>> ~/model/phrase-table.3.gz -nscores 4 -out binarised-model/phrase-table *
>>>>>>
>>>>>> But after passing 1/3, it gave following segementation fault error:
>>>>>>
>>>>>> *Pass 1/3: Creating hash function for rank assignment*
>>>>>> *Segmentation fault (core dumped)*
>>>>>>
>>>>>> I have found almost same issue on this thread:
>>>>>> http://comments.gmane.org/gmane.comp.nlp.moses.user/13033
>>>>>> However, I have provided the existing *binarised-model *folder in
>>>>>> the command. Also, I have the write-access in /tmp but still it gave
>>>>>> sementation fault.
>>>>>>
>>>>>> Can you please tell me what could be wrong here ?
>>>>>>
>>>>>> Thanking You.
>>>>>>
>>>>>> Regards,
>>>>>> Shubham
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20161210/938dfc18/attachment.html
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 122, Issue 16
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 122, Issue 16"
Post a Comment