Moses-support Digest, Vol 100, Issue 98

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: kbmira segfault (Matt Post)
2. coredump (fatma elzahraa Eltaher)
3. KenLM memory requirement (koormoosh)
4. binarise arabic LM Error (mohamed hasanien)


----------------------------------------------------------------------

Message: 1
Date: Fri, 27 Feb 2015 13:21:07 -0500
From: Matt Post <post@cs.jhu.edu>
Subject: Re: [Moses-support] kbmira segfault
To: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Cc: moses-support@mit.edu
Message-ID: <03ED8281-1FCB-4C77-B273-96B558B19101@cs.jhu.edu>
Content-Type: text/plain; charset="windows-1252"

Although, those old successful runs might have been with the old Moses kbmira. I'll look into this and report back.

matt


> On Feb 27, 2015, at 12:19 PM, Matt Post <post@cs.jhu.edu> wrote:
>
> Hi Barry ? Thanks for the response. I don't think that's it, because I use the exact same approach for lots of other tuning runs. Isn't it the header line of the features file that lists dense features? I've been using this format, where dense features are listed in each header line, and then sparse features in the individual lines:
>
> FEATURES_TXT_BEGIN_0 0 300 9 lm_0 lm_1 tm_pt_1 tm_pt_3 tm_pt_0 tm_pt_2 WordPenalty PhrasePenalty Distortion
> -82.183 -72.639 -79.162 -41.493 -60.118 -28.509 -10.857 19 -8
> -82.183 -72.639 -79.162 -41.493 -60.118 -28.509 -10.857 19 -8 OOVPenalty=-100
>
> This works in lots of places (although, it also raises a separate question, of whether kbmira actually distinguishes between sparse and dense features? I seem to remember Colin once saying that there is a single group weight between the two groups, but I've never been able to find this in the code).
>
> matt
>
>
>> On Feb 26, 2015, at 5:35 PM, Barry Haddow <bhaddow@staffmail.ed.ac.uk <mailto:bhaddow@staffmail.ed.ac.uk>> wrote:
>>
>> Hi Matt
>>
>> When mert-moses.pl runs kbmira, it always supplies a list of the dense features (and their initial values) using the --dense-init parameter. I think this is your problem. I've attached a typical file used for this feature list.
>>
>> Of course, kbmira should have a sensible message rather than a segfault. This is probably my doing,
>>
>> cheers - Barry
>>
>> On 26/02/15 22:18, Matt Post wrote:
>>> kbmira segfaults on the following command:
>>>
>>>
>>> kbmira run --ffile run1.features.dat --scfile run1.scores.dat -o mert.out
>>>
>>> Where run1.features.dat (30 MB) and run1.scores.dat (14 MB) can be downloaded here:
>>>
>>>
>>> https://www.dropbox.com/s/yim7ub1bmq5jv2g/run1.features.dat?dl=0 <https://www.dropbox.com/s/yim7ub1bmq5jv2g/run1.features.dat?dl=0>
>>>
>>> https://www.dropbox.com/s/kkek36o7aflgzuu/run1.scores.dat?dl=0 <https://www.dropbox.com/s/kkek36o7aflgzuu/run1.scores.dat?dl=0>
>>>
>>> I tracked it down to this line of mert/FeatureStats.cpp.
>>>
>>> std::string SparseVector::decode(std::size_t id)
>>> {
>>> return m_id_to_name[id];
>>> }
>>>
>>> Any obvious ideas before I go down this rabbit hole? I verified there are no blank lines or anything else funny with the formatting, at least as far as I can tell (all dense features, plus one sparse feature, OOVPenalty=-100, showing up occasionally).
>>>
>>> matt
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support <http://mailman.mit.edu/mailman/listinfo/moses-support>
>>
>> <run1.dense>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150227/be86fccd/attachment-0001.htm

------------------------------

Message: 2
Date: Fri, 27 Feb 2015 14:22:47 -0800
From: fatma elzahraa Eltaher <fatmaeltaher@gmail.com>
Subject: [Moses-support] coredump
To: moses-support@mit.edu
Message-ID:
<CAOW1BbRQvTY9QXHehmZuR2QJkQVk5jK2uK9pYWCj40zG+PggdQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

I try to run example in manual : and in this step 2.3.5 Training the
Translation System
I run the following command
ohup nice
/home/fatma/Desktop/Folder_Moses/mosesdecoder/scripts/training/train-model.perl
-root-dir train \
-corpus /home/fatma/Desktop/corpus/news-commentary-v8.fr-en.clean \
-f fr -e en -alignment grow-diag-final-and -reordering msd-bidirectional-fe
\
-lm 0:3:/home/fatma/Desktop/corpus/lm/news-commentary-v8.fr-en.blm.en:8 \
-external-bin-dir /home/fatma/Desktop/external-bin >& training.out &
but I get error .
I attached
training.out
<https://mail.google.com/mail/u/0/?ui=2&ik=087e92bbdb&view=att&th=14bcd21c3b99930e&attid=0.1&disp=safe&realattid=f_i6o56fld0&zw>


thank you,



Fatma El-Zahraa El -Taher

Teaching Assistant at Computer & System department

Faculty of Engineering, Azhar University

Email : fatmaeltaher@gmail.com
mobile: +201141600434
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150227/18cee88d/attachment-0001.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: training.out
Type: application/octet-stream
Size: 959 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20150227/18cee88d/attachment-0001.obj

------------------------------

Message: 3
Date: Sat, 28 Feb 2015 15:00:37 +1100
From: koormoosh <koormoosh@gmail.com>
Subject: [Moses-support] KenLM memory requirement
To: moses-support@mit.edu
Message-ID:
<CAN3_CDiUd__iGvOcJAE02kUoGKz6gkAeH5-PwTe6zGjOKMZQxA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

So, I am doing some experiments with giga word 4 Ed. It seems that for
running kenLM on 7 GB of data, I need to have more than 64 GB of RAM. Is
there an approximation on how much of a RAM is required with respect to the
size of training data to be processed?

Thanks,
Koorm
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150228/8e654cdb/attachment-0001.htm

------------------------------

Message: 4
Date: Sat, 28 Feb 2015 12:09:24 +0000 (UTC)
From: mohamed hasanien <mhmd_hasnen@yahoo.com>
Subject: [Moses-support] binarise arabic LM Error
To: Moses-support Support <moses-support@mit.edu>
Message-ID:
<1901551769.232230.1425125364145.JavaMail.yahoo@mail.yahoo.com>
Content-Type: text/plain; charset="utf-8"

Hi all,?
i try to binarise the?data.arpa.ar ( arabic language)
using this command?
?~/mosesdecoder/bin/build_binary ? ?data.arpa.ar ? ?data.blm.ar
iget this error?
------------------------------------------------?Reading data.arpa.ar----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100****************************************************************************************************lm/read_arpa.cc:151 in void lm::PositiveProbWarn::Warn(float) threw FormatLoadException'.Positive log probability 9.37143e-08 in the model. ?This is a bug in IRSTLM; you can set config.positive_log_probability = SILENT or pass -i to build_binary to substitute 0.0 for the log probability. ?Error in the 3-gram at byte 65275919 Byte: 65275919ERROR
what i can do to solve this issue?thanks?mohammed hassanien Mohammed
Egyption Programmers Vice-captain
01000121556
Egyption Programmers Syndicate
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150228/5509f1bd/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 100, Issue 98
**********************************************

Tweet