Moses-support Digest, Vol 101, Issue 20

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: kbmira segfault (Matt Post)
2. mert-moses.pl (mohamed hasanien)
3. Re: kbmira segfault (Barry Haddow)
4. compilation problems with the last version of moses
(Arezki Sadoune)


----------------------------------------------------------------------

Message: 1
Date: Thu, 5 Mar 2015 14:33:10 -0500
From: Matt Post <post@cs.jhu.edu>
Subject: Re: [Moses-support] kbmira segfault
To: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Cc: moses-support@mit.edu
Message-ID: <F4AD092F-BE79-4D46-96F0-97998DBFC6F3@cs.jhu.edu>
Content-Type: text/plain; charset=windows-1252

Yes, passing --dense-init worked. Although, it seems to ignore the feature names: so long as I have enough lines matching the number of dense parameters, it works, and it always outputs the following:

477/3000 updates, avg loss = 0.36341, BLEU = 0.356527
F0 3.663
F1 0.221152
F2 0.186323
F3 1.41851
F4 2.38853
F5 -0.162657
F6 0.430753
F7 3.93281

Does that sound correct?


> On Mar 5, 2015, at 10:34 AM, Barry Haddow <bhaddow@staffmail.ed.ac.uk> wrote:
>
> Hi Matt
>
> This was part of the changes to support hypergraph mira, since the hypergraphs don't have the FEATURES_TXT_BEGIN_0 sections. In fact they don't differentiate between sparse and dense features.
>
> Does it work correctly when you use the --dense-init paramater?
>
> cheers - Barry
>
> On 05/03/15 15:18, Matt Post wrote:
>> Okay, the old kbmira works, so this must be part of the 3.0 changes.
>>
>> It seems that the names of features in the header line (FEATURES_TXT_BEGIN_0) are ignored entirely. The 2.1 kbmira would output dense feature weights using names F1..FN, which I would then re-map back to the list in the header. In kbmira 3.0, it uses the file passed in, as Barry pointed out.
>>
>> Thanks for your help!
>>
>> matt
>>
>>
>>> On Feb 27, 2015, at 1:21 PM, Matt Post <post@cs.jhu.edu <mailto:post@cs.jhu.edu>> wrote:
>>>
>>> Although, those old successful runs might have been with the old Moses kbmira. I'll look into this and report back.
>>>
>>> matt
>>>
>>>
>>>> On Feb 27, 2015, at 12:19 PM, Matt Post <post@cs.jhu.edu <mailto:post@cs.jhu.edu>> wrote:
>>>>
>>>> Hi Barry ? Thanks for the response. I don't think that's it, because I use the exact same approach for lots of other tuning runs. Isn't it the header line of the features file that lists dense features? I've been using this format, where dense features are listed in each header line, and then sparse features in the individual lines:
>>>>
>>>> FEATURES_TXT_BEGIN_0 0 300 9 lm_0 lm_1 tm_pt_1 tm_pt_3 tm_pt_0 tm_pt_2 WordPenalty PhrasePenalty Distortion
>>>> -82.183 -72.639 -79.162 -41.493 -60.118 -28.509 -10.857 19 -8
>>>> -82.183 -72.639 -79.162 -41.493 -60.118 -28.509 -10.857 19 -8 OOVPenalty=-100
>>>>
>>>> This works in lots of places (although, it also raises a separate question, of whether kbmira actually distinguishes between sparse and dense features? I seem to remember Colin once saying that there is a single group weight between the two groups, but I've never been able to find this in the code).
>>>>
>>>> matt
>>>>
>>>>
>>>>> On Feb 26, 2015, at 5:35 PM, Barry Haddow <bhaddow@staffmail.ed.ac.uk <mailto:bhaddow@staffmail.ed.ac.uk>> wrote:
>>>>>
>>>>> Hi Matt
>>>>>
>>>>> When mert-moses.pl runs kbmira, it always supplies a list of the dense features (and their initial values) using the --dense-init parameter. I think this is your problem. I've attached a typical file used for this feature list.
>>>>>
>>>>> Of course, kbmira should have a sensible message rather than a segfault. This is probably my doing,
>>>>>
>>>>> cheers - Barry
>>>>>
>>>>> On 26/02/15 22:18, Matt Post wrote:
>>>>>> kbmira segfaults on the following command:
>>>>>>
>>>>>> kbmira run --ffile run1.features.dat --scfile run1.scores.dat -o mert.out
>>>>>>
>>>>>> Where run1.features.dat (30 MB) and run1.scores.dat (14 MB) can be downloaded here:
>>>>>>
>>>>>> https://www.dropbox.com/s/yim7ub1bmq5jv2g/run1.features.dat?dl=0
>>>>>> https://www.dropbox.com/s/kkek36o7aflgzuu/run1.scores.dat?dl=0
>>>>>>
>>>>>> I tracked it down to this line of mert/FeatureStats.cpp.
>>>>>>
>>>>>> std::string SparseVector::decode(std::size_t id)
>>>>>> {
>>>>>> return m_id_to_name[id];
>>>>>> }
>>>>>>
>>>>>> Any obvious ideas before I go down this rabbit hole? I verified there are no blank lines or anything else funny with the formatting, at least as far as I can tell (all dense features, plus one sparse feature, OOVPenalty=-100, showing up occasionally).
>>>>>>
>>>>>> matt
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Moses-support mailing list
>>>>>> Moses-support@mit.edu
>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>
>>>>> <run1.dense>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.




------------------------------

Message: 2
Date: Thu, 5 Mar 2015 19:55:32 +0000 (UTC)
From: mohamed hasanien <mhmd_hasnen@yahoo.com>
Subject: [Moses-support] mert-moses.pl
To: Moses-support Support <moses-support@mit.edu>
Message-ID:
<1476809164.5230144.1425585332553.JavaMail.yahoo@mail.yahoo.com>
Content-Type: text/plain; charset="utf-8"

HI all,?i try to train english arabic system and every thing is ok untill i try to tuning the system using this command?
?nohup nice ~/mosesdecoder/scripts/training/mert-moses.pl ? ~/thesiscorups/tuning.true.en ~/thesiscorups/tuning.true.ar ?~/mosesdecoder/bin/moses train/model/moses.ini --mertdir ~/mosesdecoder/bin/i found in the mert-work folder only one itration and ?run1.moses.ini
and i also open the out file i found the following lines in the end of the file?---------------------------------------
BEST TRANSLATION: ???? ??????? ???????? ? [111111] ?[total=-1.972] core=(0.000,-4.000,1.000,-2.303,-17.198,-1.386,-10.604,-0.511,0.000,0.000,0.000,0.000,0.000,0.000,-1$Line 4738: Decision rule took 0.000 seconds totalLine 4738: Additional reporting took 0.020 seconds totalLine 4738: Translation took 0.308 seconds totalTranslating: takes note with appreciation of the technical cooperation activities carried out by the UNCTAD secretariat and of the reports prepared for the Working Par$Line 4739: Initialize search took 1.250 seconds totalLine 4739: Collecting options took 0.562 seconds at moses/Manager.cpp:117sh: line 1: 13550 Killed ? ? ? ? ? ? ? ? ?/mhmd/mosesdecoder/bin/moses -config filtered/moses.ini -weight-overwrite 'PhrasePenalty0= 0.043478 WordPenalty0= -0.217391 T$Exit code: 137The decoder died. CONFIG WAS -weight-overwrite 'PhrasePenalty0= 0.043478 WordPenalty0= -0.217391 TranslationModel0= 0.043478 0.043478 0.043478 0.043478 Distortion0= 0.$
can any one told me what is the problem and how i can solve it?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150305/9a27b76f/attachment-0001.htm

------------------------------

Message: 3
Date: Thu, 05 Mar 2015 20:48:16 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] kbmira segfault
To: Matt Post <post@cs.jhu.edu>
Cc: moses-support@mit.edu
Message-ID: <54F8C110.4000909@staffmail.ed.ac.uk>
Content-Type: text/plain; charset=windows-1252; format=flowed

Hi Matt

That seems right. When reading an nbest list, sparse and dense features
are stored differently so you just need to know how many there are,
whereas in hypergraphs all features look like sparse features. This
needs a cleanup ...

cheers - Barry

On 05/03/15 19:33, Matt Post wrote:
> Yes, passing --dense-init worked. Although, it seems to ignore the feature names: so long as I have enough lines matching the number of dense parameters, it works, and it always outputs the following:
>
> 477/3000 updates, avg loss = 0.36341, BLEU = 0.356527
> F0 3.663
> F1 0.221152
> F2 0.186323
> F3 1.41851
> F4 2.38853
> F5 -0.162657
> F6 0.430753
> F7 3.93281
>
> Does that sound correct?
>
>
>> On Mar 5, 2015, at 10:34 AM, Barry Haddow <bhaddow@staffmail.ed.ac.uk> wrote:
>>
>> Hi Matt
>>
>> This was part of the changes to support hypergraph mira, since the hypergraphs don't have the FEATURES_TXT_BEGIN_0 sections. In fact they don't differentiate between sparse and dense features.
>>
>> Does it work correctly when you use the --dense-init paramater?
>>
>> cheers - Barry
>>
>> On 05/03/15 15:18, Matt Post wrote:
>>> Okay, the old kbmira works, so this must be part of the 3.0 changes.
>>>
>>> It seems that the names of features in the header line (FEATURES_TXT_BEGIN_0) are ignored entirely. The 2.1 kbmira would output dense feature weights using names F1..FN, which I would then re-map back to the list in the header. In kbmira 3.0, it uses the file passed in, as Barry pointed out.
>>>
>>> Thanks for your help!
>>>
>>> matt
>>>
>>>
>>>> On Feb 27, 2015, at 1:21 PM, Matt Post <post@cs.jhu.edu <mailto:post@cs.jhu.edu>> wrote:
>>>>
>>>> Although, those old successful runs might have been with the old Moses kbmira. I'll look into this and report back.
>>>>
>>>> matt
>>>>
>>>>
>>>>> On Feb 27, 2015, at 12:19 PM, Matt Post <post@cs.jhu.edu <mailto:post@cs.jhu.edu>> wrote:
>>>>>
>>>>> Hi Barry ? Thanks for the response. I don't think that's it, because I use the exact same approach for lots of other tuning runs. Isn't it the header line of the features file that lists dense features? I've been using this format, where dense features are listed in each header line, and then sparse features in the individual lines:
>>>>>
>>>>> FEATURES_TXT_BEGIN_0 0 300 9 lm_0 lm_1 tm_pt_1 tm_pt_3 tm_pt_0 tm_pt_2 WordPenalty PhrasePenalty Distortion
>>>>> -82.183 -72.639 -79.162 -41.493 -60.118 -28.509 -10.857 19 -8
>>>>> -82.183 -72.639 -79.162 -41.493 -60.118 -28.509 -10.857 19 -8 OOVPenalty=-100
>>>>>
>>>>> This works in lots of places (although, it also raises a separate question, of whether kbmira actually distinguishes between sparse and dense features? I seem to remember Colin once saying that there is a single group weight between the two groups, but I've never been able to find this in the code).
>>>>>
>>>>> matt
>>>>>
>>>>>
>>>>>> On Feb 26, 2015, at 5:35 PM, Barry Haddow <bhaddow@staffmail.ed.ac.uk <mailto:bhaddow@staffmail.ed.ac.uk>> wrote:
>>>>>>
>>>>>> Hi Matt
>>>>>>
>>>>>> When mert-moses.pl runs kbmira, it always supplies a list of the dense features (and their initial values) using the --dense-init parameter. I think this is your problem. I've attached a typical file used for this feature list.
>>>>>>
>>>>>> Of course, kbmira should have a sensible message rather than a segfault. This is probably my doing,
>>>>>>
>>>>>> cheers - Barry
>>>>>>
>>>>>> On 26/02/15 22:18, Matt Post wrote:
>>>>>>> kbmira segfaults on the following command:
>>>>>>>
>>>>>>> kbmira run --ffile run1.features.dat --scfile run1.scores.dat -o mert.out
>>>>>>>
>>>>>>> Where run1.features.dat (30 MB) and run1.scores.dat (14 MB) can be downloaded here:
>>>>>>>
>>>>>>> https://www.dropbox.com/s/yim7ub1bmq5jv2g/run1.features.dat?dl=0
>>>>>>> https://www.dropbox.com/s/kkek36o7aflgzuu/run1.scores.dat?dl=0
>>>>>>>
>>>>>>> I tracked it down to this line of mert/FeatureStats.cpp.
>>>>>>>
>>>>>>> std::string SparseVector::decode(std::size_t id)
>>>>>>> {
>>>>>>> return m_id_to_name[id];
>>>>>>> }
>>>>>>>
>>>>>>> Any obvious ideas before I go down this rabbit hole? I verified there are no blank lines or anything else funny with the formatting, at least as far as I can tell (all dense features, plus one sparse feature, OOVPenalty=-100, showing up occasionally).
>>>>>>>
>>>>>>> matt
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Moses-support mailing list
>>>>>>> Moses-support@mit.edu
>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>> <run1.dense>
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> --
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>



------------------------------

Message: 4
Date: Fri, 6 Mar 2015 08:55:46 +0000 (UTC)
From: Arezki Sadoune <arezkisadoune@yahoo.fr>
Subject: [Moses-support] compilation problems with the last version of
moses
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<2146288457.6534861.1425632146626.JavaMail.yahoo@mail.yahoo.com>
Content-Type: text/plain; charset="utf-8"

Hello All
I have some problems compiling the last moses on Ubuntu 12.04.5.
-When compiling moses alone: ./bjam the issues encountered are :

...failed testing.unit-test moses/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/moses_test.passed...testing.unit-test mert/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/bleu_scorer_test.passed
...failed testing.unit-test mert/bin/gcc-4.6/release/debug-symbols-on/link-static/threading-multi/vocabulary_test.passed...


?-when I try to compile with boost_1_55_0 et CMPH-2.0./bjam?--with-cmph=/root/tools/cmph-2.0/?--with-boost=/root/tools/boost_1_55_0the errors preventing moses compiling are in the log file attached
Cheers

Arezki SADOUNE?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150306/8f273025/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log.gz
Type: application/gzip
Size: 11006 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20150306/8f273025/attachment.bin

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 101, Issue 20
**********************************************

Tweet