Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Failure to Open Output when using Chart Decoder (Shyam Upadhyay)
2. Re: g++: error: unrecognized command line option
'-no-cpp-precomp' (Jeroen Vermeulen)
3. Re: Decoding Speed perfomance - suggestion and question
(Vincent Nguyen)
----------------------------------------------------------------------
Message: 1
Date: Mon, 31 Aug 2015 19:36:11 -0400
From: Shyam Upadhyay <shyamupa@gmail.com>
Subject: [Moses-support] Failure to Open Output when using Chart
Decoder
To: moses-support@mit.edu
Message-ID:
<CAAet1GvzomG+a1fpxxoKuCycJ7imnRa27BqspgtXVUykwrUfcA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
I am new to using moses and I am trying to use the chart decoder to obtain
100 best decodings as follows,
moses/bin/moses_chart -f mymodel/moses.ini --drop-unknown --n-best-list
myout/hyp.mrl.nbest 100
I encounter the following error,
Start loading text phrase table. Moses format : [0.009] seconds
Reading
/home/upadhya3/smt-semparse-fresh/work/2015-08-30T21.06.43/model/glue-grammar
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
max-chart-span: 20
max-chart-span: 1000
Created input-output object : [0.009] seconds
Exception: ./moses/OutputCollector.h:64 in
Moses::OutputCollector::OutputCollector(std::string, std::string) threw
util::Exception because `!m_outStream->good()'.
Failed to open output
file/home/upadhya3/smt-semparse-fresh/work/2015-08-30T19.44.07/hyp.mrl.nbest
My moses.ini file is, (this was generated automatically by previous steps)
#########################
### MOSES CONFIG FILE ###
#########################
# input factors
[input-factors]
0
# mapping steps
[mapping]
0 T 0
1 T 1
[cube-pruning-pop-limit]
1000
[non-terminals]
X
[search-algorithm]
3
[inputtype]
3
[max-chart-span]
20
1000
# feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryMemory name=TranslationModel0 num-features=4
path=/home/upadhya3/smt-semparse-fresh/work/2015-08-30T21.06.43/model/rule-table.gz
input-factor=0 output-factor=0
PhraseDictionaryMemory name=TranslationModel1 num-features=1
path=/home/upadhya3/smt-semparse-fresh/work/2015-08-30T21.06.43/model/glue-grammar
input-factor=0 output-factor=0 tuneable=true
KENLM name=LM0 factor=0
path=/home/upadhya3/smt-semparse-fresh/work/2015-08-30T21.06.43/mrl.arpa
order=3
# dense weights for feature functions
[weight]
# The default weights are NOT optimized for translation quality. You MUST
tune the weights.
# Documentation for tuning is here:
http://www.statmt.org/moses/?n=FactoredTraining.Tuning
UnknownWordPenalty0= 1
WordPenalty0= -1
PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2
TranslationModel1= 1
LM0= 0.5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150831/5789bf98/attachment-0001.html
------------------------------
Message: 2
Date: Tue, 01 Sep 2015 08:46:14 +0700
From: Jeroen Vermeulen <jtv@precisiontranslationtools.com>
Subject: Re: [Moses-support] g++: error: unrecognized command line
option '-no-cpp-precomp'
To: Jorg Tiedemann <tiedeman@gmail.com>, Hieu Hoang
<hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<82206CE3-7E3E-43EE-8FBD-06AF3FFB4BE1@precisiontranslationtools.com>
Content-Type: text/plain; charset="utf-8"
Coming in with very little knowledge of the context here, but it looks to me as if you have a Boost that was built in a slightly different compiler configuration than the main program.
It looks as if maybe your Boost was built with a pre-5.x gcc, or without C++11 enabled.
Jeroen
On August 31, 2015 9:14:10 PM GMT+07:00, Jorg Tiedemann <tiedeman@gmail.com> wrote:
>
>Well, I have /opt/local/ search paths in various environment variables
>to get macports to work.
>I deleted all this paths and tried again but I still get the same
>problem.
>
>I am confused. And why is gcc not working anymore when installed via
>macports? I also installed boost with macports. Is that a problem as
>well?
>
>I have also some problems with kenlm but part of it compiles and links
>fine. build_binary and query seems to compile fine but lmplz does not
>link because of some undefined symbols:
>Undefined symbols for architecture x86_64:
>"boost::program_options::value_semantic_codecvt_helper<char>::parse(boost::any&,
>std::vector<std::__cxx11::basic_string<char, std::char_traits<char>,
>std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char,
>std::char_traits<char>, std::allocator<char> > > > const&, bool)
>const", referenced from:
>?.
>
>I also had to link /opt/local/lib to /opt/local/lib64 (which didn?t
>exist in my setup).
>I am afraid that I started to make quite a mess on my system but what
>did I do wrong?
>
>Is macports not working properly anymore?
>As I said, I have gcc 5.2.0 and boost 1.59.0 via macports on my system.
>Is that bad?
>
>Thanks for helping!
>J?rg
>
>
>
>
>> On 31 Aug 2015, at 16:19, Hieu Hoang <hieuhoang@gmail.com> wrote:
>>
>> the errors for clang looks like it's coming from the stl library.
>Have you fiddled with the PATH variable or otherwise tried to make gcc
>on OSX work? You shouldn't do that, it will just mess up the
>compilation environment on your machine
>>
>> On 31/08/2015 10:28, Jorg Tiedemann wrote:
>>>
>>> Unfortunately, this didn?t work for me either. I attach both logiles
>- one for clang and one for gcc (which I installed via macports)
>>> What can I do? Thanks!
>>>
>>> J?rg
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>> On 30 Aug 2015, at 11:33, Hieu Hoang <
><mailto:hieuhoang@gmail.com>hieuhoang@gmail.com
><mailto:hieuhoang@gmail.com>> wrote:
>>>>
>>>> Add
>>>> toolset=clang
>>>> to the bjam compile command. Osx no longer has gcc
>>>>
>>>> Hieu Hoang
>>>> Sent while bumping into things
>>>>
>>>> On 29 Aug 2015 11:56 pm, "Jorg Tiedemann" <tiedeman@gmail.com
><mailto:tiedeman@gmail.com>> wrote:
>>>> Hi,
>>>>
>>>> I tried to make a fresh install of Moses on my new Mac and I get
>the following error
>>>> g++: error: unrecognized command line option '-no-cpp-precomp'
>>>>
>>>> What?s wrong? I have gcc5 and boost 1.59 on my machine via macports
>...
>>>>
>>>> Thanks for your help!
>>>> J?rg
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
><http://mailman.mit.edu/mailman/listinfo/moses-support>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
><http://mailman.mit.edu/mailman/listinfo/moses-support>
>>>
>>
>> --
>> Hieu Hoang
>> Researcher
>> New York University, Abu Dhabi
>> http://www.hoang.co.uk/hieu <http://www.hoang.co.uk/hieu>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Moses-support mailing list
>Moses-support@mit.edu
>http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150831/d38eb479/attachment-0001.html
------------------------------
Message: 3
Date: Tue, 1 Sep 2015 14:04:19 +0200
From: Vincent Nguyen <vnguyen@neuf.fr>
Subject: Re: [Moses-support] Decoding Speed perfomance - suggestion
and question
To: Philipp Koehn <phi@jhu.edu>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <55E59443.9090901@neuf.fr>
Content-Type: text/plain; charset="utf-8"
I am not clear with the syntax of filter-model-given-input.pl
target-dir = where we want the filtered PT to go ?
moses.ini = if it is not in the above directory the script does accept it
input.txt = ? what is it in the case I just want to adjust the MinScore ?
Le 31/08/2015 16:54, Philipp Koehn a ?crit :
> Hi,
>
> the filter script filters an existing phrase table. With the EMS
> settings, it would build another phrase table.
>
> Don't worry about the reordering table. It will have excess entries,
> but they will not be used.
> If you really care, you can used the script
> scripts/training/remove-orphan-phrase-pairs-from-reordering-table.perl
>
> -phi
>
> On Mon, Aug 31, 2015 at 10:50 AM, Vincent Nguyen <vnguyen@neuf.fr
> <mailto:vnguyen@neuf.fr>> wrote:
>
>
> thanks, will try and post results.
> just to be clear:
> I can re-use the previous extract file
> I have to rebuild the phrase-table with new min score (ie no way
> to just filter the previous one ?)
> do I have to rebuild the reordering table too ?
>
> Vincent
>
>
> Le 31/08/2015 16:44, Philipp Koehn a ?crit :
>> hI,
>>
>> 0.0001 should have no impact on translation quality,
>> 0.001 will have some impact
>> 0.01 is probably a bit too drastic.
>>
>> But that's the range you should explore.
>>
>> -phi
>>
>> On Mon, Aug 31, 2015 at 10:33 AM, Vincent Nguyen <vnguyen@neuf.fr
>> <mailto:vnguyen@neuf.fr>> wrote:
>>
>> is there any benchmark on what value / what impact ?
>> what should I start with as a test 0.001 ?
>>
>> the standard value 0.0001 seems really really low to me ....
>> maybe I am not getting what this probability exactly refers to.
>>
>>
>>
>> where |FIELDn| is the position of the score (typically 2 for
>> the direct phrase probability p(e|f), or 0 for the indirect
>> phrase probability p(f|e)) and |THRESHOLD| the maximum
>> probability allowed. A good setting is |2:0.0001|, which
>> removes all rules, where the direct phrase translation
>> probability is below 0.0001.
>>
>>
>>
>> Le 31/08/2015 16:14, Philipp Koehn a ?crit :
>>> Hi,
>>>
>>> I would suspect that with beam sizes <500 the bulk of the
>>> time is
>>> spent on translation option collection, not decoding. You
>>> could speed
>>> that up with tighter threshold pruning of the phrase table.
>>>
>>> See the script scripts/training/threshold-filter.perl or the
>>> setting
>>> score-settings = "--MinScore 2:0.0001"
>>> in EMS.
>>>
>>> -phi
>>>
>>> On Mon, Aug 31, 2015 at 3:03 AM, Vincent Nguyen
>>> <vnguyen@neuf.fr <mailto:vnguyen@neuf.fr>> wrote:
>>>
>>> Hi,
>>>
>>> Here are some results with several values with cube
>>> pruning pop limit :
>>>
>>> (pop limit / decoding time for 3000 sentences / BLEU score)
>>>
>>> 5000 - 15m45 - 29.59
>>> 1000 - 4m27 - 29.59
>>> 500 - 3m35 - 29.59
>>> 200 - 3m15 - 29.51
>>> 100 - 3m00 - 29.40
>>>
>>> Therefore I took 400 - 3m19 - 29.58
>>>
>>> If I am not mistaken the default value for Moses is 1000
>>> [read in the
>>> doc] but in the EMS
>>> it is 5000 right now .... which makes the experience so
>>> long .....
>>> I suggest to change the EMS default value.
>>>
>>> Is there a way to also use a cube pruning limit in the
>>> decoder at Tuning
>>> time ?
>>>
>>> Now with this optimized setting I get a ration of 15
>>> segments per second
>>> in average.
>>> What is the reason for online tools like Google / Bing
>>> to be much much
>>> faster.
>>> it's not a machine issue, is it ?
>>>
>>>
>>> Cheers
>>> Vincent
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150901/da5b9ae2/attachment.html
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 107, Issue 1
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 107, Issue 1"
Post a Comment