Moses-support Digest, Vol 114, Issue 60

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: kenlm multithreading (Kenneth Heafield)
2. Getting error in building (Pranavan Theivendiram)
3. Phrases containing brackets mistaken for malformed
nonterminals (Markus Saers)


----------------------------------------------------------------------

Message: 1
Date: Thu, 28 Apr 2016 09:00:06 +0100
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] kenlm multithreading
To: moses-support@mit.edu
Message-ID: <5721C306.4060809@kheafield.com>
Content-Type: text/plain; charset=windows-1252

Queries are too fine-grained. The application i.e. Moses takes care of
threads. Regarding the command-line query program, it isn't
multi-threaded but probably should be. That said, people who care about
performance should be using the library instead of the command-line
program to avoid string processing overhead.

Kenneth

On 04/28/2016 02:34 AM, koormoosh wrote:
> Hi, Thanks - Is this happening else where, i.e. in the query step?
>
> On Thu, Apr 28, 2016 at 2:01 AM, <moses-support-request@mit.edu
> <mailto:moses-support-request@mit.edu>> wrote:
>
> Send Moses-support mailing list submissions to
> moses-support@mit.edu <mailto:moses-support@mit.edu>
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.mit.edu/mailman/listinfo/moses-support
> or, via email, send a message with subject or body 'help' to
> moses-support-request@mit.edu
> <mailto:moses-support-request@mit.edu>
>
> You can reach the person managing the list at
> moses-support-owner@mit.edu <mailto:moses-support-owner@mit.edu>
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Moses-support digest..."
>
>
> Today's Topics:
>
> 1. Call for Participation: IEEE DIPDMWC2016 Moscow, Russia
> (Sandra Evans)
> 2. kenlm multithreading (koormoosh)
> 3. Re: kenlm multithreading (Kenneth Heafield)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 27 Apr 2016 18:03:34 +0800
> From: Sandra Evans <sandra.sdiwc@gmail.com
> <mailto:sandra.sdiwc@gmail.com>>
> Subject: [Moses-support] Call for Participation: IEEE DIPDMWC2016
> Moscow, Russia
> To: moses-support@mit.edu <mailto:moses-support@mit.edu>
> Message-ID:
>
> <CAEjUHKVizroxu4v0xy4R0uWfHM_qfZrFSi_rNcL0dkNuJ4TVmQ@mail.gmail.com
> <mailto:CAEjUHKVizroxu4v0xy4R0uWfHM_qfZrFSi_rNcL0dkNuJ4TVmQ@mail.gmail.com>>
> Content-Type: text/plain; charset="utf-8"
>
> 2016 Third International Conference on Digital Information
> Processing, Data
> Mining, and Wireless Communications (DIPDMWC)
>
> National Research Nuclear University MEPhI (Moscow Engineering Physics
> Institute), Moscow, Russia
>
> July 06-08, 2016
>
> http://goo.gl/6HTGyi
> ===================================================================================================================
>
> The conference welcomes papers on the following (but not limited to)
> research topics:
>
> - Adaptive Signal Processing
> - Parallel Programming & Processing
> - Artificial Intelligence
> - Expert Systems
> - Image Processing
> - Information Security and Cryptography
> - Modulation, Coding, and Channel Analysis
> - Multimedia Signal Processing
> - Bioinformatics & Biomedical Imaging
> - Biomedical Signal Processing
> - Natural Language Processing
> - Neural Networks and Genetic Algorithms
> - Computer-Aided Surgery
> - Data Compression and Watermarking
> - Data Mining Techniques
> - Ethics of Data Mining
> - Risk Management and Analysis
> - Data Classification and Clustering
> - Abnormally and Outlier Detection
> - Feature Extraction and Data Reduction
> - Multi-Task Learning
> - Optimization Techniques
> - Data Cleaning and Processing
> - Text and Web Mining
> - Bluetooth and Personal Area Networks
> - Wireless System Architecture
> - Mobile Management in Wireless Networks
> - Mobile Database Access and Design
> - IP Multimedia Sub-Systems
> - Key Management Protocols
> - Mobile/ Wireless Network Modeling and Simulation
> - Mobile / Wireless Network Planning
> - Wireless Network Standard and Protocols
> - Digital Right Management and Multimedia Protection
> and many more...
>
> ===================================================================================================================
> Important Dates
>
> Submission Dates Open from now until June 6, 2016
> Notification of Acceptance 2-4 weeks from the submission date
> Camera Ready Submission June 26, 2016
> Registration Deadline June 26, 2016
> Conference Dates July 6-8, 2016
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.mit.edu/mailman/private/moses-support/attachments/20160427/6bb03766/attachment-0001.html
>
> ------------------------------
>
> Message: 2
> Date: Thu, 28 Apr 2016 00:25:28 +1000
> From: koormoosh <koormoosh@gmail.com <mailto:koormoosh@gmail.com>>
> Subject: [Moses-support] kenlm multithreading
> To: moses-support@mit.edu <mailto:moses-support@mit.edu>
> Message-ID:
>
> <CAN3_CDg_n4vS1_SUOjPNaYSQr06WFC09iX4szbNdf1yHpxKRfg@mail.gmail.com
> <mailto:CAN3_CDg_n4vS1_SUOjPNaYSQr06WFC09iX4szbNdf1yHpxKRfg@mail.gmail.com>>
> Content-Type: text/plain; charset="utf-8"
>
> Hello,
>
> Out of curiosity, does anyone know if multithreading is enabled by
> default
> during the execution of lmplz of kenlm? I noticed the lmplz cpu
> usage goes
> up and down around 300% (sometimes). I didn't pass any parameter to
> enable
> multithreading in lmplz and couldn't find any parameter on the
> website so I
> assume it's done internally and by default. Does anyone know when it
> decides to do multithreading?
>
> regards,
> -k
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://mailman.mit.edu/mailman/private/moses-support/attachments/20160427/334b0cd7/attachment-0001.html
>
> ------------------------------
>
> Message: 3
> Date: Wed, 27 Apr 2016 15:54:43 +0100
> From: Kenneth Heafield <moses@kheafield.com
> <mailto:moses@kheafield.com>>
> Subject: Re: [Moses-support] kenlm multithreading
> To: moses-support@mit.edu <mailto:moses-support@mit.edu>
> Message-ID: <5720D2B3.6010408@kheafield.com
> <mailto:5720D2B3.6010408@kheafield.com>>
> Content-Type: text/plain; charset=windows-1252
>
> Yes, it uses threads when it wants to. There is no option to turn
> threads off (and no code path that would do so). One has limited
> control using block size and counts. Ideally it would be more parallel.
>
> Kenneth
>
> On 04/27/2016 03:25 PM, koormoosh wrote:
> > Hello,
> >
> > Out of curiosity, does anyone know if multithreading is enabled by
> > default during the execution of lmplz of kenlm? I noticed the
> lmplz cpu
> > usage goes up and down around 300% (sometimes). I didn't pass any
> > parameter to enable multithreading in lmplz and couldn't find any
> > parameter on the website so I assume it's done internally and by
> > default. Does anyone know when it decides to do multithreading?
> >
> > regards,
> > -k
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
>
>
> ------------------------------
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> End of Moses-support Digest, Vol 114, Issue 58
> **********************************************
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

Message: 2
Date: Thu, 28 Apr 2016 15:57:11 +0530
From: Pranavan Theivendiram <pranavan.11@cse.mrt.ac.lk>
Subject: [Moses-support] Getting error in building
To: Moses-support@mit.edu
Message-ID:
<CAFJPyTyiY6Nq0uGXrBbwqBE8rWPpowDfL3staAoLpcA6SY+mjQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Dear devs,

I ran the following command in my ubuntu machine.

make -f contrib/Makefiles/install-dependencies.gmake

I am gettting the following errors. Please find the attachment for errors.
Please help me on this issue.

Thanks
*T. Pranavan*
*BSc Eng Undergraduate| Department of Computer Science & Engineering
,University of Moratuwa*
*Mobile| *0775136836
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160428/20369cb1/attachment-0001.html
-------------- next part --------------
mkdir -p '/home/pranavan/A-T/Translation Project/Softwares/MosesNew/mosesdecoder/build/xmlrpc' && cd '/home/pranavan/A-T/Translation Project/Softwares/MosesNew/mosesdecoder/build/xmlrpc' && wget -qO- http://downloads.sourceforge.net/project/xmlrpc-c/Xmlrpc-c%20Super%20Stable/1.33.17/xmlrpc-c-1.33.17.tgz | tar xz
cd '/home/pranavan/A-T/Translation Project/Softwares/MosesNew/mosesdecoder/build/xmlrpc/xmlrpc-c-1.33.17' && ./configure --prefix='/home/pranavan/A-T/Translation Project/Softwares/MosesNew/mosesdecoder/opt' && make -j && make install
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking whether make sets $(MAKE)... yes
checking for working aclocal... found
checking for working autoconf... found
checking for working automake... found
checking for working autoheader... found
checking for working makeinfo... missing
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for wininet-config... no
configure: You don't appear to have Wininet installed (no working wininet-config in your command search path), so we will not build the Wininet client XML transport
checking whether to build Wininet client XML transport module... no
checking for curl-config... yes
checking whether to build Curl client XML transport module... yes
checking for libwww-config... no
configure: You don't appear to have Libwww installed (no working libwww-config in your command search path), so we will not build the Libwww client XML transport
checking whether to build Libwww client XML transport module... no
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for main in -lncurses... yes
checking for main in -lreadline... yes
checking whether to build tools... yes
checking whether to build the xmlrpc_pstream tool... yes
checking whether to build Abyss server module... yes
checking whether to build CGI server module... yes
checking whether to build C++ wrappers and tools... yes
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking for g++... g++
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking for socket... yes
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking wchar.h usability... yes
checking wchar.h presence... yes
checking for wchar.h... yes
checking sys/filio.h usability... no
checking sys/filio.h presence... no
checking for sys/filio.h... no
checking sys/ioctl.h usability... yes
checking sys/ioctl.h presence... yes
checking for sys/ioctl.h... yes
checking sys/select.h usability... yes
checking sys/select.h presence... yes
checking for sys/select.h... yes
checking stdarg.h usability... yes
checking stdarg.h presence... yes
checking for stdarg.h... yes
checking for size_t... yes
checking whether va_list is an array... yes
checking whether compiler has __attribute__... yes
checking for vsnprintf... yes
checking for wcsncmp... yes
checking for setgroups... yes
checking for asprintf... yes
checking for setenv... yes
checking for strtoll... yes
checking for strtoull... yes
checking for strtoq... yes
checking for strtouq... yes
checking for __strtoll... no
checking for __strtoull... no
checking for pselect... yes
checking for gettimeofday... yes
checking for localtime_r... yes
checking for gmtime_r... yes
checking for strcasecmp... yes
checking for stricmp... no
checking for _stricmp... no
checking whether to use Abyss pthread function... yes
checking for curl-xmlrpc-config... no
checking for curl-config... /usr/bin/curl-config
checking for Curl library directory... /usr/lib
checking whether to use SSL with libwww... no
checking whether to build the libxml2 backend... no
configure: creating ./config.status
config.status: creating srcdir.mk
config.status: creating config.mk
config.status: creating xmlrpc_config.h
config.status: creating xmlrpc_amconfig.h
config.status: xmlrpc_amconfig.h is unchanged
config.status: executing default-1 commands
make[1]: Entering directory `/home/pranavan/A-T/Translation Project/Softwares/MosesNew/mosesdecoder/build/xmlrpc/xmlrpc-c-1.33.17'
make[1]: *** /home/pranavan/A-T/Translation: Is a directory. Stop.
make[1]: Leaving directory `/home/pranavan/A-T/Translation Project/Softwares/MosesNew/mosesdecoder/build/xmlrpc/xmlr

------------------------------

Message: 3
Date: Tue, 26 Apr 2016 10:24:48 +0800
From: Markus Saers <masaers@gmail.com>
Subject: [Moses-support] Phrases containing brackets mistaken for
malformed nonterminals
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAE3Z42W0+jS6QmAFp3x1YChiU7Jwzfdzd8czaw4c3ui42082nw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hello,

I am having problems reading in a phrase table derived from a corpus
that (I have learned now) contained bracketed expressions such as "to
like [someone]". I appears that Moses confuses these string with
nonterminals. I built a regular phrase-based model, so I was a bit
confused when it contained malformed nonterminals. Is there any way to
tell Moses that this is a regular phrase-based model, and that it
(he?) shouldn't look for nonterminals?

/Markus


The error message I got was:

Exception: moses/Phrase.cpp:214 in void
Moses::Phrase::CreateFromString(Moses::FactorDirection, const
std::vector<long unsigned int>&, const StringPiece&, Moses::Word**)
threw util::Exception because `nextPos == string::npos'.
Incorrect formatting of non-terminal. Should have 2 non-terms, eg.
[X][X]. Current string: [someone]
Exit code: 1

And my moses.ini contains the following:

#########################
### MOSES CONFIG FILE ###
#########################

# input factors
[input-factors]
0

# mapping steps
[mapping]
0 T 0

[distortion-limit]
6

# feature functions
[feature]
UnknownWordPenalty
WordPenalty
PhrasePenalty
PhraseDictionaryMemory name=TranslationModel0 num-features=4
path=path/to/phrase-table.gz input-factor=0 output-factor=0
LexicalReordering name=LexicalReordering0 num-features=6
type=wbe-msd-bidirectional-fe-allff
input-factor=0 output-factor=0
path=path/to/reordering-table.wbe-msd-bidirectional-fe.gz
Distortion
KENLM name=LM0 factor=0 path=path/to/lm.arpa order=4

# dense weights for feature functions
[weight]
UnknownWordPenalty0= 1
WordPenalty0= -1
PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2
LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3
Distortion0= 0.3
LM0= 0.5


------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 114, Issue 60
**********************************************

Tweet