Moses-support Digest, Vol 82, Issue 48

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Building Moses with statically linked libraries?
(Nicola Bertoldi)


----------------------------------------------------------------------

Message: 1
Date: Wed, 28 Aug 2013 11:57:41 +0000
From: Nicola Bertoldi <bertoldi@fbk.eu>
Subject: Re: [Moses-support] Building Moses with statically linked
libraries?
To: Kenneth Heafield <moses@kheafield.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <58603717-3ED3-4D75-B532-956769F2C799@fbk.eu>
Content-Type: text/plain; charset="us-ascii"

Maybe this can help to solve the issue

I also noted that XMLRPC libraries is not statically link

this is what I got running ldd on moses first and mosesserve
ldd moses
linux-vdso.so.1 => (0x00007fff3f3a1000)
librt.so.1 => /lib64/librt.so.1 (0x0000003620a00000)
libSegFault.so => /lib64/libSegFault.so (0x00007fc75f667000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003620e00000)
libm.so.6 => /lib64/libm.so.6 (0x0000003620200000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003621200000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000361fe00000)
libc.so.6 => /lib64/libc.so.6 (0x000000361f600000)
/lib64/ld-linux-x86-64.so.2 (0x000000361f200000)

ldd mosesserver
linux-vdso.so.1 => (0x00007fffa59ff000)
librt.so.1 => /lib64/librt.so.1 (0x0000003620a00000)
libSegFault.so => /lib64/libSegFault.so (0x00007f70dbc55000)
libxmlrpc_server_abyss++.so.4 => not found
libxmlrpc_server++.so.4 => not found
libxmlrpc_server_abyss.so.3 => not found
libxmlrpc_server.so.3 => not found
libxmlrpc_abyss.so.3 => not found
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000361fe00000)
libxmlrpc++.so.4 => not found
libxmlrpc.so.3 => not found
libxmlrpc_util.so.3 => not found
libxmlrpc_xmlparse.so.3 => not found
libxmlrpc_xmltok.so.3 => not found
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x0000003620e00000)
libm.so.6 => /lib64/libm.so.6 (0x0000003620200000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000003621200000)
libc.so.6 => /lib64/libc.so.6 (0x000000361f600000)
/lib64/ld-linux-x86-64.so.2 (0x000000361f200000)



Nicola



On Aug 27, 2013, at 12:41 AM, Kenneth Heafield wrote:

> Hi,
>
> Ugh sorry something was weird with bjam. I've put a kludge in that
> forces static linkage unless link=shared appears on the command line.
>
> Kenneth
>
> On 08/26/13 21:59, Lane Schwartz wrote:
>> I'm attempting to compile Moses in such a way that at least the boost
>> libraries are statically compiled. I'm fine if other libraries are shared.
>>
>> My interpretation of this thread (and BUILD-INSTRUCTIONS.txt) is that if
>> I compile boost (following instructions in BUILD-INSTRUCTIONS.txt) so
>> that it makes available static versions of its libraries, then compile
>> moses using link=static, boost should be statically linked with moses.
>> Unfortunately, that doesn't appear to be happening for me:
>>
>> ./bjam link=static -j8 --with-cmph=/tools/moses/cmph-2.0
>> --with-srilm=/tools/SRILM/SRILM-1.7.0
>> --with-boost=/tools/moses/boost_1_53_0 -q --debug-configuration -d2
>>
>> The log for the above is attached. If I try running moses after this
>> compile (without adding /tools/moses/boost_1_53_0 to LIBRARY_PATH and
>> LD_LIBRARY_PATH), I get the following error:
>>
>> ./bin/moses: error while loading shared libraries:
>> libboost_system-mt.so.1.53.0: cannot open shared object file: No such
>> file or directory
>>
>> Running ldd on bin/moses confirms that there is a dynamic boost library
>> linking:
>>
>> linux-vdso.so.1 => (0x00007fff8adff000)
>>
>> libz.so.1 => /lib64/libz.so.1 (0x0000003949200000)
>>
>> librt.so.1 => /lib64/librt.so.1 (0x0000003949600000)
>>
>> libbz2.so.1 => /lib64/libbz2.so.1 (0x0000003955a00000)
>>
>> libboost_system-mt.so.1.53.0 => not found
>>
>> libSegFault.so => /lib64/libSegFault.so (0x00007f1671726000)
>>
>> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x000000394fa00000)
>>
>> libm.so.6 => /lib64/libm.so.6 (0x0000003948600000)
>>
>> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000000394d600000)
>>
>> libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003948e00000)
>>
>> libc.so.6 => /lib64/libc.so.6 (0x0000003948200000)
>>
>> /lib64/ld-linux-x86-64.so.2 (0x0000003947e00000)
>>
>>
>> Looking in /tools/moses/boost_1_53_0/lib I can verify that the static .a
>> file exists:
>>
>>
>> libboost_system-mt.a
>>
>> libboost_system-mt.so
>>
>> libboost_system-mt.so.1.53.0
>>
>>
>> Any help would be appreciated. I'm attaching the compile log and a
>> listing of the boost files installed in /tools/moses/boost_1_53_0/lib.
>>
>>
>> Thanks,
>>
>> Lane
>>
>>
>>
>>
>> On Mon, Aug 6, 2012 at 12:39 PM, Kenneth Heafield <moses@kheafield.com
>> <mailto:moses@kheafield.com>> wrote:
>>
>> Ok, committed. Here's how the build system now behaves:
>>
>> link=shared: Everything linked dynamically.
>>
>> Default: internal libraries are statically linked. Boost and zlib
>> statically linked if possible. libSegFault dynamically linked.
>> Dynamically linked executable.
>>
>> --without-libsegfault: Same as default but no libSegFault. Still a
>> dynamically linked executable, even if you have static boost and zlib.
>> It's complicated to do detect then be automatic.
>>
>> --static: No libSegFault. Print warning messages if you're missing
>> static libraries, but keep building anyway. Static executable.
>>
>> Kenneth
>>
>> On 08/06/2012 12:00 PM, Kenneth Heafield wrote:
>>> D'oh, it's a feature, not a bug. Add runtime-link=static and
>> you'll get
>>> a fully-static executable.
>>>
>>> Plan to add this to the Moses build system. Testing now.
>>>
>>> Kenneth
>>>
>>> On 08/06/2012 11:31 AM, Tom Hoar wrote:
>>>> FYI, this build is on U-10.04.4 using the bjam shipped with
>> Moses. I can
>>>> send you the complete log if it helps. Let me know if you need
>> anything
>>>> else.
>>>>
>>>> Tom
>>>>
>>>>
>>>> On Mon, 06 Aug 2012 11:27:27 -0400, Kenneth Heafield
>>>> <moses@kheafield.com <mailto:moses@kheafield.com>> wrote:
>>>>> Hi,
>>>>>
>>>>> There appears to be a bug in boost-build that prevents it from
>> making
>>>>> fully-static binaries with g++. This might take a while to sort out.
>>>>>
>>>>> Kenneth
>>>>>
>>>>> On 08/06/2012 10:48 AM, Tom Hoar wrote:
>>>>>> Thanks, Ken. I think this is easier than adding another option
>> to Moses
>>>>>> if we can understand the output. The ldd manpage says "ldd
>> prints the
>>>>>> shared libraries required by each program or shared library
>> specified on
>>>>>> the command line." I assume "shared libraries" means not statically
>>>>>> linked?
>>>>>>
>>>>>> Here's the output on the moses I just built with today's github
>> updates.
>>>>>> The log output reported exit code 0 for all lines with "g++
>> -static".
>>>>>>
>>>>>> tahoar@library1:~$ ldd `which moses`
>>>>>> linux-vdso.so.1 => (0x00007fffa2954000)
>>>>>> librt.so.1 => /lib/librt.so.1 (0x00007f818aa7b000)
>>>>>> libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f818a767000)
>>>>>> libm.so.6 => /lib/libm.so.6 (0x00007f818a4e3000)
>>>>>> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f818a2cc000)
>>>>>> libpthread.so.0 => /lib/libpthread.so.0 (0x00007f818a0af000)
>>>>>> libc.so.6 => /lib/libc.so.6 (0x00007f8189d2b000)
>>>>>> /lib64/ld-linux-x86-64.so.2 (0x00007f818aca7000)
>>>>>>
>>>>>> Is this what I should expect if all of the libboost libraries are
>>>>>> statically linked? I think so because there are no references
>> to the
>>>>>> lboost_* or lz libraries in the log file.
>>>>>>
>>>>>> Tom
>>>>>>
>>>>>>
>>>>>> On Mon, 06 Aug 2012 10:14:35 -0400, Kenneth Heafield
>>>>>> <moses@kheafield.com <mailto:moses@kheafield.com>> wrote:
>>>>>>> On linux,
>>>>>>>
>>>>>>> ldd bin/moses
>>>>>>>
>>>>>>> On 08/06/2012 10:03 AM, Tom Hoar wrote:
>>>>>>>> Thanks Ken. I downloaded/compiled with the latest changes up to
>>>>>>>> BOOST_CHECK_CLOSE. The exit code 1 disappeared and everything
>> seems
>>>>>>>> okay.
>>>>>>>>
>>>>>>>> One more question. Other than the log output at compile time,
>> is there
>>>>>>>> any way to query the Moses binary to see which libraries are
>>>>>>>> statically
>>>>>>>> vs dynamically linked?
>>>>>>>>
>>>>>>>> For example, a while back, someone on the list gave this
>> command to
>>>>>>>> test
>>>>>>>> if the binary was compiled --with-srilm, and there are others for
>>>>>>>> IRSTLM, RANDLM and KENLM.
>>>>>>>>
>>>>>>>> nm -C "/path/to/moses" | grep "vtable for
>> Moses::LanguageModelSRI"`
>>>>>>>>
>>>>>>>> I tried the obvious two grep searches "static" and "dynamic"
>> (below),
>>>>>>>> but they don't seem to relate to the libraries. Does anyone
>> know a way
>>>>>>>> to find/test if libraries are dynamically or statically linked?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Tom
>>>>>>>>
>>>>>>>>
>>>>>>>> tahoar@library1:~/domy-2.5$ nm -C `which moses` | grep "static"
>>>>>>>> 00000000005ddbc0 t
>> __static_initialization_and_destruction_0(int, int)
>>>>>>>> 00000000005de7c1 t
>> __static_initialization_and_destruction_0(int, int)
>>>>>>>> 00000000005ded37 t
>> __static_initialization_and_destruction_0(int, int)
>>>>>>>> 00000000005e0067 t
>> __static_initialization_and_destruction_0(int, int)
>>>>>>>> 00000000005e2fc6 t
>> __static_initialization_and_destruction_0(int, int)
>>>>>>>> 00000000005ecae3 t
>> __static_initialization_and_destruction_0(int, int)
>>>>>>>> 00000000005ef140 t
>> __static_initialization_and_destruction_0(int, int)
>>>>>>>> 00000000005f475f t
>> __static_initialization_and_destruction_0(int, int)
>>>>>>>> 00000000005f4cd9 t
>> __static_initialization_and_destruction_0(int, int)
>>>>>>>> 00000000005f5e53 t
>> __static_initialization_and_destruction_0(int, int)
>>>>>>>> 00000000005f6f8e t
>> __static_initialization_and_destruction_0(int, int)
>>>>>>>> 00000000005f7183 t
>> __static_initialization_and_destruction_0(int, int)
>>>>>>>> 0000000000883cc0 d static_bl_desc
>>>>>>>> 0000000000883ca0 d static_d_desc
>>>>>>>> 0000000000610680 r static_dtree
>>>>>>>> 0000000000883c80 d static_l_desc
>>>>>>>> 0000000000610200 r static_ltree
>>>>>>>>
>>>>>>>> tahoar@library1:~/domy-2.5$ nm -C `which moses` | grep "dynamic"
>>>>>>>> 0000000000570c50 W
>>>>>>>>
>> boost::unordered_detail::hash_table_unique_keys<std::pair<float const,
>>>>>>>> boost::dynamic_bitset<unsigned long, std::allocator<unsigned
>> long>
>>>>>>>>>> ,
>>>>>>>> float, boost::hash<float>, std::equal_to<float>,
>>>>>>>> std::allocator<std::pair<float const,
>> boost::dynamic_bitset<unsigned
>>>>>>>> long, std::allocator<unsigned long> > > >
>>> ::operator[](float const&)
>>>>>>>> 000000000057baf0 W
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> boost::unordered_detail::hash_table_data_unique_keys<std::allocator<std::pair<std::pair<unsigned
>>>>>>>>
>>>>>>>>
>>>>>>>> char, unsigned char> const, boost::dynamic_bitset<unsigned long,
>>>>>>>> std::allocator<unsigned long> > > >
>>> ::~hash_table_data_unique_keys()
>>>>>>>> 0000000000570af0 W
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> boost::unordered_detail::hash_table_data_unique_keys<std::allocator<std::pair<float
>>>>>>>>
>>>>>>>>
>>>>>>>> const, boost::dynamic_bitset<unsigned long,
>> std::allocator<unsigned
>>>>>>>> long> > > > >::create_buckets()
>>>>>>>> 0000000000570c00 W
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> boost::unordered_detail::hash_table_data_unique_keys<std::allocator<std::pair<float
>>>>>>>>
>>>>>>>>
>>>>>>>> const, boost::dynamic_bitset<unsigned long,
>> std::allocator<unsigned
>>>>>>>> long> > > > >::node_constructor::~node_constructor()
>>>>>>>> 0000000000570b70 W
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> boost::unordered_detail::hash_table_data_unique_keys<std::allocator<std::pair<float
>>>>>>>>
>>>>>>>>
>>>>>>>> const, boost::dynamic_bitset<unsigned long,
>> std::allocator<unsigned
>>>>>>>> long> > > > >::~hash_table_data_unique_keys()
>>>>>>>> 000000000057ba60 W
>>>>>>>>
>>>>>>>>
>>>>>>>>
>> boost::unordered_detail::hash_table_data_unique_keys<std::allocator<std::pair<unsigned
>>>>>>>>
>>>>>>>>
>>>>>>>> int const, boost::dynamic_bitset<unsigned long,
>>>>>>>> std::allocator<unsigned
>>>>>>>> long> > > > >::~hash_table_data_unique_keys()
>>>>>>>> U __dynamic_cast@@CXXABI_1.3
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, 06 Aug 2012 08:20:00 -0400, Kenneth Heafield
>>>>>>>> <moses@kheafield.com <mailto:moses@kheafield.com>> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> You're correct. There doesn't seem to be a static version of
>> this
>>>>>>>>> library. I've added the --nosegfault option (which isn't as
>> cool as
>>>>>>>>> it sounds) to skip this library.
>>>>>>>>>
>>>>>>>>> Kenneth
>>>>>>>>>
>>>>>>>>> On 08/06/2012 02:08 AM, Tom Hoar wrote:
>>>>>>>>>> I read the comment "In order to obtain a fully static Moses,
>>>>>>>>>> every g++
>>>>>>>>>> command line that includes "-static" must pass with exit
>> code 0."
>>>>>>>>>> with
>>>>>>>>>> interest.
>>>>>>>>>>
>>>>>>>>>> When we compile moses, this log output line shows an exit
>> code 1.
>>>>>>>>>> All
>>>>>>>>>> the others are 0.
>>>>>>>>>>
>>>>>>>>>> bash -c "g++ -static -lSegFault -x c++ -<<<'int main() {}' -o
>>>>>>>>>> /dev/null
>>>>>>>>>>> /dev/null 2>/dev/null"
>>>>>>>>>> 1
>>>>>>>>>>
>>>>>>>>>> Any suggestions as to what's missing/how to correct so we
>> have a
>>>>>>>>>> fully-static Moses?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Tom
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, 03 Aug 2012 18:13:47 -0400, Kenneth Heafield
>>>>>>>>>> <moses@kheafield.com <mailto:moses@kheafield.com>> wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Moses attempts to link statically by default but falls back to
>>>>>>>>>>> dynamic
>>>>>>>>>>> links. You must have static versions of all the dependencies
>>>>>>>>>>> installed
>>>>>>>>>>> as well. Run
>>>>>>>>>>>
>>>>>>>>>>> bjam --debug-configuration
>>>>>>>>>>>
>>>>>>>>>>> and, near the top, it will show you some command lines
>> followed by
>>>>>>>>>>> their
>>>>>>>>>>> exit code. In order to obtain a fully static Moses, every g++
>>>>>>>>>>> command
>>>>>>>>>>> line that includes "-static" must pass with exit code 0.
>>>>>>>>>>>
>>>>>>>>>>> Kenneth
>>>>>>>>
>>>>>>
>>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>>
>> --
>> When a place gets crowded enough to require ID's, social collapse is not
>> far away. It is time to go elsewhere. The best thing about space travel
>> is that it made it possible to go elsewhere.
>> -- R.A. Heinlein, "Time Enough For Love"
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support




------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 82, Issue 48
*********************************************

Tweet