class: center, middle # Experiences with RPATH support in EasyBuild .small[Pablo Escobar Lopez [sciCORE](https://scicore.unibas.ch/) / [UniBasel](https://www.unibas.ch/en) / [SIB](https://www.sib.swiss/) [3rd EasyBuild User Meeting](https://github.com/easybuilders/easybuild/wiki/3rd-EasyBuild-User-Meeting) January 2018 Amsterdam [https://pescobar.github.io/easybuild-user-meeting-slides/](https://pescobar.github.io/easybuild-user-meeting-slides/)] --- # Agenda 1. Who I am 2. My cluster 3. My EasyBuild Setup 4. RPATH 5. Questions / Discussion --- # Who I am * Pablo Escobar Lopez * Linux/HPC systems engineer * EasyBuild user since ... 2014? * Easybuild contributor * Mainly bioinfo easyconfigs * Some easyblocks, some small tweaks in the framework * Bug hunter --- # My Cluster * [sciCORE - Scientific computing facility at the University of Basel](https://scicore.unibas.ch/) * Near ~8K x86_64 CPUs (multiple cpu generations) * Infiniband networking (multiple infiniband islands) * 5 PB storage * GPFS * BeeGFS * Some GPUs * CentOS 7 --- # My Cluster * Around ~1K user accounts * Between 50 to 100 different active users every day * Many different workloads. bioinfo, physics, chemistry, economics... --- # My Cluster ### Near 1K modules created with easybuild (lua format) ```terminal [user@scicore ~]$ find /scicore/soft/modules/all/ -type f -iname *.lua | wc -l 963 ``` ### Lmod for modules management ```terminal [user@scicore ~]$ ml -v Modules based on Lua: Version 7.4.5 2017-04-04 08:58 -07:00 by Robert McLay mclay@tacc.utexas.edu ``` We use Lmod's cache which is very convenient for speed when having so many modules. We don't remove old modules. We only hide them (available since [Lmod-7.0](https://github.com/TACC/Lmod#lmod-70)) --- # EasyBuild setup * Two dedicated accounts for easybuild * easybuild user: Test and Development * soft user: Deployments to production * `eb --optarch=GENERIC` * Main toolchains `goolf/1.7.20` and `intel/2017.01` * [RPATH](http://easybuild.readthedocs.io/en/latest/RPATH-support.html) enabled by default ... since months ago --- # EasyBuild setup We use gitlab to keep track of installed easyconfigs and notify the admins about new software installations .center[] --- # What is RPATH From [Wikipedia](https://en.wikipedia.org/wiki/Rpath) ```text In computing, rpath designates the run-time search path hard-coded in an executable file or library. Dynamic linking loaders use the rpath to find required libraries. Specifically it encodes a path to shared libraries into the header of an executable (or another shared library). This RPATH header value (so named in the Executable and Linkable Format header standards) may either override or supplement the system default dynamic linking search paths. ``` RPATH takes precedence over $LD_LIBRARY_PATH (which is the most commonly used in environment modules) ```text LD_LIBRARY_PATH is a colon-separated set of directories where libraries should be searched for first, before the standard set of directories ``` --- # How is RPATH support enabled in EasyBuild From the [official EasyBuild docs](http://easybuild.readthedocs.io/en/latest/RPATH-support.html#implementation) .small[When EasyBuild is configured to use RPATH, wrapper scripts are put in place for the dynamic linker commands (ld, ld.gold), as well as for every compiler command that is part of the toolchain being used. This is done during the prepare step. The wrapper scripts will analyze and rewrite the list of arguments supplied to the command they are wrapping as needed, i.e.: * inject an -rpath argument for every -L argument that specifies a library directory (with some exceptions, see also Filtering RPATH entries via --rpath-filter) * lter out arguments that affect RPATH (e.g., --enable-new-dtags) * sure that the library subdirectories (lib, /lib64) of the installation directory also have an RPATH entry * include additional arguments related to RPATH (e.g. --disable-new-dtags) ] --- # Why RPATH We have many unexperienced users who don't know what `LD_LIBRARY_PATH` is (and don't care at all) Many of the bioinfo workflows executed in the cluster require to load many modules together. Unexperienced users start mixing modules (even if it's no allowed by default) and end up with unpredictable environments ... only God knows what libraries they are using. By using RPATH we assure that if app "X" has been linked with library "Y" this library will be used on runtime, no matter what mess the user has done with his/her environment (`LD_LIBRARY_PATH`) --- # RPATH advantages I don't need `$LD_LIBRARY_PATH` to find the proper libraries: ```terminal [user@scicore ~] module list No modules loaded [user@scicore ~] echo $LD_LIBRARY_PATH [user@scicore ~] [user@scicore ~] ldd /scicore/soft/apps/SAMtools/1.3-goolf-1.7.20/bin/samtools linux-vdso.so.1 => (0x00007f23f5717000) libncursesw.so.6 => /scicore/soft/apps/ncurses/5.9-goolf-1.7.20/lib/libncursesw.so.6 (0x00007f23f54ae000) libm.so.6 => /usr/lib64/libm.so.6 (0x00007f23f519c000) libz.so.1 => /scicore/soft/apps/zlib/1.2.8-goolf-1.7.20/lib/libz.so.1 (0x00007f23f4f86000) libdl.so.2 => /usr/lib64/libdl.so.2 (0x00007f23f4d82000) libpthread.so.0 => /usr/lib64/libpthread.so.0 (0x00007f23f4b66000) libc.so.6 => /usr/lib64/libc.so.6 (0x00007f23f47a4000) /lib64/ld-linux-x86-64.so.2 (0x000056012848f000) ``` And even if I define a wrong `$LD_LIBRARY_PATH` the right libraries used during compilation would be used on runtime --- # RPATH advantages Binaries can be executed even if I don't load any module or define `$LD_LIBRARY_PATH` ```terminal [user@scicore ~] module list No modules loaded [user@scicore ~] echo $LD_LIBRARY_PATH [user@scicore ~] [user@scicore ~] /scicore/soft/apps/SAMtools/1.3-goolf-1.7.20/bin/samtools --help | head -3 Program: samtools (Tools for alignments in the SAM format) Version: 1.3 (using htslib 1.3) ``` Of course for this to work I need to know the full path to the binary. If I don't load the module `$PATH` is not defined --- # RPATH exceptions? There are few applications which I have to build without RPATH enabled in easybuild. For those I use `toolchainopts = {'rpath': False}` Right now we only have 7 exceptions in the cluster: ```terminal [user@scicore installed_easyconfigs] grep -R "{'rpath': False}" * | wc -l 7 ``` Most of those apps provide 'exotic' build systems where injecting the RPATH flags require some test and debug. They can probably be fixed to be built with RPATH but IMHO it's not worth the effort. --- # RPATH #### So you don't need $LD_LIBRARY_PATH at all ? .small[Well....yes and no.... You don't need `$LD_LIBRARY_PATH` for applications __built with easybuild and RPATH support enabled__ but many users still want to compile their own code on top of the cluster software stack and they don't use easybuild (neither use RPATH "manually") That's why all our modules still provide `$LD_LIBRARY_PATH` (runtime library search) and `$LIBRARY_PATH` (build time library search)] ```terminal [user@scicore ~] ml --redirect show SAMtools/1.3.1-goolf-1.7.20 | grep LIBRARY_PATH prepend_path("LD_LIBRARY_PATH","/scicore/soft/apps/SAMtools/1.3.1-goolf-1.7.20/lib") prepend_path("LIBRARY_PATH","/scicore/soft/apps/SAMtools/1.3.1-goolf-1.7.20/lib") ``` --- # RPATH ## And what problems I found? Sometimes (not often) enabling the RPATH support in easybuild can introduce "obscure" errors We do all the test installations and easyconfigs development with user "easybuild" __without RPATH enabled__ We only enable RPATH when deploying to production with "soft" user. This way if the problem was introduced when enabling the RPATH support we can easily identify it. --- # RPATH ## And how many new tickets I got since I enabled RPATH ? --- # RPATH ## And how many new tickets I got since I enabled RPATH ? ### .center[__ZERO__] --- # RPATH ## And how many new tickets I got since I enabled RPATH ? ### .center[__ZERO__] .center[] .center[seriously?] --- # RPATH ## And how many new tickets I got since I enabled RPATH ? We got no tickets or user complains since we enabled the RPATH support in EasyBuild 99% of users haven't noticed the change. The 1% who noticed it is because I told them --- # RPATH ## Possible issues? Maybe some applications haven't been properly linked with RPATH but we haven't notice it yet because we still define `LD_LIBRARY_PATH` in the modules --- # RPATH ## Possible issues? Maybe some applications haven't been properly linked with RPATH but we haven't notice it yet because we still define `LD_LIBRARY_PATH` in the modules ## Have you reviewed all your ~1000 installations to make sure that RPATH is used in every binary? --- # RPATH ## Possible issues? Maybe some applications haven't been properly linked with RPATH but we haven't notice it yet because we still define `LD_LIBRARY_PATH` in the modules ## Have you reviewed all your ~1000 installations to make sure that RPATH is used in every binary? .center[] .center[NO] --- class: center, middle # Why RPATH support is still a experimental feature in EasyBuild? .center[Ask [Kenneth](https://github.com/boegel) ;)] --- class: center, middle #That's all! Thank you! .center[] # .center[QUESTIONS?]