Identifying running processes that uses deleted shared librairies

Updates:

  • 2015-02-11: The format of /proc/%pid/maps seems different between Ubuntu 12.04 and Ubuntu 14.04 (the « (deleted) » string occurs before or after the library name). So I updated the script to handle both cases. Also addded a one-liner version of the script.
  • 2015-03-03: Changed ps xh to ps axh to search in all processes.

Sometimes, when you log into your Ubuntu server, you get a message warning you that the system requires a reboot :

*** System restart required ***

This happens after a kernel upgrade, or after upgrading a shared library like openssl (libssl, libcrypto, etc.).

In the case of a kernel upgrade, it’s pretty obvious that a restart is required in order to boot on the new kernel.

But in the case of a library upgrade I don’t think such reboots are absolutely required.

What you need, is to stop and re-run the processes/services that use this library. They have been loaded in memory when the processs got started, and your new library is in fact not used by the running processes until they die.

So, instead of restarting the system, it may be more suitable to only restart the set of processes that use this library, isn’t it?

You could go with restarting all your /etc/init.d/* services, but here is a script I made to identify the running processes that are now using a dead/deleted shared library :

ps axh -o pid \
| while read PROCID; do
        grep '(deleted)' /proc/$PROCID/maps 2> /dev/null | grep '\.so';
        if [ $? -eq 0 ]; then
                CMDLINE=$(sed -e 's/\x00/ /g' < /proc/$PROCID/cmdline)
                echo -e "\tPID $PROCID $CMDLINE\n"
        fi
done

One-liner version for easy execution on remote machines by copy/paste (it only show processes that need to be reloaded) :

( ps xh -o pid | while read PROCID; do grep '(deleted)' /proc/$PROCID/maps 2> /dev/null | grep '\.so'; if [ $? -eq 0 ]; then CMDLINE=$(sed -e 's/\x00/ /g' < /proc/$PROCID/cmdline); echo -e "\tPID $PROCID $CMDLINE\n"; fi; done ) | grep PID

The script inspect the maps' file, in the/proc’ process space, for files marked as (deleted)'. Themaps’ file contains the list of all the memory mapped files of a process.

So, if you have processes using deleted shared libraries, you shoud get an output looking like this :

7f6d7a5ce000-7f6d7a736000 r-xp 00000000 08:02 9437350 /lib/libcrypto.so.0.9.8 (deleted)
7f6d7a736000-7f6d7a935000 ---p 00168000 08:02 9437350 /lib/libcrypto.so.0.9.8 (deleted)
7f6d7a935000-7f6d7a942000 r--p 00167000 08:02 9437350 /lib/libcrypto.so.0.9.8 (deleted)
7f6d7a942000-7f6d7a95a000 rw-p 00174000 08:02 9437350 /lib/libcrypto.so.0.9.8 (deleted)
    PID 725 /usr/sbin/ntpd

7fa48360d000-7fa483775000 r-xp 00000000 08:02 9437350 /lib/libcrypto.so.0.9.8 (deleted)
7fa483775000-7fa483974000 ---p 00168000 08:02 9437350 /lib/libcrypto.so.0.9.8 (deleted)
7fa483974000-7fa483981000 r--p 00167000 08:02 9437350 /lib/libcrypto.so.0.9.8 (deleted)
7fa483981000-7fa483999000 rw-p 00174000 08:02 9437350 /lib/libcrypto.so.0.9.8 (deleted)
    PID 735 /usr/sbin/sshd

From this output, you know that ntpd' andsshd’ are using the `/lib/libcrypto.so.0.9.8′ which as been removed, or changed (inode change), on disk.

Now you can schedule the restart of these services.

Some more informations.

The « System restart required » message is triggered by the presence of a /var/run/reboot-required' which is set by the upgraded library. You can find who set this file by inspecting the/var/run/reboot-required.pkgs’ file :

# cat /var/run/reboot-required
*** System restart required ***
# cat /var/run/reboot-required.pkgs
libssl0.9.8

So, it’s the upgrade of libssl which triggered the message.

After restarting your processes, you can then remove the `/var/run/reboot-required’ file, and get back your usual « clean » motd.

Ce contenu a été publié dans system. Vous pouvez le mettre en favoris avec ce permalien.

5 réponses à Identifying running processes that uses deleted shared librairies

  1. ILIV dit :

    Great post. It summarizes my Google querying session just neatly 🙂 I think this is invaluable in deciding whether you actually need to reboot your Ubuntu Server or not. And your little script also comes in handy on standard Debian system which doesn’t seem to report this type of information the way Ubuntu Server does it.

  2. ILIV dit :

    Now that I have played a little with both checkrestart and your script I can see that I’d favor your approach any time. checkrestart seems to be capturing ALL deleted files that a process still has a reference to in a memory. For example, I’ve seen PostgreSQL process being reported by checkrestart because it was referencing a deleted pg_xlog file. Not a reason to restart a service.

    What I like about grep’ing maps is that I have full control over what I need to define as a reason for a service restart.

  3. Ping : How can I install just security updates from the command line? - Popular Ubuntu Questions

  4. Ping : Ubuntu:How can I install just security updates from the command line? – Ubuntu Linux Questions

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *

*