A recent update with a core Linux package has been causing intermittent issues with various programs on servers, so we wanted to make a post about the issue we’re seeing as well as some possible workarounds. We’ve ensured the appropriate development teams are aware of the issue but since the problem package has already been released in the wild, this post can help you stay informed.

The package that is causing the problems is NSS (Network Security Services). NSS is a set of libraries designed to support cross-platform development of security-enabled client and server applications (gleaned from the package description). A bug report has been made for this issue. If you’d like to follow it, the url is: https://bugs.centos.org/view.php?id=10930

The issue manifests itself when using something that depends on it, such as curl, git, php, etc. The most recognizable indication of a problem is the response Illegal instruction after using a tool that depends on it. For example:

curl https://google.com 

We have identified some temporary workarounds, but their implementations vary. For the curl example above simply set this environmental variable:

export NSS_DISABLE_HW_GCM=1

then re-run the curl command or altogether: 

NSS_DISABLE_HW_GCM=1 curl https://google.com

If you'd prefer to see verbose output instead, run:

NSS_DISABLE_HW_GCM=1 curl -vvvv https://google.com

It is important to understand that this issue affects more than just curl. The NSS package is a core package, and many programs that are integral to Linux depend on it. To help you get a feel for the scope of the problem, here is a dependency graph that shows some of the other programs NSS affects:

 

Another way to check for any programs on your server that NSS affects would be by running this command:

# lsof /usr/lib64/libnss3.so

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 317 root mem REG 253,1 1304208 2626976 /usr/lib64/libnss3.so
java 16941 root mem REG 253,1 1304208 2626976 /usr/lib64/libnss3.so
sshd 19634 root mem REG 253,1 1304208 2626976 /usr/lib64/libnss3.so
sshd 31164 root mem REG 253,1 1304208 2626976 /usr/lib64/libnss3.so
pickup 31910 postfix mem REG 253,1 1304208 2626976 /usr/lib64/libnss3.so

If you encounter any intermittent problems with your system that you believe might be caused by this NSS issue, the above command is a good way to check. We’ll monitor the situation and we’re hopeful a fix will be released by the package developers soon.