Recently I encountered an issue wherein the CA Certificates Service was in maintenance state.
sudo svcs -xv
svc:/system/ca-certificates:default (CA Certificates Service)
State: maintenance since September 7, 2017 01:24:17 AM UTC
Reason: Start method failed repeatedly, last exited with status 1.
See: http://support.oracle.com/msg/SMF-8000-KS
See: man -M /usr/share/man -s 5 openssl
See: /var/svc/log/system-ca-certificates:default.log
Impact: This service is not running.
I tried to clear the maintenance state manually a couple of times but with no success.
The svcs log file /var/svc/log/system-ca-certificates:default.log for the service showed the following error:
[ Sep 7 09:27:46 Executing start method ("/lib/svc/method/svc-ca-certificates start"). ]
Re-generating OpenSSL hash Links
Traceback (most recent call last):
File "/lib/svc/method/svc-ca-certificates", line 108, in <module>
smf_include.smf_main()
File "/usr/lib/python2.7/vendor-packages/smf_include.py", line 115, in smf_main
File "/lib/svc/method/svc-ca-certificates", line 91, in start
generate_links()
File "/lib/svc/method/svc-ca-certificates", line 43, in generate_links
os.symlink(os.path.join(RELCDIR, cfile), os.path.join(LINKDIR, shash))
OSError: [Errno 17] File exists
[ Sep 7 09:27:46 Method "start" exited with status 1. ]
The error log explained two things. The first being that the service startup method was apparently written in python and it was unable to start because some file that was supposed to be created by the startup method was already in existence.
After some investigation I found out that the /etc/openssl/certs directory should be empty to allow the service to startup correctly. So I emptied the directory while keeping a backup handy with me.
[ssuri@:~] $ ls -l /etc/openssl/certs
total 118
lrwxrwxrwx 1 root root 44 Sep 7 09:52 024dc131.0 -> ../../certs/CA/Microsec_e-Szigno_Root_CA.pem
lrwxrwxrwx 1 root root 51 Sep 7 09:52 03179a64.0 -> ../../certs/CA/Staat_der_Nederlanden_EV_Root_CA.pem
lrwxrwxrwx 1 root root 65 Sep 7 09:52 039c618a.0 -> ../../certs/CA/TURKTRUST_Certificate_Services_Provider_Root_2.pem
lrwxrwxrwx 1 root root 38 Sep 7 09:52 03f0efa4.0 -> ../../certs/CA/Wells_Fargo_Root_CA.pem
lrwxrwxrwx 1 root root 42 Sep 7 09:52 062cdee6.0 -> ../../certs/CA/GlobalSign_Root_CA_-_R3.pem
lrwxrwxrwx 1 root root 34 Sep 7 09:52 0750887b.0 -> ../../certs/CA/IPS_CLASE3_root.pem
lrwxrwxrwx 1 root root 35 Sep 7 09:52 080911ac.0 -> ../../certs/CA/QuoVadis_Root_CA.pem
lrwxrwxrwx 1 root root 49 Sep 7 09:52 0810ba98.0 -> ../../certs/CA/Root_CA_Generalitat_Valenciana.pem
lrwxrwxrwx 1 root root 39 Sep 7 09:52 0a8f0c78.0 -> ../../certs/CA/IPS_Chained_CAs_root.pem
lrwxrwxrwx 1 root root 32 Sep 7 09:52 0ac62cd9.0 -> ../../certs/CA/SUNWSolarisCA.pem
[ssuri@:~] $ sudo ls -ld /etc/openssl/certs
drwxr-xr-x 2 root sys 120 Sep 7 09:52 /etc/openssl/certs
[ssuri@:~] $ sudo mv /etc/openssl/certs /etc/openssl/certs.old
[ssuri@:~] $ sudo mkdir /etc/openssl/certs
[ssuri@:~] $ sudo chown root:sys /etc/openssl/certs
I tried to restart the service now but was still receiving the same error in the log files.
I did some more digging and read a few articles and they suggested to look in the /etc/certs/CA directory for any files that shouldn't actually be there.
[ssuri@usny-infrasi049-p:~] $ sudo ls -ltr /etc/certs/CA| tail
-rw-r--r-- 1 root sys 1545 Apr 3 08:20 SUNWSolarisCA.pem
-rw-r--r-- 1 root sys 1249 Apr 3 08:20 SecureSign_RootCA11.pem
-rw-r--r-- 1 root sys 1261 Apr 3 08:20 Security_Communication_RootCA2.pem
-rw-r--r-- 1 root sys 1948 Apr 3 08:20 Staat_der_Nederlanden_EV_Root_CA.pem
-rw-r--r-- 1 root sys 2069 Apr 3 08:20 Staat_der_Nederlanden_Root_CA_-_G2.pem
-rw-r--r-- 1 root sys 1952 Apr 3 08:20 Staat_der_Nederlanden_Root_CA_-_G3.pem
-rw-r--r-- 1 root sys 1399 Apr 3 08:20 Starfield_Root_Certificate_Authority_-_G2.pem
lrwxrwxrwx 1 root root 45 Jun 26 16:37 usny-infrasi041-p.pem -> /var/opt/SUNWldm/trust/usny-infrasi041-p.pem
lrwxrwxrwx 1 root root 45 Jun 26 16:37 usny-infrasi043-p.pem -> /var/opt/SUNWldm/trust/usny-infrasi043-p.pem
lrwxrwxrwx 1 root root 45 Sep 5 13:52 usny-infrasi042-p.pem -> /var/opt/SUNWldm/trust/usny-infrasi042-p.pem
The last three files .pem files were for different servers. I'm not sure as to how they got there.
I removed those soft links and emptied the /etc/openssl/certs directory again.
sudo unlink usny-infrasi042-p.pem
sudo unlink usny-infrasi043-p.pem
sudo unlink usny-infrasi041-p.pem
I followed this up with a disable/enable operation on the service and it finally came online.
[ssuri@usny-infrasi049-p:~] $ sudo svcadm disable svc:/system/ca-certificates:default^C
[ssuri@usny-infrasi049-p:~] $ svcs -xv
[ssuri@usny-infrasi049-p:~] $ svcs -xvudo svcadm disable svc:/system/ca-certificates:defaulten
[ssuri@usny-infrasi049-p:~] $ svcs -xv
[ssuri@usny-infrasi049-p:~] $ sudo svcs -l svc:/system/ca-certificates:default
fmri svc:/system/ca-certificates:default
name CA Certificates Service
enabled true
state online
next_state none
state_time September 7, 2017 10:11:16 AM UTC
logfile /var/svc/log/system-ca-certificates:default.log
restarter svc:/system/svc/restarter:default
manifest /lib/svc/manifest/system/ca-certificates.xml
dependency require_all/none svc:/system/filesystem/minimal (online)
[ssuri@usny-infrasi049-p:~] $ sudo svcs -l svc:/system/ca-certificates:default
fmri svc:/system/ca-certificates:default
name CA Certificates Service
enabled true
state online
next_state none
state_time September 7, 2017 10:11:16 AM UTC
logfile /var/svc/log/system-ca-certificates:default.log
restarter svc:/system/svc/restarter:default
manifest /lib/svc/manifest/system/ca-certificates.xml
dependency require_all/none svc:/system/filesystem/minimal (online)
Since I was unsure of how the .pem files for the other servers got into /etc/certs/CA directory, I re-created the soft links again.
[ssuri@usny-infrasi049-p:/etc/certs/CA] $ sudo ln -s /var/opt/SUNWldm/trust/usny-infrasi041-p.pem usny-infrasi041-p.pem
[ssuri@usny-infrasi049-p:/etc/certs/CA] $ sudo ln -s /var/opt/SUNWldm/trust/usny-infrasi043-p.pem usny-infrasi043-p.pem
[ssuri@usny-infrasi049-p:/etc/certs/CA] $ sudo ln -s /var/opt/SUNWldm/trust/usny-infrasi042-p.pem usny-infrasi042-p.pem
sudo svcs -xv
svc:/system/ca-certificates:default (CA Certificates Service)
State: maintenance since September 7, 2017 01:24:17 AM UTC
Reason: Start method failed repeatedly, last exited with status 1.
See: http://support.oracle.com/msg/SMF-8000-KS
See: man -M /usr/share/man -s 5 openssl
See: /var/svc/log/system-ca-certificates:default.log
Impact: This service is not running.
I tried to clear the maintenance state manually a couple of times but with no success.
The svcs log file /var/svc/log/system-ca-certificates:default.log for the service showed the following error:
[ Sep 7 09:27:46 Executing start method ("/lib/svc/method/svc-ca-certificates start"). ]
Re-generating OpenSSL hash Links
Traceback (most recent call last):
File "/lib/svc/method/svc-ca-certificates", line 108, in <module>
smf_include.smf_main()
File "/usr/lib/python2.7/vendor-packages/smf_include.py", line 115, in smf_main
File "/lib/svc/method/svc-ca-certificates", line 91, in start
generate_links()
File "/lib/svc/method/svc-ca-certificates", line 43, in generate_links
os.symlink(os.path.join(RELCDIR, cfile), os.path.join(LINKDIR, shash))
OSError: [Errno 17] File exists
[ Sep 7 09:27:46 Method "start" exited with status 1. ]
The error log explained two things. The first being that the service startup method was apparently written in python and it was unable to start because some file that was supposed to be created by the startup method was already in existence.
After some investigation I found out that the /etc/openssl/certs directory should be empty to allow the service to startup correctly. So I emptied the directory while keeping a backup handy with me.
[ssuri@:~] $ ls -l /etc/openssl/certs
total 118
lrwxrwxrwx 1 root root 44 Sep 7 09:52 024dc131.0 -> ../../certs/CA/Microsec_e-Szigno_Root_CA.pem
lrwxrwxrwx 1 root root 51 Sep 7 09:52 03179a64.0 -> ../../certs/CA/Staat_der_Nederlanden_EV_Root_CA.pem
lrwxrwxrwx 1 root root 65 Sep 7 09:52 039c618a.0 -> ../../certs/CA/TURKTRUST_Certificate_Services_Provider_Root_2.pem
lrwxrwxrwx 1 root root 38 Sep 7 09:52 03f0efa4.0 -> ../../certs/CA/Wells_Fargo_Root_CA.pem
lrwxrwxrwx 1 root root 42 Sep 7 09:52 062cdee6.0 -> ../../certs/CA/GlobalSign_Root_CA_-_R3.pem
lrwxrwxrwx 1 root root 34 Sep 7 09:52 0750887b.0 -> ../../certs/CA/IPS_CLASE3_root.pem
lrwxrwxrwx 1 root root 35 Sep 7 09:52 080911ac.0 -> ../../certs/CA/QuoVadis_Root_CA.pem
lrwxrwxrwx 1 root root 49 Sep 7 09:52 0810ba98.0 -> ../../certs/CA/Root_CA_Generalitat_Valenciana.pem
lrwxrwxrwx 1 root root 39 Sep 7 09:52 0a8f0c78.0 -> ../../certs/CA/IPS_Chained_CAs_root.pem
lrwxrwxrwx 1 root root 32 Sep 7 09:52 0ac62cd9.0 -> ../../certs/CA/SUNWSolarisCA.pem
[ssuri@:~] $ sudo ls -ld /etc/openssl/certs
drwxr-xr-x 2 root sys 120 Sep 7 09:52 /etc/openssl/certs
[ssuri@:~] $ sudo mv /etc/openssl/certs /etc/openssl/certs.old
[ssuri@:~] $ sudo mkdir /etc/openssl/certs
[ssuri@:~] $ sudo chown root:sys /etc/openssl/certs
I tried to restart the service now but was still receiving the same error in the log files.
I did some more digging and read a few articles and they suggested to look in the /etc/certs/CA directory for any files that shouldn't actually be there.
[ssuri@usny-infrasi049-p:~] $ sudo ls -ltr /etc/certs/CA| tail
-rw-r--r-- 1 root sys 1545 Apr 3 08:20 SUNWSolarisCA.pem
-rw-r--r-- 1 root sys 1249 Apr 3 08:20 SecureSign_RootCA11.pem
-rw-r--r-- 1 root sys 1261 Apr 3 08:20 Security_Communication_RootCA2.pem
-rw-r--r-- 1 root sys 1948 Apr 3 08:20 Staat_der_Nederlanden_EV_Root_CA.pem
-rw-r--r-- 1 root sys 2069 Apr 3 08:20 Staat_der_Nederlanden_Root_CA_-_G2.pem
-rw-r--r-- 1 root sys 1952 Apr 3 08:20 Staat_der_Nederlanden_Root_CA_-_G3.pem
-rw-r--r-- 1 root sys 1399 Apr 3 08:20 Starfield_Root_Certificate_Authority_-_G2.pem
lrwxrwxrwx 1 root root 45 Jun 26 16:37 usny-infrasi041-p.pem -> /var/opt/SUNWldm/trust/usny-infrasi041-p.pem
lrwxrwxrwx 1 root root 45 Jun 26 16:37 usny-infrasi043-p.pem -> /var/opt/SUNWldm/trust/usny-infrasi043-p.pem
lrwxrwxrwx 1 root root 45 Sep 5 13:52 usny-infrasi042-p.pem -> /var/opt/SUNWldm/trust/usny-infrasi042-p.pem
The last three files .pem files were for different servers. I'm not sure as to how they got there.
I removed those soft links and emptied the /etc/openssl/certs directory again.
sudo unlink usny-infrasi042-p.pem
sudo unlink usny-infrasi043-p.pem
sudo unlink usny-infrasi041-p.pem
I followed this up with a disable/enable operation on the service and it finally came online.
[ssuri@usny-infrasi049-p:~] $ sudo svcadm disable svc:/system/ca-certificates:default^C
[ssuri@usny-infrasi049-p:~] $ svcs -xv
[ssuri@usny-infrasi049-p:~] $ svcs -xvudo svcadm disable svc:/system/ca-certificates:defaulten
[ssuri@usny-infrasi049-p:~] $ svcs -xv
[ssuri@usny-infrasi049-p:~] $ sudo svcs -l svc:/system/ca-certificates:default
fmri svc:/system/ca-certificates:default
name CA Certificates Service
enabled true
state online
next_state none
state_time September 7, 2017 10:11:16 AM UTC
logfile /var/svc/log/system-ca-certificates:default.log
restarter svc:/system/svc/restarter:default
manifest /lib/svc/manifest/system/ca-certificates.xml
dependency require_all/none svc:/system/filesystem/minimal (online)
[ssuri@usny-infrasi049-p:~] $ sudo svcs -l svc:/system/ca-certificates:default
fmri svc:/system/ca-certificates:default
name CA Certificates Service
enabled true
state online
next_state none
state_time September 7, 2017 10:11:16 AM UTC
logfile /var/svc/log/system-ca-certificates:default.log
restarter svc:/system/svc/restarter:default
manifest /lib/svc/manifest/system/ca-certificates.xml
dependency require_all/none svc:/system/filesystem/minimal (online)
Since I was unsure of how the .pem files for the other servers got into /etc/certs/CA directory, I re-created the soft links again.
[ssuri@usny-infrasi049-p:/etc/certs/CA] $ sudo ln -s /var/opt/SUNWldm/trust/usny-infrasi041-p.pem usny-infrasi041-p.pem
[ssuri@usny-infrasi049-p:/etc/certs/CA] $ sudo ln -s /var/opt/SUNWldm/trust/usny-infrasi043-p.pem usny-infrasi043-p.pem
[ssuri@usny-infrasi049-p:/etc/certs/CA] $ sudo ln -s /var/opt/SUNWldm/trust/usny-infrasi042-p.pem usny-infrasi042-p.pem
No comments:
Post a Comment