When a database in running on a server it'll be using some shared memory segments. When the DB is shutdown those shared memory segments are removed. If for some reason those shared memory segments are not cleared then the database will not be able to start up again.
I ran into this situation a couple of days back.
To view the current status of the shared memory segments, login to the server and switch to the database user and run the ipcs -m command:
[orcltestdb@dbtetstnode1 ~]$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x06347849 950274 root 666 65544 2
0x0c6629c9 1015811 root 666 1203136 3
0x31ad0002 1048580 root 666 131176 3
0x01ad02c9 1081349 root 664 4192 1
0x01ad02c6 1114118 root 664 4192 1
0x01ad0349 1867783 root 664 4192 1
0x01ad0263 1179656 root 664 4192 1
0x01ad0191 1212425 root 664 4192 1
0x01ad027f 1245194 root 664 4192 1
0x01ad029f 1310731 root 664 4192 1
0x01ad02b1 1507340 root 664 4192 1
0x01ad02b9 1671181 root 664 4192 1
0x01ad0356 1769486 root 664 4192 1
0x01ad02be 1900559 root 664 4192 1
0x00000000 1660387344 grid 640 4096 0
0x00000000 1660420113 grid 640 4096 0
0xc947b448 1660452882 grid 640 4096 0
0x00000000 1763115027 orcltestdb 640 2550136832 1 dest
0x00000000 1763147796 orcltestdb 640 274877906944 1 dest
0x00000000 1763180565 orcltestdb 640 205755777024 1 dest
0x00000000 1763213334 orcltestdb 640 2097152 1 dest
The segments highlighted in bold were preventing the database from starting up and had to be removed.
Adding a couple of more switches to the ipcs command gave more verbose output:
ipcs -pmc
------ Shared Memory Segment Creators/Owners --------
shmid perms cuid cgid uid gid
950274 666 root root root root
1015811 666 root root root root
1048580 666 root root root root
1081349 664 root root root root
1114118 664 root root root root
1867783 664 root root root root
1179656 664 root root root root
1212425 664 root root root root
1245194 664 root root root root
1310731 664 root root root root
1507340 664 root root root root
1671181 664 root root root root
1769486 664 root root root root
1900559 664 root root root root
1660387344 640 grid dba grid dba
1660420113 640 grid dba grid dba
1660452882 640 grid dba grid dba
1763115027 640 orcltestdb dba orcltestdb dba
1763147796 640 orcltestdb dba orcltestdb dba
1763180565 640 orcltestdb dba orcltestdb dba
1763213334 640 orcltestdb dba orcltestdb dba
The common way to remove those shared memory segments is to use the ipcrm command followed by shm and the shmid. So that is what I did:
[orcltestdb@dbtetstnode1 ~]$ ipcrm shm 1763180565
resource(s) deleted
[orcltestdb@dbtetstnode1 ~]$ ipcrm shm 1763147796
resource(s) deleted
[orcltestdb@dbtetstnode1 ~]$ ipcrm shm 1763115027
resource(s) deleted
But when I ran the ipcs command again I was surprised to see that the segments were still there.
[orcltestdb@dbtetstnode1 ~]$ ipcs -m
------ Shared Memory Segments --------
key shmid owner perms bytes nattch status
0x06347849 950274 root 666 65544 2
0x0c6629c9 1015811 root 666 1203136 3
0x31ad0002 1048580 root 666 131176 3
0x01ad02c9 1081349 root 664 4192 1
0x01ad02c6 1114118 root 664 4192 1
0x01ad0349 1867783 root 664 4192 1
0x01ad0263 1179656 root 664 4192 1
0x01ad0191 1212425 root 664 4192 1
0x01ad027f 1245194 root 664 4192 1
0x01ad029f 1310731 root 664 4192 1
0x01ad02b1 1507340 root 664 4192 1
0x01ad02b9 1671181 root 664 4192 1
0x01ad0356 1769486 root 664 4192 1
0x01ad02be 1900559 root 664 4192 1
0x00000000 1660387344 grid 640 4096 0
0x00000000 1660420113 grid 640 4096 0
0xc947b448 1660452882 grid 640 4096 0
0x00000000 1763115027 orcltestdb 640 2550136832 1 dest
0x00000000 1763147796 orcltestdb 640 274877906944 1 dest
0x00000000 1763180565 orcltestdb 640 205755777024 1 dest
0x00000000 1763213334 orcltestdb 640 2097152 1 dest
I ran sysrev to get a status of the DB and it showed that the DB was not running.
[orcltestdb@dbtetstnode1 ~]$ sysresv
IPC Resources for ORACLE_SID "orcltestdb1" :
Shared Memory
ID KEY
No shared memory segments used
Semaphores:
ID KEY
No semaphore resources used
Oracle Instance not alive for sid "orcltestdb1"
Then I tried to search for the shmids with the lsof command and that was it. I found that the particular segment was being used by a DB query process still in running state:
[orcltestdb@dbtetstnode1 ~]$ lsof | grep 1763147796
oracle 26669 orcltestdb DEL REG 0,12 1763147796 /SYSV00000000
[orcltestdb@dbtetstnode1 ~]$ ps -ef | grep 26669
817836 4781 87652 0 11:58 pts/7 00:00:00 grep 26669
817836 26669 26568 0 11:28 ? 00:00:00 orcltestdb1n121 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
I killed the process and that finally removed the corresponding memory segment.
No comments:
Post a Comment