我们的文章会在微信公众号IT民工的龙马人生和博客网站( www.htz.pw )同步更新 ，欢迎关注收藏，也欢迎大家转载，但是请在文章开始地方标注文章出处，谢谢！
由于博客中有大量代码，通过页面浏览效果更佳。
来自一位兄弟的案例分享，某银行的某系统rac数据库版本19.6，二节点的mmon slave进程一直在报ORA-01000，导致awr、ash等等很多MMON的功能收到了影响。
2021-11-05T14:38:50.244256+08:00
Errors in file /u01/app/oracle/diag/rdbms/rcg02/rcg022/trace/rcg022_m002_65032.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01000: maximum open cursors exceeded
ORA-00604: error occurred at recursive SQL level 1
ORA-01000: maximum open cursors exceeded
2021-11-05T14:53:51.739041+08:00
Errors in file /u01/app/oracle/diag/rdbms/rcg02/rcg022/trace/rcg022_m000_64999.trc:
ORA-00604: error occurred at recursive SQL level 1
ORA-01000: maximum open cursors exceeded
ORA-00604: error occurred at recursive SQL level 1
查询v$open_cursor，发现每个m00n的session都open了很多同一个递归sql，cursor type为OPEN-RECURSIVE：
select ts# from ts$ where bitmapped <> 0 and contents$ <> 1 and online$=1 and ts# >= :1 order by ts# asc
并且发现rac的所有节点，就只有2节点的异常
SQL> select a.inst_id,program,ADDRESS,CHILD_ADDRESS,a.sql_id,count(*) from gv$open_cursor a,gv$session b2 where a.inst_id=b.inst_id and a.sid=b.sid and b.program like '%M00%'3 group by a.inst_id,program,ADDRESS,CHILD_ADDRESS,a.sql_id having count(*)>10;INST_ID PROGRAM ADDRESS CHILD_ADDRESS SQL_ID COUNT(*)
---------- ------------------------------------------------ ---------------- ---------------- --------------- ----------2 oracle@mgmrcgarac02 (M004) 000000063F2A95B0 000000063F2A8188 38hpmnhjywkz4 9952 oracle@mgmrcgarac02 (M005) 000000063F2A95B0 000000063F2A8188 38hpmnhjywkz4 9952 oracle@mgmrcgarac02 (M002) 000000063F2A95B0 000000063F2A8188 38hpmnhjywkz4 9952 oracle@mgmrcgarac02 (M000) 000000063F2A95B0 000000063F2A8188 38hpmnhjywkz4 9942 oracle@mgmrcgarac02 (M003) 000000063F2A95B0 000000063F2A8188 38hpmnhjywkz4 9982 oracle@mgmrcgarac02 (M001) 000000063F2A95B0 000000063F2A8188 38hpmnhjywkz4 995这非常不正常，通常每个session不会一直open同一个递归cursor，怀疑是这些递归sql在运行结束都没有关闭游标。为了更好的分析问题，对M000进程做了一个processstate dump
SQL> oradebug setospid 64999;
Oracle pid: 74, Unix process pid: 64999, image: oracle@mgmrcgarac02 (M000)
SQL> oradebug dump processstate 266;
Statement processed.
SQL> oradebug tracefile_name
/u01/app/oracle/diag/rdbms/rcg02/rcg022/trace/rcg022_m000_64999.trc
从dump中发现确实同时打开了多个该cursor，单对某个LIBRARY CACHE LOCK进行分析，发现session以NULL模式持有该sql父游标handle的library cache lock，说明已经打开了游标。SO: 0x61a4b63a8, type: LIBRARY OBJECT LOCK (118), map: 0x448394090state: LIVE (0x4532), flags: 0x1owner: 0x86ffdc2f8, proc: 0x86fef8078link: 0x61a4b63c8[0x61e0f7ae8, 0x61e0ef1e8]child list count: 0, link: 0x61a4b6418[0x61a4b6418, 0x61a4b6418]conid: 1, conuid: 1, SGA version=(1,0), pg: 0SOC: 0x448394090, type: LIBRARY OBJECT LOCK (118), map: 0x61a4b63a8state: LIVE (0x99fc), flags: INIT (0x1)LibraryObjectLock: Address=0x448394090 Handle=0x63f2a95b0 Mode=N CanBeBrokenCount=1 Incarnation=1 ExecutionCount=1 User=0x863b17220 Session=0x863b19a08 ReferenceCount=1 Flags=CNB/[0001] SavepointNum=c21c50 Time=09/02/2021 03:04:59 LibraryHandle: Address=0x63f2a95b0 Hash=23ee4be4 LockMode=N PinMode=0 LoadLockMode=0 Status=VALD ObjectName: Name=select ts# from ts$ where bitmapped <> 0 and contents$ <> 1 and online$=1 and ts# >= :1 order by ts# asc 游标打开时间09/02/2021 03:04:59，两个月前。。。。，并且对应的子游标handle的状态已经失效了SO: 0x61e0ef1c8, type: LIBRARY OBJECT LOCK (118), map: 0x44b1ca9b0state: LIVE (0x4532), flags: 0x1owner: 0x86ffdc2f8, proc: 0x86fef8078link: 0x61e0ef1e8[0x61a4b63c8, 0x86ffdc368]child list count: 0, link: 0x61e0ef238[0x61e0ef238, 0x61e0ef238]conid: 1, conuid: 1, SGA version=(1,0), pg: 0SOC: 0x44b1ca9b0, type: LIBRARY OBJECT LOCK (118), map: 0x61e0ef1c8state: LIVE (0x99fc), flags: INIT (0x1)LibraryObjectLock: Address=0x44b1ca9b0 Handle=0x63f2a8188 Mode=N CanBeBrokenCount=1 Incarnation=1 ExecutionCount=33565358 Context=0x7f44760c4f80 User=0x863b17220 Session=0x863b19a08 ReferenceCount=1 Flags=BRO/CBK/[0020] SavepointNum=0 Time=09/02/2021 03:04:59 LibraryHandle: Address=0x63f2a8188 Hash=0 LockMode=N PinMode=S LoadLockMode=0 Status=INVL
说明sql早已执行完成，但是游标并未关闭。
解决该问题的方法很简单，就是关闭session，由于m00n进程是mmon的slave进程，可以随便kill，kill之后问题解决。但是并没有找到该问题的原因，MOS上也没有查到相关的案例，个人怀疑是在关闭递归游标的时候出现了异常，导致反复的打开该游标。