May 21: 10.2.0.2 Listener bug, new child listener process forked

Today I solved a nasty intermittent problem we'd been having on one of our Production servers so I thought I'd blog about it here in case it saves someone else some pain. The symptoms were

1) No-one was able to connect to the database.

2) There were two listener processes running, which was apparent from a ps-ef|grep tns

oracle 2647896 1495262  1 May 18 -  0:00 /oem/oracle/product/10.2/bin/tnslsnr LISTENER 
						-inherit 
oracle 1495262       1  1 May 13 -  2:32 /oem/oracle/product/10.2/bin/tnslsnr LISTENER 
						-inherit

(Note the different start times and the later process was definitely not started manually.)

This had happened once or twice and we found that when we killed the child listener (note that one listener's parent process ID is the process id of the other) everything worked correctly.

This time I had a good look through the listener.log and noticed these warning messages related to the ONS service, even though we're not using RAC.

WARNING: Subscription for node down event still pending

It took a while to track it down, but it turns out that this is a known, but unpublished bug number 4518443 in 10.2.0.2 (and no doubt previous versions), fixed in 10.2.0.3. For those of you with Metalink accounts, the relevant note is here. You can either apply patch 4518443 or, as a temporary workaround, disable that service by adding the following to your listener.ora configuration and restarting the listener.

SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF
Posted by Doug Burns Comments: (16) Trackbacks: (0)
Defined tags for this entry:
Related entries by tags:
Listener bug - Part 2

Trackbacks
Trackback specific URI for this entry

No Trackbacks

Comments
Display comments as (Linear | Threaded)

#1 - Marco Gralike said:
2007-05-21 22:56 - (Reply)

Maybe the following gives a little bit more insight...

Metalink:

Bug: 3881276
Note:284602.1

#2 - Jeremy Schneider said:
2007-05-21 23:35 - (Reply)

Yeah I ran into the same thing a month or two ago... even did a writeup of it.

http://www.ardentperf.com/2007/04/05/crazy-networking-problem-on-linux/

We were also using 10.2.0.2; really caused us some problems at first until we figured out what it was!

#3 - Mike Rothouse said:
2007-05-22 01:47 - (Reply)

I encountered the same issue in December 2006. I don't believe there was a patch available at that time so I had to implement the workaround. Details can be found on my blog.

http://mrothouse.wordpress.com/2006/12/21/oracle10g-additional-listener-process-spawned/

Glad to hear there is a patch if you can't move to 10.2.0.3.

#4 - Doug Burns said:
2007-05-22 08:03 - (Reply)

Once again, I forgot to mention the O/S - AIX 5.3. However, as the Metalink note says

This problem can occur on any platform.

All new connections via TNS listener hang, no errors reported

#5 - Doug Burns said:
2007-05-22 08:33 - (Reply)

I noticed this in today's Oracle-L digest. I should point out that in our case we are not using RAC at all.

http://www.freelists.org/archives/oracle-l/05-2007/msg00724.html

#5.1 - Jeremy Schneider said:
2007-05-22 17:26 - (Reply)

No RAC... so was something else listening on port 6101? Or was the listener configured to look for ONS on a different port, on which some different app was running? Seems like I've heard that Oracle Application Server can cause this problem; were that related to the problem?

#5.1.1 - Niall Litchfield said:
2007-05-24 11:25 - (Reply)

It certainly can and does happen with iAS.

#5.1.2 - Martin Berger 2008-03-06 08:16 - (Reply)

even without RAC or iAS, if you use ASM, an ONS will be running and therefore you can face this problem.

#5.1.2.1 - Doug Burns said:
2008-03-06 13:30 - (Reply)

Good point, Martin, thanks.

Because I rarely get to work with ASM at the moment, I wouldn't have thought of this.

#6 - Gaurav Batta 2008-04-10 16:15 - (Reply)

Good Article Doug,

I ran into this problem, saw your article and helped me to resolve the issue.
Thanks for sharing the information.

#7 - Cindy Fark said:
2009-06-15 19:36 - (Reply)

Thanks much for blogging this issue and sharing the solution!

#8 - joel garry 2009-08-05 17:15 - (Reply)

In 10.2.0.4 (EE on hp-ux, at least) it still leaves copious warning messages in non-rac listener.log.

SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF (or appropriate listener name) in listener.ora fixes.

I'm not using iAS, RAC or ASM, but those messages are still annoying. It's seems the targets.xml file used by dbconsole defines which listener is used, simply using dbconsole is using enough of ONS to create these warning messages.

#9 - Pauline Wan 2010-01-14 21:50 - (Reply)

I ran into the same problem at yesterday
midnight on our production server and the way I fixed it was same as Doug's. I want to figure out why the second listener would come out. I search from google and get the snswer from your blog. Now I know it is not people manually started process, it is a bug in 10.2.0.2 and lower release (we are using 10.2.0.1).

Thanks for the Doug's good blog and all other people's helpful information related this issue.

#9.1 - Doug Burns said:
2010-01-14 22:35 - (Reply)

Glad you got it sorted out, Pauline.

Another win for blogs :-)

#10 - Will O said:
2010-02-06 01:54 - (Reply)

Thanks a Lot for this solutions ... It´s work very good ...

#11 - chirdeep 2011-12-14 12:24 - (Reply)

Hi This is chirdeep from India, blog is really very much helpful. We actually have a same scenario but our database version is of 9.2.0.8. Hence if some one could help it would be very much helpful.

Regards,
Chirdeep


Add Comment

Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.
BBCode format allowed