Doug's Oracle Blog

Work is a little busy at the moment, but I had time to look at this a bit more today ...

Many thanks for the useful follow-up comments on the last posting. They were very interesting and if I'd picked them up in some of my Google searches, we might have solved the problem more quickly, although I would re-emphasise a couple of general points :-

1) This particular problem is nothing to do with RAC, in as much as the server concerned is not part of a cluster, so this is just as likely to happen on a single instance/node.

2) What's irritating about this is that it's because the Listener is configured to use ONS by default, even if we're unlikely to be interested in using it in a single-instance environment. (Actually, is that true? I'll come back to that.)

What I neglected to mention last time is that there is another work-around mentioned in the various Metalink notes. We can rename the ons.config file so that the Listener won't see it and try to start ONS. This is from my server at home.

Sun Microsystems Inc.   SunOS 5.10      Generic January 2005
$ cd $ORACLE_HOME/opmn/conf
$ ls -ltra
total 6
drwxr-x---   7 oracle   oinstall     512 Oct  7  2006 ..
-rw-------   1 oracle   oinstall      44 Oct  7  2006 ons.config
drwx------   2 oracle   oinstall     512 Oct  7  2006 .
$ cat ons.config
localport=6100
remoteport=6200
loglevel=3

So I could just rename ons.config and restart the Listener and that should fix it.

However, I much prefer the original solution I mentioned. Rather than renaming configuration files in the ORACLE_HOME with no obvious reason why (which seems a bit of a hack to me), I prefer having the parameter in listener.ora which explicitly disables the subscription, with a comment explaining why it's there. e.g.

# Workaround for bug 4518443 - should be fixed in 10.2.0.3
SUBSCRIBE_FOR_NODE_DOWN_EVENT_LISTENER=OFF

Better still, just patch to the right level, but we found the listener change less risky and faster to implement. It went in last night although I'm glad to say it wasn't me that made the change - thanks Richard

Oh, yes, so why do Oracle configure this even in a non-RAC environment? Well, for all I know it's just a mistake, but it occurred to me that just because it's single-instance and it might not be so useful, maybe an application would want to use FAN? Maybe not.

Today I solved a nasty intermittent problem we'd been having on one of our Production servers so I thought I'd blog about it here in case it saves someone else some pain. The symptoms were

1) No-one was able to connect to the database.

2) There were two listener processes running, which was apparent from a ps-ef|grep tns

oracle 2647896 1495262  1 May 18 -  0:00 /oem/oracle/product/10.2/bin/tnslsnr LISTENER 
						-inherit 
oracle 1495262       1  1 May 13 -  2:32 /oem/oracle/product/10.2/bin/tnslsnr LISTENER 
						-inherit 

(Note the different start times and the later process was definitely not started manually.)

This had happened once or twice and we found that when we killed the child listener (note that one listener's parent process ID is the process id of the other) everything worked correctly.

This time I had a good look through the listener.log and noticed these warning messages related to the ONS service, even though we're not using RAC.

WARNING: Subscription for node down event still pending

It took a while to track it down, but it turns out that this is a known, but unpublished bug number 4518443 in 10.2.0.2 (and no doubt previous versions), fixed in 10.2.0.3. For those of you with Metalink accounts, the relevant note is here. You can either apply patch 4518443 or, as a temporary workaround, disable that service by adding the following to your listener.ora configuration and restarting the listener.

SUBSCRIBE_FOR_NODE_DOWN_EVENT_<listener_name>=OFF