Oct 27: The Reality Gap (1) - Software Maintenance

It's so long now since the OUG Scotland Conference and I ended up leaving early because I wasn't feeling too good so I'm not going to attempt a review of the day. I asked a few others what they thought afterwards and the general view was that the day was pretty good, if perhaps too sales-orientated. Most of the morning was taken up with Oracle business-focussed presentations and I'm not sure that ever goes down very well at a UKOUG conference. I heard a particularly good review of Jason Arneil's presentation which I was hoping to see until it was rescheduled later in the day. I particularly enjoyed the last presentation I watched Jason give and then, just the other day, I noticed that Jason has a blog too so that should be worth following.

I think my presentation went reasonably well, based on the feedback. For UKOUG members, the presentation slides are here but I have to say I was very reluctant to hand them over because they're useless in isolation - just a few visual gags.

In keeping with the style of a keynote and the chap I was standing in for, I wanted something reasonably controversial but high-level. The original idea for the presentation came from a series of blogs that had been floating around in my head for a while, discussing 'The Reality Gap' or :-

"The difference between what Consulting Firms, Oracle Marketing, Technical Architects, Bloggers and Security Researchers say we should be doing and what most of us really are doing"

One of my favourite stories has always been The Emperor's New Clothes. Nothing drives me nuts like everyone toe-ing an almost religious 'line' which has no basis in reality. If I can play the little boy and cut through the misinformation a little that makes me feel better. In fact, I think it's what every consultant, contractor or whatever we want to call ourselves should be doing - asking tricky questions to get to the truth.

Having stolen blog material for a presentation, it's only fair that it should appear here eventually. For my first example, let's talk about Software Maintenance and, more specifically, upgrading to the latest versions of Oracle and applying patches on a regular basis.

I conducted a show-of-hands survey near the start of the presentation which went something like this (and thanks again for everyone who joined in, I was a little nervous that a Scottish crowd might be a little reticent!)

  • Who is running Oracle? (Nearly everyone raised their hand. Good start.)
  • Who has all of their databases on 9.2.0.8 or 10.2.0.x? (Less than a third kept their hands raised)
  • Who has applied the July 2007 CPU to all of their databases? (Not a single hand remained in the air. In October.)
The most important thing is that I guessed the result in advance and, if it was so obvious to me before I met any of the respondents, isn't there a Reality Chasm between what Oracle Support, bloggers and security researchers write and what most companies are doing? Every time a CPU is released, it causes at least a minor flutter in the blogging community, yet it seems to pass by most corporate customers I work with (and I've worked with a few). Why? Well, because they have to actually roll the patch out across hundreds of databases, not just talk about it!

  • To start with, you need a regularly scheduled outage on every database. (Plus all the associated Change Management)
  • Next you need enough DBAs to plan, test and implement the change.
  • Plus the people to perform regression testing? (Well, maybe not and you're prepared to take the risk that no new problems have been introduced)
Actually, that's just a small sample of the issues involved. I see patch management as possibly the most important challenge that DBAs have to address and the more work that Oracle put into making this easier, the better.

You and I probably both know that security patches are essential to ensure the security of the business data, but do you think every business truly understands that? They've got better, thanks to SOX legislation, but what do you think they want their DBAs doing - implementing shiny new applications or applying patches to existing applications when they won't see any functional improvement from their perspective? That's our job, though, isn't it? To educate the business about the importance of patching? Mmmm, but who pays us? Who is always looking to cut costs to the bear minimum (ah, the wonders of Capitalism). Who really controls IT departments these days?

However, none of the pros and cons matter to my specific argument and whether you agree or disagree with me about the importance of security patches isn't the issue. (Oh, but please can we mention company firewalls when we're discussing exposure risks, otherwise people are being disingenuous at least.) In fact, I spend a lot of time working to help companies apply patches more regularly so this isn't an argument about whether we should apply CPUs or not.

The issue is that I predict confidently (is that a guess, Alex?) that the majority* of Oracle customers aren't applying CPUs rigorously, so can we please stop kidding ourselves? Because until the issue of applying the patches is addressed, all the discussion about them might keep a few people busy, but to no material effect!

* Please note I did not say "all"

Updated later - here's an interesting related blog that I wanted to point out but thought I'd lost.
Posted by Doug Burns Comments: (28) Trackbacks: (2)
Defined tags for this entry: ,
Related entries by tags:
CPUs again ...
The Reality Gap (5) - Conclusion
The Reality Gap (4) - It's never the SAN
The Reality Gap (3) - A Single Instance per Server
The Reality Gap (2) - Disaster Recovery

Trackbacks
Trackback specific URI for this entry

The Reality Gap (5) - Conclusion
Time to wrap things up (if you'll excuse the festive pun). Not because there aren't numerous other examples of the gap between accepted wisdom and reality that I could laugh or sigh at, but because I want to revisit some old topics in the next week or so
Weblog: Doug's Oracle Blog
Tracked: Dec 27, 17:35
CPUs again ...
I'm pleased to see a few posts debating the (non)-application of quarterly Critical Patch Updates. Here are a few examples ...Is Poor Security Hygiene Rampant?Do DBAs care about Oracle’s latest Critical Patch Update?Survey finds that 66% of Oracle use
Weblog: Doug's Oracle Blog
Tracked: Jan 16, 20:53

Comments
Display comments as (Linear | Threaded)

#1 - jason arneil said:
2007-10-27 21:26 - (Reply)

Cheers Doug, the tenner is in the post! Hey, you missed my stage diving antics.
I confess to a dread of the CPU's being released, being able to install them online would be a massive boost to applying them in a timely fashion, and obviously the more db's your a managing the more of an automated solution you require. To be honest, I prefer waiting for a patchset than a cpu - bad dba.
btw, I also blog at http://blog.nominet.org.uk

#1.1 - Doug Burns said:
2007-10-27 21:34 - (Reply)

bad dba

Exactly, one of the consequences of the reality gap is that DBAs start to feel bad about themselves because they think everyone else is doing better!

Sorry I missed you presentation. Will you be giving it again at UKOUG?

#2 - Alex Gorbachev said:
2007-10-28 03:57 - (Reply)

The issue is that I predict confidently (is that a guess, Alex?) that the majority* of Oracle customers aren't applying CPUs rigorously

Well, talking in absolute measures, yes this is a guess. You just used another verb - "predict". ;-)

If you knew all Oracle customers and status of their CPU's, you could say how many of them applied CPU and how much they bother. However, you used somewhat ambiguous term - "not rigorously" and not 4,2%.

However, assuming you know enough Oracle customers in uniformly distributed set, you could claim that this is a statistically true statement. Though, in this case, someone could question uniformity and whether your set is representative.

Based on that I should conclude that technically it's a guess but practically, it's a statement based on your experience. The question is -- should we trust it? ;-)

OK. OK. :-) My personal point of view on CPU's is similar to yours and, in addition, I want to mention another point. As far as I can see, majority of customers have far more serious security holes than ones fixed with Oracle's CPU patches. Fixing those issues should be priority instead of applying CPU patches. Who cares about security patches if it's easier to become oracle Unix user than messing with exploits?

#2.1 - Doug Burns said:
2007-10-28 07:00 - (Reply)

Yeah, I always knew it was a guess, really ;-) A room of conference attendees is not very representative, or just talking to colleagues and others I know. If anything, though, you'd expect them to be *more* likely to apply patches. But, it's a guess.

... customers have far more serious security holes than ones fixed with Oracle's CPU patches. Fixing those issues should be priority instead of applying CPU patches. Who cares about security patches if it's easier to become oracle Unix user than messing with exploits?

Excellent point. That's behind a lot of my frustration, too. When there's so much discussion about the interesting stuff, it can distract from the reality that sometimes we haven't even got to first base yet (if you'll excuse the Americanism).

OK, time for (hopefully) my last Sunday morning refresh (well, let's wait and see about that).

#2.1.1 - Alex Gorbachev said:
2007-10-28 07:05 - (Reply)

QUOTE Doug said:
OK, time for (hopefully) my last Sunday morning refresh (well, let's wait and see about that).


And for me to go to bed but I'll keep my fingers crossed for your refresh(es). ;-)

#2.1.2 - Doug Burns said:
2007-11-02 21:59 - (Reply)

OK, time for (hopefully) my last Sunday morning refresh (well, let's wait and see about that).

Well, I was wrong about that :-( Hopefully this weekend will *definitely* be the last. Besides, I'll be in SF next Sunday ;-)

#3 - Paweł Barut said:
2007-10-28 10:05 - (Reply)

Another issue is Oracle Support for CPU. It should be handled with highest priority. I have a case were applying CPU on test server run without any issue, while causing ORA-600 on production system. And after week I do not have good solution for that. I had to roll back CPU, and I do not have time to experiment with next tries of applying CPU.

Regards,
Pawel

#4 - Noons said:
2007-10-28 11:24 - (Reply)

"Well, because they have to actually roll the patch out across hundreds of databases, not just talk about it!"


Bingo! Couldn't have said it better! One day it will finally dawn on the "minds" who come up with these grand schemes for bug fixes that installing a patch is not just plonking it in a single system and then blog ad-infinitum about its contents.

There is also the VERY REAL COST of testing its contents against a PRODUCTION system! And the equally real cost of installing it in potentially hundreds of systems, with the consequent downtime needed!

And they want us to do this every 3 months?

yeah, right: like, it's gonna happen...


"Oracle customers aren't applying CPUs rigorously, so can we please stop kidding ourselves?"

well, I've only been ranting about this state of affairs for a few years now, about time it caught on...

#5 - Chris Muir said:
2007-10-29 22:00 - (Reply)

To take things a little of track: "The difference between what Consulting Firms, Oracle Marketing, Technical Architects, Bloggers and Security Researchers say we should be doing and what most of us really are doing" or "The Emperor's New Clothes", isn't this indicative of Western Society as a whole?

For instance you rarely see rave reviews of the 1992 Ford Mondeo in the Saturday paper. Yet not a month goes by without a look at the concept cars of the future.

Admittedly the IT industry is probably the worst of the bunch given software's non-physical qualities.

To go slightly more off track: I feel a little sorry for user groups though in trying to deliver content applicable to their members' today-needs. They're caught in between a number of hard places. Usually they're volunteers (not always) with only spare time to help out, require sponsorship from the very organisations that want them to show the latest stuff to actually run an event (the alternative is raising membership fees to pay for events, which causes minor riots), near apathy from the user base in letting the user group know what they want to here (until the conference of course), and in an industry that is constantly changing directions. Besides what percentage of users will pay attention to any conference marketing collateral ranting on about Oracle 7 or Forms 4.5?

In turn the very people who tend to volunteer for committees are the people you referred to above, because they're the ones who tend to pull their fingers out. So they're involved in all that latest stuff, and they use their natural resources for obtaining content.

Usual $0.02 worth. Off to work I go!

CM.

#5.1 - Doug Burns said:
2007-10-29 22:08 - (Reply)

Off to work I go!

Timezone differences - I love 'em ;-)

... and, because I'm going to bed, I'll reply later ...

#5.2 - Doug Burns said:
2007-11-01 21:36 - (Reply)

Chris,

I'm not sure I agree, actually.

There is a tendency for people to want to hear about new and exciting stuff and Oracle's well-attended 11g stream at the recent OUG Scotland event was evidence of that. However, I think there's an equal or even greater demand (it's difficult for me to measure) for solid fundamentals from technical authorities that people trust. When I think about some of the most respected conferences ... Miracle, Hotsos, UKOUG, several in Eastern Europe etc ... then I don't think they're awash with New Features presentations.

Who wouldn't want to learn something new? The novelty might seem more attractive, but my personal experience is that a conference doesn't need to be about Fusion to be well attended or satisfy attendees.

Still, I don't envy the OUG volunteers.

#5.2.1 - Chris Muir said:
2007-11-02 03:57 - (Reply)

I'm not sure how you would measure that either. I did a little work last year on trying to find out what technologies some of the Victorian OUG members were using, but the replies were near 0. In turn requests for what topics should we present was also met by near 0 result.

(It could be that this problem is limited to Australia though... given our "no worries" attitude?)

That's not to say there weren't replies, but just not enough to see a general pattern, to then put on a presentation that you'd know would be well attended because of demand, and worth the time and effort or organising.

I always thought it would be kind of cool to publish each year, all sorts of statistics from the user base, about salaries, databases used, interests etc. Mainly from an interest point of view for both the committees and members, but also once you have those stats, you can start making informed guesses about popular topics beyond new features.

However the organisation and collation data for such a task, is again beyond the resources of most OUG volunteers. In addition given the apathy from the user base, would either meet with failure, or would take a couple of years to get running by building up momentum.

Maybe the best way to do it then, is to propose some sort of stream at a conference that covers old-tech, but with a catchy slogan or marketing angle, that people would pay attention to. Otherwise those "past-topic" papers are drowned by the new features marketing collateral.

You know, something like "Ensure your legacy system doesn't retire before you do" .... or .... "Legacy system triage" .... or .... "Keep the old girl running" ;-)

CM.

#5.2.1.1 - Doug Burns said:
2007-11-02 22:10 - (Reply)

I always thought it would be kind of cool to publish each year, all sorts of statistics from the user base, about salaries, databases used, interests etc.

The UKOUG conduct an annual survey and I know they also listen carefully to what the members want as presentation subjects. That's also partly an argument against my previous comments because if you take a look at the UKOUG agenda, there are tons of presentations on new development tools and the like. It's definitely not just old-school or DBA-focussed, but there's a good portion of that too.

some sort of stream at a conference that covers old-tech, but with a catchy slogan or marketing angle, that people would pay attention to. Otherwise those "past-topic" papers are drowned by the new features marketing collateral.

I definitely think that there's room for both. I also think that (unfortunately) people will go to see 'name' speakers, (or, more charitably, those they have grown to trust through the net) regardless of what they're talking about.

But I know getting things going is always tough for an OUG. The one in Scotland struggled for years and it's only through the really determined efforts of a few people that it's taken off again. (Oh, and lest I be accused of being egotistical, I am *not* one of those people!)

#5.2.1.2 - Noons said:
2007-11-03 01:23 - (Reply)

Chris,

for as long as AUSOUG insists on classifying anything to do with the database engine as "old-tech" - and other equally derogatory insinuations that anyone who doesn't "play fusion" is therefore incompetent - you will continue to get 0 feedback to UG initiatives.

Apparently, it doesn't sink with you folks that if you are getting 0 feedback and 0 interest, the problem is NOT with your public: the problem is with *your message*.

Why is it that the lack of interest started since the whole fusion rigmarole started? There is no such thing as coincidences, Chris.

But keep insinuating everyone in sight who doesn't buy into fusion is "old and a dinossaur": it's fun to watch and the disinterest reality speaks volumes as to why it is wrong.

Then again, never let reality or facts get in the way of "good" marketing, eh?

#5.2.1.2.1 - Chris Muir said:
2007-11-03 09:56 - (Reply)

Noons, I'm getting a bit sick of this.

Stop taring me with the same brush as the whole of AUSOUG. My interests in AUSOUG are I believe people should try to assist the UG rather than just being critical, full of hot air, and not actually do anything productive. Actions speak louder than words mate, and I've told you again and again, go take your grievances directly to the NSW chapter of AUSOUG, or run a presentation or something rather than hiding out on the blogosphere where you seem to "type-a-good-rant".

The fact that you also group all the branches of AUSOUG under one branch is also not fair. Ever attended a Victorian or WA event? Do you know of any of the personalities on those committies, how much work they've put in, and what directions they are trying to force AUSOUG national, and even if they agree with the national AUSOUG choices? I'm thinking you're going to answer a big fat NO.

Also you have no history of what battles I have fought in AUSOUG, what camp I stand in, what direction I've attempted to push it, as well as why I'm not involved in organising the conference series this year.

The fact that I'm here talking to Doug is I respect his opinion and I hope to learn from him about his experiences and ethics about all things UG.

And for the record, your insinuations about AUSOUG and nobody paying attention is probably blunted by the fact that this year's conference numbers have already broken last year's. So much for the user base not listening. Any other bit of trivia you're going to make up?

Like I said to you the other day Noons, I'm not the enemy, and I assure you, if you meet me face to face, you wouldn't spend so much time ranting at me in particular. I'm not that bad a chap.

Doug, apologies for leaving this on your blog but Noon's comment has been published.

#5.2.1.2.1.1 - Doug Burns said:
2007-11-03 10:26 - (Reply)

Doug, apologies for leaving this on your blog but Noon's comment has been published.

No problem. It's just a heated debate, that's all.

But, as someone who knows nothing about the situation down under, I have nothing sensible to add.

#5.2.1.2.1.1.1 - Chris Muir said:
2007-11-03 10:49 - (Reply)

Thanks Doug.

Anyway, I like the sound of what the UKOUG is doing. From all angles I'm hearing they do sound subjectively very progressive and I believe there is collaboration between the UKOUG and AUSOUG which is great to "here".

I think I'll take the UKOUG's survey idea on board and see if I can influence anyone at AUSOUG to pick it up.

By the way, at the UKOUG conference this year I believe the papers/presentations were picked and ranked by the actual speakers that submitted. Is this true? Any thoughts and experiences on how well this has worked, as well as any pitfalls from your own point of view?

#5.2.1.2.1.1.1.1 - Doug Burns said:
2007-11-03 11:10 - (Reply)

I believe the papers/presentations were picked and ranked by the actual speakers that submitted. Is this true?

More or less. The UKOUG ask for volunteers who are prepared to vote in specific subject areas. All speakers are included in that invitation as well as UKOUG volunteers and some members of staff?

Once everyone has cast their votes (and I hope I'm not giving away too much here because it's based on conversation, not personal experience), there's a long meeting of (I think) various directors and SIG volunteers and the vote influences their decision making, but they're not bound by it.

If anyone would care to correct any inaccuracies, feel free.

Any thoughts and experiences on how well this has worked, as well as any pitfalls from your own point of view?

I think the advantage of this approach is that it highlights any presentations that might be on particularly interesting *subjects* by first time speakers, as well as giving the people who've organised the conference year after year the opportunity to include speakers that they know have been good at previous conferences.

The one major complaint that everyone has, every year, is that they don't like the scoring system which is 1-4 if I recall correctly. However, having had a number of debates with the organisers about this, nothing seems to work well! Another difficult is that you're only voting on abstracts without seeing the presentation. So it's a difficult process and there's a temptation to give almost all abstracts a similar middling score.

I know quite a few of the people involved in the process and, believe me, they put a power of work into getting it right. They're trying to combine well known and trusted speakers, with some new blood every year and are determined to keep the subject range as wide as possible. They don't let RDBMS bigots like me influence them too heavily ;-) Equally, you'll find most RDBMS topics covered as well as any general Oracle conference I've been to.

It helps, of course, that there are so many streams.

#5.2.1.2.1.1.1.1.1 - Chris Muir said:
2007-11-03 11:35 - (Reply)

That's very interesting, though I can't see why anybody would want an RDBMS presentation at an Oracle User Group event, apparently it's "old-tech", especially at an Oracle related conference ;-)

When you say that UKOUG asks for volunteers, is that open to everyone within UKOUG (inc members), or only helpers/volunteers? If everyone, that's a great idea! Do you know what sort of response rate they get to this sort of request for helpers? And is it generally the same crew again and again putting their hand up? And what's the enticement to the volunteers beyond choosing presentations they want to see?

As for the people actually developing the conference schedule, having been there and done that, I know what you mean about the effort. The other things is you always end up in a situation where you can please "80% of the delegates 20% of the time".... and to extend the analogy "20% of the sponsors 80% of the time" ;-) It can be a thankless task.

#5.2.1.2.1.1.1.1.1.1 - Doug Burns said:
2007-11-04 07:51 - (Reply)

apparently it's "old-tech", especially at an Oracle related conferenc

Now, now ... take your fight elsewhere now, boys!

When you say that UKOUG asks for volunteers

Those questions I'm not sure of the answers too, but I might ask someone who knows more than me if they'd care to answer ...

#5.2.1.2.1.1.1.2 - Alex Gorbachev said:
2007-11-03 13:43 - (Reply)

I don't know as well as Doug does but here are my 2 cents.

QUOTE Chris:
By the way, at the UKOUG conference this year I believe the papers/presentations were picked and ranked by the actual speakers that submitted. Is this true?


This year isn't the first, actually. At least last year it was the same and I volunteered to evaluate presentations on one or two streams. Couldn't afford to find enough time this year though though. :-( It's quite a commitment as it turned out.

QUOTE Chris:
When you say that UKOUG asks for volunteers, is that open to everyone within UKOUG (inc members), or only helpers/volunteers? If everyone, that's a great idea!


I believe to everyone. And there are some incentives besides obvious community recognition. For example, APC posted about benefits of becoming a Session Chair at UKOUG.

#5.2.1.2.1.1.1.2.1 - Chris Muir said:
2007-11-04 01:55 - (Reply)

Thanks Alex

Doug, sorry for taking over your blog again but....

Alex, I see you're still presenting in Melbourne. I look forward to meeting you in person. Drop me an email at (email address deleted) and I'd be happy to give you a few suggestions on what you can do and see while in town (being an old Melbourne boy myself). Great time of year to visit.

#5.2.1.2.1.1.1.2.1.1 - Alex Gorbachev said:
2007-11-04 06:06 - (Reply)

Chris,

Just sent you an email.

I'll be in Melbourne early morning on Sunday and have the whole day free, actually, so yes suggestions are welcome!
The week before that I'll be in Sydney but I've been there couple times already. Never in Melbourne though.

Looking forward to see you as well!

#5.2.1.2.1.1.1.2.1.1.1 - Doug Burns said:
2007-11-04 07:42 - (Reply)

(Posted on behalf of Alex because he was getting an error when he tried to comment.)

Doug, I suggest you edit Chris' comment (if he is good with that) and remove his email in that "lightly obscured" format if he doesn't want to get spammed even more than he gets now. ;-) Spammers are smarter than DBA's these days and recognize those email formats. ;-)

#5.2.1.2.1.1.1.2.2 - Doug Burns said:
2007-11-04 07:55 - (Reply)

last year it was the same and I volunteered to evaluate presentations on one or two streams. Couldn't afford to find enough time this year though though. :-( It's quite a commitment as it turned out.

Snap!

For example, APC posted about benefits of becoming a Session Chair at UKOUG.

Yes, but be careful with that one - I think APC jumped the gun a bit there! See this blog posting from this year.

#5.2.1.2.1.1.1.2.2.1 - Alex Gorbachev said:
2007-11-04 15:52 - (Reply)

Oups... Thanks for correction - I missed that one. (remember how I "cleaned up" my Google Reader one day? ;-)

#6 - Chris Muir said:
2007-10-30 07:15 - (Reply)

Oops, please excuse my early morning bad spelling. here = hear.

#7 - Chris Muir said:
2007-11-04 07:46 - (Reply)

Not to worry, it all goes through Gmail's spam filter anyway, but thanks for the effort.


Add Comment

Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.
BBCode format allowed