Doug's Oracle Blog

[One thing that's great about jet-lag is that it allows you to catch up on blogging and all the email that's built up while you've been away at the conference. Not much else you can do at 2:30 in the morning.]

I'm glad I went to the Hotsos Symposium again this year. Attending conferences is an expensive business when you're a contractor as it means more lost income, so I can't go to every conference I'd like to attend but this one should probably remain near the top of my list.

Lows

- My presentation, obviously. There was a funny moment in the office today when I was looking at the OEM Top Activity page - because, yet again, it had highlighted a performance problem we were previously unaware of - and our architect said perhaps I shouldn't be using it because I couldn't be trusted not to break OEM!

- Somehow contriving to miss both of Richard Foote's presentations but at least I was able to enjoy his entertaining company at dinner one night.

- The weather - not what I had in mind at all!

Highs

- The people, as always. Not only are there lots of smart speakers and attendees at Hotsos, but the informal setting makes it easier to catch up with them. It was particularly good to get time to speak properly to a few people I've only met briefly before, buy Kerry Osborne that beer and to see Polish Paul Matuszyk there after I recommended it to him 4 years ago! As for Monique, she's one of a kind

- Disco Night. Can I request that we have more party nights themed on women wearing glittery mini-dresses and Oracle geeks dancing in duck costumes?

- Alex G playing chopsticks on the piano in a large suite. I was expecting something a little more elegant!

- Quite a few presentations, not least, Bob Sneed's CPU QoS, Kevin Closson ranting intelligently about where we are today and where we're heading, Alex G reducing the room to fits of laughter whilst reinforcing the Battle Against Any Guess message and Wolfgang's Anatomy of a SQL Tuning Session.

- Feeling inspired to think about performance even more now that I'm back in the office.

So, expensive or not, I'm glad I went but next year I might stick to text-mode demos that don't need a network connection!

I generally wouldn't visit the Hotsos Training Day, mainly because I've been away from home and work for long enough, particularly when you add the travelling time at either end, but this time I was determined to attend because Tanel was presenting.

It was a busy room with a very high percentage of attendees staying for the additional day. I suppose once people were actually allowed out of the office in the current economic climate, they figured they might as well pack in as much learning as they could. Tanel described it as one of the peaks of his career and I can understand that. It's an honour to present at the Symposium, so a whole day must be a pretty special privliege, but he's more than earned it with the number of good presentations and blog posts he's come up with.

It didn't get off to a good start, though, as there was a problem with the focus of the projectors, but that was soon adjusted a little and people were happy as long as it was good enough to read the text.

Speaking of text, he kicked off spending quite a bit of time talking about using the right tools for the job - sqlplus, basically - and how we can make our own performance more efficient before we even start looking at performance issues. Make no mistake - despite having a love of pictures these days, I'm fundamentally a command line chap who frequently finds himself doing the things Tanel talked about, which consequently made me chuckle.

- When diagnosing Oracle problems reported by others, I ask them to stop using TOAD, their JDBC application or whatever it is they're using and login to sqlplus. Once the problem is recreated there, I know it's a real problem.

- On windows I use the CMD prompt version of sqlplus too and can't stand sqlplusw (but I should perhaps keep that quiet because lots of others seem to like it )

- On Linux he spent a lot of time on RLWRAP and command line completion and there were some really good ideas in there that I don't use enough so I'm going to revisit them.

He showed some neat and entertaining tricks with colours that I took some photos of but they wouldn't really do them justice. I'm expecting him to have a movie playing from sqlplus next year

While talking about his Snapper tool, he drew the comparisons between it and ASH data, which he was pretty positive about and mentioned how ASH data is like a DW fact table with multiple dimensions that allow flexible and relatively complex drill-down into specific dimensions to help diagnose performance issues. That's something I talk about quite a lot when I'm teaching people about ASH.

But he came at it more from the angle of his own Snapper tool, which is cool if you've never taken a look and between that and the section on latchprof and latchprofx, demonstrated pretty effectively that these tools allow you to look at most problems and to limit your investigations to those that matter. For example, for someone who is known for his work on diagnosing latch contention problems (amongst other things) he was very careful to point out that you shouldn't become obsessed by latch contention unless it's causing a problem! He didn't stop there, though, next up was the utility that I know a few people have been waiting for - the Mother Of All Tuning Scripts, or MOATS.

I wouldn't want to steal any of Tanel's thiunder by blogging about it in too much detail, I'm sure he'll do that himself, but MOAT is a bit like a top utility for Oracle. Although other such tools exist, e.g. OraTop, I've had a quick look and I think MOAT is probably more extensive and only requires some PL/SQL and SQL. I was pleased to see him acknowledge that it was something that he worked on with Adrian Billington who is the guy behind oracle-developer.net and someone who I've enjoyed a few beers with since I started working in London again.

Next up were SQL Performance Tools and thank goodness he re-emphasised that using the EXPLAIN PLAN FOR command is asking for trouble, particularly as there are far better tools these days to look at the plan from the child cursor instead. (My TOAD equivalent for this in the office is to berate people for using that bl**dy ambulance! LOL)

Unfortunately there were some growing Production problems at work that demanded my attention so I had to bail out early to work in my room which meant that I missed most of the SQL Tuning and Visualisation sections but I knew the Perfsheet content already and you can count on me revisiting the whole day once I'm back in the office. I particularly liked his course notes - not too detailed to be unwieldy, but just enough detail to remind me of the day.

Although it was a really enjoyable day, I obviously knew a lot of the content already, partly because I have similar interests and partly from reading Tanel's blog. It made me wonder why people who go to Hotsos don't read blogs and stuff the rest of the year, because I sensed a lot of the content was completely new to some others in the room. I suppose now they've found his blog, they might start.

After a few hours working in my room, things were sorted out and I could go to the bar across the street for a burger and a few beers with Marco. As soon as we walked in, Tanel shouted over and there were was an interesting little crew in there, with Tanel basking in the glow of a job well done. It was a shame about the incredibly loud karaoke or I might have stayed longer but it's safer to keep me away from karaoke - although I was quite looking forward to the Estonian version It was a good evening, though, and ended up with quite a few of us in the hotel bar. My main memories are of me defending 'my' pictures and Tanel and I praising Mr. Billington to the heavens.

The "Adrian Billington must be allowed out to a conference" campaign starts here! We won't take no for an answer.

There she goes, squirreling into the corner of your carry-on.  Ready for another adventure.  Tucked between a fun floppy hat and the flourescent sun-block, of which she uses plenty, pale creature that she is, Monique is situated to prove to you that a seasoned traveller knows best.  Dressed in a curly mohair (so as not to show the wrinkles that betray one new to the jet set) and sporting a pale chiffon scarf - to double as a disquise in those cities where she is already known for her somewhat silly escapades - she seems to wink at you and beckon you along.  Forget your cares, but by no means your playing cards, and step lightly into whatever dreams your next voyage should hold for you:  let Monique be your guide.  

Her very presence alleviates all the maladies brought on by the jostling of trains, the dipping of ships, the swerving of buses and even by the simple fact that you are far from home.  For she has discovered the secret to successful travel; with the exception of the luscious few minutes in the X-ray machine, where she delights in making funny faces at the operators searching for more dangerous contents.  Monique never looks back.

Charleen Kinser Designs

(With thanks to Carol Dacko for bringing along an interesting dinner companion for my own cuddly friends. Somehow I doubt they'll ever be the same - she's terribly sophisticated for my lot!)

I heard someone bemoaning the lack of Swag at Hotsos - just an event program, couple of magazines and a small clockwork toy sponsored by Oracle. I don't think I ever come to the Symposium expecting Swag - it's just not that kind of event. But you do tend to get good speaker gifts, including the best speaker gift I ever received. This year, it's a digital photo frame.

As for that clockwork toy, the Cuddly Toys think it's *brilliant*!!! At first, their reaction was "What's THAT?!" then when I wound it up and let it dance in front of them, they fell in love.



The only thing is that it doesn't run for long per wind-up, so my right wrist is aching from them continually beggind me to "Do it again, Douglas!"

Then again, their reaction wasn't quite as relaxed as when they met Carol Dacko's friend while we were at the restaurant. Apologies for the quality of the photo, it was dark in there, but at least it proves that Little H and Chris were allowed out for dinner and that not all Cuddly Toys are made the same.



They were just stunned by their new friend. What's THAT?

All will be revealed in the next post ....

First up was Cary Millsap's - Lessons Learned, Version 2010.03 As Cary pointed out, they always try to put the best speakers in the toughest slots - 8:30 in the morning post-party. I think local guys are slightly more reliable too because they might have actually gone home the night before! He started with a quick Hangover Survey (me - check!) and then pressed on talking about how we test system performance.

He showed a video of Boeing stress-testing the wing of the 787 and, as he pointed out, aircraft manufacturers really know how to stress-test! (Of course whether that reassures you as it does me, or makes you wish no-one would talk about wings disintegrating, as it probably would Mads, is personal.) They showed Boeing test equpiment which is complicated, expensive and non-revenue generating. Those tests are expensive but when people's lives are on the line, what choice is there? Boeing knows that it has to test the analytic models used in the design. He spent a lot of time talking about good test design. A few thoughts that stood out to me ...

- Some stress tests are a waste of time. Will the Boeing 787 land on the moon? If this test fails, what has it proven? If it passes, then it's awesome but it would be a very expensive way to prove it can cope with commercial flights in Earth's atmosphere.

- Why test for more than you will see in Production? Because you don't really know for sure what you'll see in Production.

- At some point, but I can't remember the context, he used a Scottish phrase that he'd heard Billy Connolly shout (although the Big Yin was only fully credited later in the day) ...

    "There's no such thing as bad weather, just the wrong clothes"

... looked over at me and said - "I'd love to hear you say that, with the proper accent". I declined politely.

- Most people try to prove only that their systems will work.

- Most tests of systems that are destined to fail never proved it in advance.

- Test to destruction

    a) Test
    b) Until the system melts
    c) Decide whether your real requirements are likely to be lower or higher than melting point.

There was a small amount of time for questions and once it looked like they were done, I granted Cary's wish (never thought I'd say that), stuck my hand up and repeated The Big Yin's words. It was only after the laughter had stopped that I realised I might have ruined his big closing, but I think he was ok about it

Next was Tanel Poder talking about LGWR, log file sync waits and COMMIT performance and shock, horror, I was actually going to say that this was one of the least rewarding sessions of the week for me. What?!? Tanel? But he's, like, an Oracle God! LOL But there were reasons

- I realise that I know a *lot* about how log file sync and log file parallel write work, how they relate to each other and some of the problems they might help you identify. Because it's a subject I'm *so* familiar with, I didn't learn much.

- His main demo didn't quite show what he wanted it to because it didn't run multiple sessions but, frankly, I'm in no position to talk about demos this week!

By the end, the presentation turned out ok, not least because there was another unexpected appearance from Bob Sneed to talk about the I/O components involved in  redo log management including a suggestion that LGWR be put into a higher scheduling class (but not Real Time!) Updated later - make sure you read Bob and Kevin's comments below. I'll try to find a link to his slides and let you take a look yourself.

I loved Tanel's Big Log File Sync Tuning Secret, though ...

    COMMIT LESS!

It was particularly relevant to me because I had a Big Log File Sync Tuning Secret as the closing moment of my own presentation. The problem was I couldn't use it after the demos went wrong!

    USE ASYNCHRONOUS COMMITS

But, in my case, that was supposed to be funny, too.

I ran off to try and use the free breakfast voucher that Marco had given me but I was just too late. No food again, then Well, I had a couple of slices of cold meat at lunchtime, but mainly to catch up with Alex G before he had to present and then head back to Ottawa. I managed to skip one session at this stage but, after a quick call home, I decided to go along to Alex's RAC Connection Management presentation after all (a little late). Although I have seen some of this stuff before, I always enjoy watching Alex's demos and was particularly impressed by the fact that he'd managed to write his own RAC connection load balancer! I was waiting for the applause in the room but either people didn't quite get it or there was just a lack of energy post-lunch on the last day. I suspect the latter.

Of course, once I'd said goodbye to Alex properly (don't see him nearly enough), I was a little late for whichever session was going to be my final one of the conference and I was hopelessly torn between Kyle Hailey's modern SQL performance tools presentation (Kyle's done a lot of cool work in the area of Oracle Performance Visualisation) and Chris Antognini's Diagnosing Parallel Executions Performance. In the end I plumped for the latter because I thought it was going to be like something I'd unsuccessfully attempted a couple of years ago and I wanted to see if Chris had a different angle on it and had been more successful. In the end, I probably made the wrong choice because although Chris' presentation was great, it was really all stuff I already knew. Definitely my bad call, though. Hopefully I'll get a chance to catch up with Kyle's presentation at some point in the future too!

After that there was just the usual short farewell and thanks from Gary Goodman of Hotsos. Although the thanks were appreciated, I'm glad they were spread around everybody because the attendees are one of the things that make this conference great and Becky and Rhonda did their usual sterling job of organising everything.

Then it was time for some Fajitas with a few friends (actually, a whopping great number of friends who practically filled the Mexican restaurant!) and a few very sedate beers. (We are old men (and women) now and the night before was a big one!) While we were waiting to go to the Mexican, I had one great surprise left - Alex's flights weren't going to get him home, so he came back from the airport and had to check in overnight! At least I got a chance to talk to him properly when I wasn't hopelessly drunk and didn't try to seduce him this time.

Now I need to stop blogging and get back to listening to Tanel's Training Day (good stuff, too, but more about that later)