Feb 8: Time Matters: Throughput vs. Response Time
#1 - Chen Shapira said:
2009-02-09 01:32 - (Reply)
Good discussion 
1) When you measure time to run a business process, and there is going to be a human waiting for the results, you want to tune the 90 percentile response time. Because usually they start complaining when around 10% of the transactions take too long. Convincing them that everything is fine on average doesn't normally have the desired effect. I'd love to see the "90th percentile response time" in this benchmark. I suspect that variance gets quite bad very early in the game, and that averages don't tell you the whole story. Less loaded system is not just a faster one, it is also a more predictable one.
2)There is also the "refresh effect". Any click that takes over 10 seconds to respond with cause a user to refresh his browser window. If your application transmits this to the database as a second request - you are pretty much doomed. Any temporary slowdown turns into something you cannot recover from. Queue theory nightmare - the longer the wait time, the higher the arrival rate.
3) Keeping the workload at 16 vs cranking it up to 24 will leave some capacity to handle unexpected requests, which are statistically quite expected.
#1.3 - Doug Burns said:
2009-02-10 05:28 - (Reply)
Chen,
you want to tune the 90 percentile response time. Because usually they start complaining when around 10% of the transactions take too long.
Whilst I know that that's what's commonly used, why be so fixed on 90? I've worked on systems where people wouldn't accept one in 10 responses being too long! It also depends on how much longer we're talking about.
But I know what you mean and 11g DB/Grid Control has some really useful additions in this area that I'll blog about soon.
I just worry about what seem like certain statements that I'm not sure apply in all situations.
I'd love to see the "90th percentile response time" in this benchmark.
Sadly, it's not available from Swingbench, but the coming blog post will use 11g DB Control, which is better at showing that.
I suspect that variance gets quite bad very early in the game, and that averages don't tell you the whole story. Less loaded system is not just a faster one, it is also a more predictable one.
Yes, of course and quoting the average response time made things look better than they really were. In the context of the blog, though, when I was trying to illustrate that bad response time *might* be good, I would have thought I was erring on the side of caution and doing the response time side of the argument a favour 
That's the thing, I was trying to write a blog to say - 'look - here is a case where lousy response time = best results' and everyone just wants to talk about response time all over again! LOL
#1.3.1 - Chen Shapira said:
2009-02-13 18:40 - (Reply)
I don't have any special attachment to 90% specifically.
However, we do agree that it will be impossible to tune for 100% of the queries, and that tuning for 50% is probably not good enough.
It'll be nice to do an empirical study of when users get frustrated
And I'm sorry for derailing the discussion away from throughput...
(BTW. Did you see that the guys in Pythian say you are old?)
#1.3.1.1 - Doug Burns said:
2009-02-14 06:43 - (Reply)
Hi Chen,
(BTW. Did you see that the guys in Pythian say you are old?)
I know! Pesky kids!
However, we do agree that it will be impossible to tune for 100% of the queries, and that tuning for 50% is probably not good enough.
Yes, we agree about that.
I'm sorry for derailing the discussion away from throughput...
Don't be. Any frustration on my part might include an aversion to anything that sounds like dogma but also contains a large chunk of frustration with myself because I obviously didn't communicate what I was trying to very well.
#3 - robyn 2009-02-09 15:14 - (Reply)
My premise on variance was derived from this issue. I needed to improve performance in a data warehouse, but my questions about 'Which processes were most critical?' or 'Which processes are the performance issues?' met with blank stares. The only target metric anyone could verbalize was that batch processing needed to complete by 2:00 am. Digging deeper into the nights that failed to meet the target led me to a handful of processes that performed inconsistently, e.g. processes with high levels of variance.
Most of the problems I'm asked to solve are throughput issues. (all processing must complete in X hours) Investigation usually leads to finding the individual processes with the response time issue. Once the individual response time issue is improved, the overall throughput also improves. If the overall throughput is satisfactory, the fact that individual process response time is slightly slower tends to go unnoticed. (but this last corollary doesn't apply to actively watched processes.)
The two targets are interwoven, and ignoring one to optimize the other would be silly. And I like the comment about AWR and StatsPack - both tools are geared toward system performance as a whole and they are best used to monitor systems that have already been optimized at the process level. Any real improvement I've ever made in a system has come from improving individual response times within the workload. Fortunately we don't have to chose one just one tool or one approach.
#3.1 - Doug Burns said:
2009-02-10 05:43 - (Reply)
Robyn,
I don't know how much you've looked at 11g DB/GC, but if not much, you're in for a treat. It's Variance City!
Investigation usually leads to finding the individual processes with the response time issue. Once the individual response time issue is improved, the overall throughput also improves.
Mmm, yes, that's often the case (and I did say 'all of these tests would have yielded even more impressive results if I had tuned the SOE application at the session level *first*'), but it's not what I've illustrated here. All sessions are running precisely the same transactions. None of them had their performance improved by tuning. Which were the 'problem' transactions was completely variable, by which I mean they were all equally bad at different times, because the system was hopelessly overloaded.
... and yet, the results from a business perspective got better. Didn't they?
Any real improvement I've ever made in a system has come from improving individual response times within the workload.
I almost took issue with that statement, but it depends on what you mean by 'system'. If you mean application then I probably agree but, if you mean system in the widest sense, including the workload, the performance of the infrastructure etc., then didn't I just 'improve' the system in a very real way without improving the response times? I made them *worse* and I didn't tune the application one iota.
The thing is, I know as well as the next person that all the real gains tend to come from application tuning and individual session response time tuning, but I think it's a pretty limited view of performance that I was trying to challenge a little bit.
I'm not sure I did a very good job, though
#5 - Chris Adkin 2009-02-10 16:31 - (Reply)
Your comment about AWR and statspack makes the use of these tools sound like guilty pleasures . At the risk of falling the wrong side of the extended tracing brigade, I've used db time on a project which involved the bench marking of batch processes and I found it really useful. Because of the shear number of things that could be tuned and monitored across the software stack (which included a J2EE app server), db time worked well as a high level metric and stopped us from drowning in statistics. As you are probably acutely aware, you tune one thing and the bottleneck usually moves somewhere else. The batch tuning work I did was a bit like pushing down on one end of a see-saw, i.e. if the load on the database was reduced it would increase on the app server and vica versa. Using db time made this effect easy to see.
#5.1 - Doug Burns said:
2009-02-11 06:32 - (Reply)
I wondered how long it would be before DB Time put in an appearance
I, like you, have had nightmares trying to tune batch throughput using a single session view. It's not impossible, and Robyn highlighted what sounded like just the right approach, but it gets more difficult when you've tuned your application to death and there's still variable SAN performance or some other mystical factor (that is meant to be a joke - there should be no mysteries) that screws up your carefully laid plans.
As for Statspack and AWR - don't start me! It hurts my head that apparently they shouldn't help me solve performance problems but they still do. Every time I solve a problem and haven't traced a session, it makes me doubt my sanity and I know I must be doing something wrong.
#6.1 - Doug Burns said:
2009-02-14 06:59 - (Reply)
Do I care more about throughput or response times?
That's what I was trying to show. That there is more than one tuning goal - individual session response time - and that they might conflict with each other.
with your homogeneous swingbench test
That's a very important point. I'd hoped that nobody thought that this looked like any realistic system but it's probably to other peoples credit that most of the comments (including one or two emails) started by trying to apply my comments to their own real world systems and pointed out that this was all a bit, erm, artificial. It was meant to be, but I didn't realise how much that would undermine what I was trying to say. I think it's safe to say I was wrong to over-simplify things, too, but that was a deliberate approach (albeit an ineffective one! LOL)
It was precisely *because* the workload was so limited, repeated across all sessions that all started and ran for the same length of time that I thought it acceptable to use high level averages.
Then again, the nature of the workload also means it would have been safer to look at one of the sessions individually, too!
This has allowed me to save time by not tuning at all when the bulk of the time was outside the database (most of this was done just using DB Time, Elapsed Time, txn/sec from AWR reports)
Very interesting point because it reflects my personal experience - that DB Time is often an extremely effective measure of when the database workload is *not* the problem. Funny how often that is the case these days
Oh, and I noticed you're on the Hotsos agenda this year. Good luck with that and I'll be interested to hear how it goes.
#7 - Graham 2009-02-13 23:15 - (Reply)
Interesting blog post and discussion. I'd like to raise a couple of issues from the original post.
Firstly there is the issue of the average response time increasing from 79ms to 587ms (not metres/seconds Doug ). Frankly, an end use is not likely to notice a 0.5sec increase in response time of a transaction. The actual response time seen by the user will, of course, be longer; there are likely to be multiple networks and probably middle tiers involved. I do agree that in the fully loaded system there will be a much higher variance in the response time and users will notice increased response times once it gets beyond a couple of seconds for outlier transactions that are at or near the max transaction time. I have in fact been involved with systems were data was actually delayed in being returned to the users so that a more consistent user response time was perceived i.e. slow down the fast stuff, rather than speeding up the slow stuff. But again we were talking of providing response times of a small ( less than 2) seconds.
Secondly there are the results for the throughput of the system are not consistent. Why would throughput drop from 16 users to 20 users to 24 users but then rise quite dramatically at 28 users before dropping again? It seems that it is likely that there were other factors involved here. Typically throughput will start to drop when we are approaching the limit of a resource, disk, CPU, software etc. This is just simple queuing theory. Were these results from a single run of each concurrent user count? I would not expect the surge in throughput at 28 users to reproduce.
#7.1 - Doug Burns said:
2009-02-14 07:20 - (Reply)
Graham - your less than sign cut off your comment initially, which confused me for a moment. It's fixed now.
(not metres/seconds Doug )
Don't Oracle guys use metres/sec, then? You don't know what you're missing - better than that DB Time nonsense.
(Embarassed grin)
Good catch and fixed.
Frankly, an end use is not likely to notice a 0.5sec increase in response time of a transaction.
Oh, ok, it won't matter to a user, but if I'd picked a different transaction with a higher single user response time then that 0.5 sec would, in turn, have been much higher.
there will be a much higher variance
There was. My original table had the maxium transaction response time, too, but I took it out because I thought it would add more detail that would detract from what I was trying to say. Probably a mistake, in retrospect, because it would have shown the increasingly erratic behaviour of the system.
It seems that it is likely that there were other factors involved here.
I don't doubt that there were.
This is running on a laptop where, with all my best efforts to hold my breath, close other programs, not touch the thing, avoid AWR snapshots, etc, etc ... there are no guarantees that some background service or other wasn't a factor.
But that's why ...
Were these results from a single run of each concurrent user count?
They weren't based on a single run. I tend to be quite paranoid about these things before showing numbers and probably perform even more runs than necessary, just to build my own level of confidence.
Now, whilst not every run showed the same surge at the same workload levels, if there was a large discrepancy, I repeated runs to check that.
In the end, I decided that
a) At those higher workload levels, the laptop is, frankly, on it's *rse. (I'm sure I could put it more delicately, but think you'll appreciate the description.) That's going to cause such variable madness, I decided it was a lost cause to try to get consistent looking figures.
b) I thought I had illustrated clearly enough that increased transaction response times are the consequence of the increased workload but that *might be what we want*! Sorry to sound like a stuck record, but I think it's necessary in a world where people think the only tuning is single session response time tuning.
I should know better than to try to simplify any performance testing discussion too much, though, but it was worth it for the discussion in the comments.
#8 - Tim Hall said:
2009-02-15 18:24 - (Reply)
Hi.
Nice post. I think you've done a great job of putting into a few simple sentences quite a complex issue. I'm sure I would have written twice as much and bee half as clear.
Cheers
Tim...
#12.1 - Doug Burns said:
2011-06-13 14:25 - (Reply)
Hi Matteo,
I have had a number of follow-up comments asking for more detailed metrics from this test but the test, the hardware and the results have long since disappeared into the ether so I can't help you I'm afraid.
However, I do know that the results were both accurate and illustrated my main point, which was that response time is not the only way to measure performance.
How this fits into Gunters USLaw I couldn't say, but as it's just data and not a model, maybe it doesn't count for much anyway
Metric Baselines were designed to be easy to implement. There are only two options :-1) Pick how much recent activity you want to use and let Oracle recompute the statistics based on the most recent metric values over time (Moving Window). 2) Pick
Tracked: May 07, 18:39
Tracked: Jul 07, 04:28
Tracked: Oct 03, 15:27
Well, that was a nice start to the day! Someone came up to me just before Wolfgang Breitling's presentation to point out that something from my presentation had helped him fix a problem at work last night. It was the OEM Raw Data drill-down that shows you
Tracked: Mar 10, 17:43
Tracked: Mar 16, 00:23
In one of my Hotsos Symposium 2010 posts I mentioned that Peter Stadler had plugged some test results from an earlier blog post into Neil Gunther's Universal Scalability Law to see how well the model applied. Peter's posted his slides now and I've added t
Tracked: Mar 28, 19:35