May 20: Ambler the Agile
In an earlier guest submission, I made reference to following up a rather critical review of the presentation by Scott Ambler at Collaborate 06 on agile database techniques.
Ambler included a paper to go with his presentation, which I have examined.
I stand by those original comments. In particular, I am shocked by this comment from Ambler: 'you do not need to do data modelling up front, its a waste of time, its the kiss of death'. A thoroughly irresponsible and outrageous assertion.
What was presented that day was nothing less than the recycling of a previously short-lived IT fashion, namely RAD, with a few added extras. Remember that? Rapid Application Development. I thought it odd that Ambler never made mention of the close similarity. Then perhaps not, as we never hear of it nowadays. A brief web search will confirm the similarity between the two approaches. There are differences too - check out Simon Evan's blog for a balanced comparison of the two approaches. Although he makes the case for a difference between the two, the development of agile from a RAD baseline is pretty clear. So generically, one can see them as being from the same origin.
Ambler's paper on Agile Database Techniques is insubstantial and misleading. Lets look at some of the things he says:
'The goal of the Agile Data method is to define strategies that enable IT professionals to work together effectively on the data aspects of software systems.' Really? So what's new - some of us have been doing this very thing for decades. (PS - Don't you think that word 'method' should be capitalised? )
'Enterprise issues. Development teams must consider and act appropriately regarding enterprise issues.' I cannot understand what the purpose of stating the obvious is in this context. As if any self-respecting developer would do anything other than this. One is left wondering if by making non-controversial and obvious statements, some credibility is being attached to the agile fluff by association.
'IT professionals must work together effectively, actively striving to overcome the challenges that make it difficult to do so.' Would anyone ever recommend IT staff to do other than work together effectively? Again, another padding statement of the obvious.
Here is a definition: 'An agile DBA is anyone who is actively involved with the creation and evolution of the data aspects of one or more applications.' Now some DBAs have been doing this for decades - does this mean that a) they were always agile anyway, and b) the agile DBA is in fact an old concept? The assertion is simply misleading, as though the cult of agility is trying to sequestrate long established practices and concepts as something unique to their way of seeing things.
'The critical new skills for agile DBAs enable them to work in an evolutionary manner.' Yes, working in an evolutionary manner is part of the critical skills set, but they owe NOTHING to the agile approach Ô they have been in place for years! Interestingly, despite referring to 'new skills', nowhere are they actually enunciated, other than under the vague umbrella of terms such as 'evolution' and 'iteration', terms which are part of the bedrock of sound data and systems analysis and modelling.
A diagram is presented. Thousand words etc. Well, in a technical diagram, if one is presenting a new approach to doing something, one would expect the symbology to be defined. There are arrows here, but nowhere are they actually defined in terms of their purpose and function. Yes, we can guess, but in a technical presentation, is that good enough? The very process of definition itself appears somewhat weak in this paper.
The interesting concept of the sandbox was introduced, but in such a way as to leave open the question of whether it is an agile feature or not. Of course it is not. It is defined - as '... a fully functioning environment in which a system may be built, tested, and/or run.' 'Environment'? Unless that word is defined, the overall definition is simply vague to the point of uselessness. Why does the word 'isolated' not feature? Why not discuss the relevance of VM to the sandbox approach?
This paper is shallow, uninformative, and completely overlooks if not denies the serious and careful work done by conscientious IT professionals over many years. I take great exception to the way the advocates of new fads spend time rubbishing all that has gone before. Whatever Ambler may say, or allow his reader to infer, the agile approach is most certainly not the IT panacea he would have us believe.
For another examination of agile software in general, look at Wikipedia. Read down, and you will find a discussion of those areas where agile software may not be so suitable :
- Large scale development efforts (greater than 20 developers)
- Distributed development efforts (non-co-located teams)
- Mission- and life-critical efforts
Have a look at what Niall Litchfield has to say on the subject - I absolutely agree with him.
Speaking personally, I would prefer not to fly in an aircraft in which the computer systems were built by a team of agile programmers. This is not so flippant as it may seem. Ed Yourdon, in a talk to a meeting of New York City SPIN (that's the Software Process Improvement Network - yes really!), [see blog by Francis Hwang] noted the increasing litigation directed at IT systems. One of the sure defences against litigation is complete documentation, and it is this area which would make a failed agile system highly vulnerable. The agile crowd don't dismiss documentation out of hand, but would reduce considerably the amount that is provided with a system.
It seems to me that Ambler has taken some of the core values of data methodologies, wrapped them up in a flashy covering, and presented them as something new, while castigating the very sources from which he cherry picks. This is to be regretted, particularly when the IT industry as a whole is struggling to establish truly professional standards. To paraphrase a comment from a friend while discussing this subject, he suggested that the presentation style had more in common with an agile used car salesman than a serious IT professional - a point of view on which I simply could not possibly comment ...
Nice post Doug. I think that a lot of the "new" methodology comes from people who do not understand the long tradition of data processors/database engineering/database administrators that stretches back to the 60s. People in this class understand integrity is more important that agility, and will push back when they see integrity being potentially compromised. However, the "now, now, now; me, me, me" attitude is threatened by the ideal of continuity and competence, as it is essentially antithetical to the idea of a meritocratic criterion.
Thanks for this interesting and well reasoned post.
BTW, the most quoted and longest lived publications on software engineering are still those from the 60s and 70s, such as Weinberg, Knuth, Wirth, etc., etc. Who will remeber the books about "agile" methodology in 20 years?
"Thanks for this interesting and well reasoned post."
It's all Peter Robson's work, but I agree with you.
I also completely agree with Peter's post. It would be an interesting exercise to compare the levels of cost and effort required to provide long-term support and maintenance for properly modelled and developed systems against those for systems developed using rapid or agile methods.
Without any collected metrics this isn't possible, but my own off-the-cuff, gut-feeling for this based on a decade of working in a combined DBA/Support role in an organisation that takes data modelling seriously is that the rapid/agile products take about twice as much effort to support and maintain as those developed using what I suppose could be described as traditional methods. The rapid/agile products are also frequently devoid of any documentation or commented code - possibly a symptom of a lack of direction or control exercised during development.
Grown-up and serious organisations don't invest heavily in databases and modelling/development tools for the fun of it.
"Amblerisms" are nothing new, I'm afraid: this particular insidious form of "emerging" ignorant has unfortunately been pushing his disinformation for years now.
I've been extremely vocal about his total incompetence many times before at c.d.o.s., for many years. And by the looks of it the guy is still around and still spreading his total ignorance...
I was also looking at some methodologies/processes (RUP, XP, Agile). My result:
Some love "light-weight" methods, some don't.
There're + points and - points.
One of the articles I liked the most, came from Robert Baillie: http://robertbaillie.blogspot.com/2006/03/uk-oug-and-running.html
Somebody else about Agile DB: http://www.mayocchi.com/blog/agile-database/ (db release mngt)
I think we need *something* to build and evolve our databases (in all environments) in a proper way.
Whether you follow a "known" methodology or not, it should be clear for everybody! (I'm a consultant and at a lot of my clients it is NOT that clear!)
I'm using an obscure RAD technology that I've used off and on in it's various generations since 25 years now (short-lived IT fad, you say?). What I see on the maintenance side is the predictable mess. But on the plus side, enterprise software where you get all the source is pretty rare and desirable these days, especially in process manufacturing industries where you have to deal with food and drug governmental entities.
In the end, maintenance and incremental developmental programming winds up being about as slow as old 3GL stuff anyways. I'm not sure it is avoidable when you start throwing various generations/methodologies of EDI, transaction monitors, XML and barcoding stuff at it.
As far as the data modelling up front: I don't think that is necessarily incompatible with RAD, it just depends what you are doing. For example, if you are writing some product to compete with various existing products, you can abstract a fairly reasonable data model as you steal, er, emulate all the competitors features - and perhaps wind up with a better model than those others. If you are plugging in some EDI to an existing module, you aren't likely to get any real value from a formal architecting.
Completely unrelated article about San Diego-Edinburgh Sister City Society.
Word verification: pitvlwzd
pitiful wizard? pi tv lizard?
"I think we need *something* to build and evolve our databases (in all environments) in a proper way."
Yes, absolutely! It's called "database design". It's been around for yonks. It'd be good if folks like Ambler actually READ and UNDERSTOOD it before spouting off with their mental "theories".
Which 999 times out of 1000 have NEVER been tried in anything more serious than a shopping cart or some other similar infantile design. If that.
I've spoken to Java developers that are very excited about being 'Agile' and have tried to convince me that database development should go the same way.
But as much as I try, I just can't see anything useful in it.
As to the comment about data modelling being a waste of time - I doubt if Ambler could issue a single statement that would lose him more respect. What a joker.
I have just finished reading Scott Ambler's entry on Oracle Technet and it just sends chills down my spine. I won't comment in length here (but I now have another entry for my own blog...when I find time to write), but I will say that I am left with the impression that code coming from Agile Development is likely to be fundamentally flawed and unsupportable. By the time this has been revealed to the operations support groups and business users, the developers are off on another project.
During my career as a developer and then dba, the best, most stable software has come from techniques and methodologies where the focus was on requirements, documentation, sign offs, testing, etc. Yes, it took time, the development cycle was measured in years, not days. But the long term benefits of increased usability and decreased support costs far outweighed the gain from a reduced development cycle.
For another perspective, Sue Harper has an entry which offers a more reasonable view of Agile Development. It is at http://sueharper.blogspot.com/2006/03/agile-again.html
"I am left with the impression that code coming from Agile Development is likely to be fundamentally flawed and unsupportable. By the time this has been revealed to the operations support groups and business users, the developers are off on another project."
And that, folks, is a perfect definition of the entire body of work from Ambler y sus muchachos.
IOW: change "paradigm" before anyone can catch up with the mess their "theories" leave behind.
I've loudly complained to Oracle in more than one occasion that he should NOT be allowed to publish ANYTHING in OTN: it's mostly deranged, unscientific ranting.
OTN claims that he's entitled to a voice there. Obviously, someone on the inside likes his stuff. I think I know which group it is...