Mar 20: There's Hope for us all

I empathise with Dom Brook's recent blog post. We've all been there and I don't just mean being unable to write a technical post when emotions and opinions need to be dealt with first ;-) In fact, a day wll come soon (I'm full of promises, eh?) when I'll have a wee rant about the state of the industry myself. Isn't that what blogs are for sometimes?

However, what this post is really about is a comment that Tim Gorman sent to the Oak Table mailing list, where Dom's posting has been today's favourite topic. I believe he posted it on Oracle-L first. Although it doesn't reflect my personal experience in the slightest, there was something about what he had to say and the way he said it that rung very true to me and I wanted to post it here.

"Back in the 1980s, I worked for a company that had built some really cool applications in the area of travel reservations. Eventually, the travel providers (i.e. airlines, hotels, car rental agencies, etc) caught on to what we were doing and did it themselves, effectively putting us out of business overnight. So, it came time to sell the company off in pieces. We tried to sell the applications, but nobody wanted them -- they had their own, or could buy or build better. We sold the hardware and facilities, but for pennies on the dollar. Then, when we tried to sell the data, we hit the jackpot -- everybody wanted the data, and we were able to sell it over and over again, to multiple buyers.

I never forgot that lesson, and several years later traded being a programmer for being a DBA because (as Michael just said, below) I like working with data. Data, not programs, is the only thing that matters -- applications are transient and have no value except to acquire, manipulate, and display data. Data is the only thing with value. The long-term value of data is the reason I've moved toward data warehousing and business intelligence, too.

Data is important. Databases manage data. DBAs architect, configure, and manage databases. So, being a skilled database administrator will always be necessary as long as data exists. If the state of the art ceases advancing, then automation will finally catch up to extinguish the DBA role/job. But until then, being a DBA is a career.

...and that's my story, and I'm sticking to it, regardless of the latest flavor-of-the-month heralding the death of the database..."

Thanks, Tim.

Posted by Doug Burns Comments: (34) Trackbacks: (2)

Trackbacks
Trackback specific URI for this entry

This is one of the web's most interesting stories on Fri 21st Mar 2008
These are the web's most talked about URLs on Fri 21st Mar 2008. The current winner is ..
Weblog: purrl.net |** urls that purr **|
Tracked: Mar 21, 14:08
Data Matters, but So Do Algorithms
I just read this post about the importance of data over applications. It reminded me of a passage from Eric Raymond, in The Cathedral and the Bazaar, where he paraphrased Fred Brooks: Show me your code and conceal your data structures, and I shall cont...
Weblog: Tekhne
Tracked: Mar 21, 19:53

Comments
Display comments as (Linear | Threaded)

#1 - Dominic Brooks said:
2008-03-20 22:22 - (Reply)

Solidarity Brother!

I'm sure Tim's right and will continue to be right.

I was having an emotional moment... the flat earth society are everywhere!

#2 - Joe 2008-03-21 13:50 - (Reply)

For the kind of stuff I build at work, he's totally right. But I'd wager a significant sum that my databases at work will have less long-lasting value than Emacs.

Of course, Emacs manipulates data, like most applications. Then again, much of the data it manipulates is the sourcecode for other applications.

For an even purer counterexample, how about Mathematica. When you generate a 3-D graph of a complex equation, what data did you manipulate? Seems to me the answer is: none whatsoever. And I'm pretty sure my data is more transient than Mathematica as well.

And what about SecondLife? Is the data in there really important? Does it have any value whatsoever apart from the application?

#3 - Anonymous 2008-03-21 13:56 - (Reply)

Man, Second Life is all about the data that the users create and exchange.

#4 - Sergio 2008-03-21 14:07 - (Reply)

Joe,

I'd argue that in all three of your examples (Emacs, Mathematica, and Second Life), the argument can still be made that the data is more significant than the tool that manipulates it.

Emacs: without any source, text documents, etc. to edit, you'd have a really nice way to put characters on the screen and run some logic (I know, I know, you could probably implement an OS with Emacs, but you get the point)

Mathematica: Here again, if you're only interested in single session use, then the ability to save your work may not be of value. However, if you're like most people, you'll want to save the equations you're working with.

SecondLife: As with the previous examples, SecondLife by itself is a great virtual world simulator. However, if I had to recreate all the structures inside everytime I used the program, I'd get pretty frustrated in a hurry.

The point is that it is the data being manipulated by these programs that provides lasting value. It is the data that can be transferred between apps and re-used long after a particular data manipulator is gone.

#5 - Anonymous 2008-03-21 14:26 - (Reply)

Nope. If that were true, they could just as easily have made the interface email, a wiki, or a db login. SecondLife is ALL ABOUT the interface--that's the entire point.

#6 - Mike Cohen 2008-03-21 14:43 - (Reply)

Typical database guy's view of the world. Still though, hard to fault him for his error in logic. http://www.nizkor.org/features/fallacies/hasty-generalization.html

It was probably fairly traumatic to watch the company you worked for quickly go out of business, and would probably lead most people to draw erroneous conclusions from life-changing events going on around you.

Data in a database is worthless in and of itself. It only becomes valuable to the business when an application is available to let the business users view and manipulate the data in specific ways. In the end, it doesn't matter if that data sits in an rdbms, in xml files, in flat files, comes from web services, whatever, the important thing is the application that exposes the data, often in new and interesting ways.

The data behind Google Maps was there for how long? It was how valuable to a few cartographers? The GMaps application is what counts, not the data.

#7 - Doug Burns said:
2008-03-21 14:56 - (Reply)

Just deleted a comment. I don't usually but call people idiots, argue the case but don't just stream a trail of vitriol that doesn't even address the argument (if there even is one!)

Tedious.

#8 - dave 2008-03-21 15:57 - (Reply)

I'm going to have to side with author here. Applications are the conduit, the data is the prize.

Of course, insofar as the application is able to manage the data it is not insignificant. But it is the data that counts.
When the data is unique and cannot be re-created/recaptured and for some reason it gets lost, that is when we find ourselves up shite creek without a paddle.
An appilcation can be installed again, unique data is priceless to the point where if it vanishes so does the entity that was supposed to manage it.

It's even more true with personal data. Stuff you created that can be found nowhere else and can never be re-created. I have never worried about the availability of applications. Applications are a dime a dozen.
My data and that in my organization is everything. It does depend on the kind of data of course, something like Google Maps can always generate fresh images. The planet will always be there [or else we've got other problems]
You can't ask your 10-year old to become 2 once more and make that face again.
You can't run that unique test on the batch of user data again when the source data gets lost/corrupted.

It's data. It truly is.

All hail the database administrator for they are the keepers of the world. And, because I'm not exclusive or naturally hate mongering: a big, wet kiss for our valiant application providers when they manage to create a masterful user experience that allows us to think about the job we're doing and not the way we're supposed to fumble with the widgets.

See: everybody gets to win.

#9 - Cirquelar 2008-03-21 16:02 - (Reply)

I had a colleague who referred to agencies having 'cemetaries of data' (i.e. paper printouts in file cabinets) that are essentially worthless unless they are digitized.

While I respect the arguement that some applications make data useful, sometimes more useful than in their original form (e.g. geographic information systems, GIS), but the application itself can not operate in a vacuum.

However, I'd argue that it is the 'form' that the data comes in that is important. How much government data is online, but essentially useless because it in an archaic format or simply not accessible due to interface. The 'form' of data is now becoming intimately tied to applications in some sense, so their is less pure data and pure app, but a symbiosis of the two.

#10 - Edward 2008-03-21 16:30 - (Reply)

A very entertaining fantasy.

Tell an artist that his brushes have no value. Tell him his paint is worthless. Tell him the lenses in his glasses which he uses to see the canvas are unimportant. And then somehow claim his artwork is a masterpiece. See what kind of artwork the artist can make when you take away his paint!

Applications turn data into information. Without applications, databases are just unfilled bit buckets. Without databases applications create no value. The two coexist and are codependant.

Your assertion is better stated as data oftentimes being a costly or irreplaceable asset.

Many applications are commodity -- they provide the same value for the same purpose. Oracle and SQL Server -- commodity software. Word processors, spreadsheets are also commodity.

In lieu of software patents, applications can be recreated. Data rarely has this luxury.

But to say that applications have no value is simply a false statement.

#11 - Doug Burns said:
2008-03-21 16:36 - (Reply)

But to say that applications have no value is simply a false statement.

I agree.

a) It wasn't my statement, I just thought it was thought-provoking.

b) It was a light-hearted piece meant to support a poor fed-up database bloke. I hadn't realised the strong feelings it would expose. I must confess my feelings on the subject aren't that strong one way or the other.

All it's really served is to illustrate the power of Reddit. Someone decided to take it on themselves to link to this (not my doing I might add) and before I know it, I have a comment storm!

Some interesting thoughts here, though.

#12 - Anonymous 2008-03-21 17:08 - (Reply)

But programs are data (c.f. McCarthy, and Abelson and Sussmann), terribly important data.

#13 - Mark 2008-03-21 18:38 - (Reply)

Even in emacs, data is king.

The text file equivalent of the DBA is the guy that writes the perl scripts to process the data.

#14 - Dominic Brooks said:
2008-03-21 18:42 - (Reply)

Applications are bloody important, right? Data is bloody important. Both evolve. There is a simbiotic relationship.

But, to bastardise an Edmund Blackadder witicism, an application without data is like a broken pencil - pointless.

#15 - Dominic Brooks said:
2008-03-21 18:43 - (Reply)

symbiotic even.

#16 - boingo 2008-03-21 19:46 - (Reply)

The argument of data vs. application seems to be a false dichotomy.

When you ring a bell, what makes the sound? Is it the metal forged in the shape of the bell that makes it, or is it the air inside the bell vibrating?

The sound comes from both. As value comes from both the application and the data. Without a meaningful way of viewing and understanding the data, the data is worthless. Without meaningful data to drive it, even the coolest application is useless.

#17 - Adam Helps said:
2008-03-21 19:47 - (Reply)

That's only true for data-centric applications. I've been doing computer programming for years (specifically 15 years), and have yet to end up working on any projects that even used a database.

Is the control system for a space-shuttle "all about the data?" No. It's about the expert system, encoded in the application, that flies the shuttle. Is a video game "all about the data?" No. It's about creating an interesting algorithmic framework that's fun to play with.

Is an operating system about the data? No, it's about creating a harness for other applications to run in. How about 3-d modeling software, my own current field? Arguably, the data there is the models that you're creating, but the algorithms for generating that data from raw thoughtstuff are absolutely critical to the process. The only data being processed exists solely in the mind of the designer, and its final form is intrinsically affected by the shape of the software.

In short, the opinion here is one of those biases that shows up when you confine yourself to a particular domain. If you've done nothing but pound nails your whole life, you're going to wonder why anyone needs tools other than a hammer. Don't get me wrong--hammers are terribly useful devices--but computing is more than data processing.

#18 - Doug Burns said:
2008-03-21 19:55 - (Reply)

On this specific point ...

Is a video game "all about the data?" No. It's about creating an interesting algorithmic framework that's fun to play with.

I used to write video games (albeit simple ones) so I know you're right. There are tons of apps which are not just about the data.

However, for bread-and-butter business apps, maybe the data is much more important? I can understand that without having confined myself to a particular domain.

Oh and the shelf-life of games (and shelf-life was one of the issues Tim was addressing) is short.

#19 - Dominic Brooks said:
2008-03-21 20:41 - (Reply)

Interesting reading the comments to your post on reddit and the others here.

Absolutely right to distinguish between data-centric applications and their use of the database and applications where processes involve little or no data.

I can totally appreciate that video games, industrial control systems, etc are about the application and its processes and not about any data that gets left behind.

When you've got a bunch of guys talking about Oracle, you can almost guarantee that we're talking about data-centric systems.

Interesting perspective though. I wonder how many of the problems that I have in data-centric systems are caused by people whose IT perspective and education come from non-data-centric applications?

#20 - Jasper 2008-03-21 21:38 - (Reply)

Well programs are "data" for us programmers.

The data (as stored in databases) is just for testing purposes. It is there to ensure that our programs are correct and are designed well - whether they do or don't have the clever little hacks we put in.

Data engenders itself by virtue of mere existence of humans (number of toilet rolls consumed, litres of petrol burned, etc.) but programs are brought into existence by a conscious effort of innovation, creativity & deadlines.

#21 - Jasper 2008-03-21 21:41 - (Reply)

To add to my comment above, life is also merely data - simply a collection of chemicals. But it takes a program (also known as DNA) to tell them how to combine to make a human or a dolphin.

#22 - Steven Talcott Smith said:
2008-03-21 22:13 - (Reply)

Look, code is data and data is code. It all depends on what you think of as "the machine." Dig the oneness. If you're after power, own the data. If you want beauty, breath code. If you want beauty and power, do both.

#23 - Mark Santaniello said:
2008-03-21 22:46 - (Reply)

Amen.

I work on compilers. Your code is my data.

#24 - damient 2008-03-22 08:25 - (Reply)

Whilst I agree that data can be important, the kinds of applications I have worked on involve a team of programmers building and refining an application over a number of years.

The application then becomes a repository of design decisions, requirements, innnovations, inventions, optimisations, etc etc. All largely undocumented, so that the application becomes a valuable resource in and of itself, in which is encoded the 'process'. Asked to recreate the application from scratch, most times, no-one could do it - its irreplaceable.

What has the most value? Something irreplaceable.

#25 - Doug Burns said:
2008-03-22 08:32 - (Reply)

What has the most value? Something irreplaceable.

Agreed. It's entirely dependent on the type of application.

#26 - Joe 2008-03-22 09:19 - (Reply)

Not a single text file I've edited with Emacs will be valuable to myself or anyone else ten years from now. Emacs has been a valuable program for decades already.

It would be easy to build a new text editor, but replacing Emacs would take quite a long time.

Working with Mathematica, in many cases at least, is not about analyzing data. It's about writing code. Not so different than writing Visual Basic code. If you want to stretch your definition so far that your own sourcecode counts as "data," then ok, but you're essentially saying nothing at that point.

Of course SecondLife uses data. But in the absence of the application, nobody would care about it.

And for an even more pure example, how about the real-time code running embedded devices, without saving any data whatsoever?

#27 - Joe 2008-03-22 09:29 - (Reply)

Oh, just having fun ;-)

We probably have a pretty similar outlook, actually. I get pretty fed up with application programmers who work with relational databases yet refuse to learn SQL well.

Or the various people who claim that relational databases are old legacy technologies, and want to build all the new stuff with some kind of primitive object persistence mechanism, with nowhere near the querying flexibility that SQL has.

#28 - Doug Burns said:
2008-03-22 09:40 - (Reply)

Oh, just having fun

Cool - I'm just not used to this much attention! ;-)

#29 - Niall Litchfield said:
2008-03-24 18:50 - (Reply)

The data behind Google Maps was there for how long? It was how valuable to a few cartographers? The GMaps application is what counts, not the data.

Really? The ability to quickly and easily display maps as a representation of geographic data is indeed neat - and mainstreaming that is way beyond time - but how is it used? For directions from place to place, to show where points of interest are, to highlight areas of interest. By doing what manipulating data. Imagine you had the maps but no co-ordinate references. Imagine those maps had no roads/towns etc displayed upon them. That api only works because of the data. It isn't a control system or similar it's just a neat way of visualising data. Same as excel - well cooler obviously unless you are an accountant - but still used for data visualisation.

Incidentally GIS systems were valuable to more than just a few cartogrophers - likely you drive on highways, use water systems and consume government services all of which have been provided in part thanks to GIS. You weren't a user, you likely were a consumer.

#30 - Cristian Cudizio said:
2008-03-26 09:02 - (Reply)

I remember to have read something similar on Thomas Kyte book where he states something like (i've not the book at hand now) "the software technologies came and go, but data remains" for example today java is very popular, but tomorrow? I would also refer to O'Reilly article (http://www.oreillynet.com/pub/a/oreilly/tim/news/2005/09/30/what-is-web-20.html?page=1) about web2 where states that data is important.

#31 - joel garry 2008-03-26 19:54 - (Reply)

Can't believe no one beat on the obvious here - a video game is uninteresting without being played, shuttle won't fly long without input and output.

Kids were complaining this weekend they couldn't save games on Playstation 1 (I was too cheap to buy modules for). DS has them spoiled. :-)

But I agree with the symbiosis crowd, data by itself is valueless - just ask NASA what's happening to all that Pioneer data. (Oh, guess I was out of date on that one: http://www.space.com/scienceastronomy/070327_scitues_pioneeranom.html )

Here's one of the Tom Kyte discussions, Christian: http://tkyte.blogspot.com/2007/10/it-all-about-data.html

#32 - Thormick said:
2008-03-27 13:28 - (Reply)

That'd be "applications come and go", or variations thereof. He's said that several places, here for example. Which is part of a discussion that's reminiscent of this.

#33 - Marco Gralike said:
2008-03-29 00:57 - (Reply)

I think most of it is in the how you define data. Data for me (DBA dude) is everything that is molded in a "container", so stuff like collections, arrays, clusters, tables. In cache, memory, on hard disks. Coherence, Oracle, Hypersonic, database block, bittorrent pieces flying around. Especially in this internet century it will be more and more fuzzy regarding its definition

#34 - PaulM said:
2008-03-29 11:24 - (Reply)

The thing totally missed from this debate between application and data is the logic, the model of the business or game or process that the application (or the database) is trying to replicate.

No one has defended that the database can in fact model the business or a subset of the business if done properly.

The whole reason for having a relational database is to have a relational model. As a DBA you get enlightened from tinkering and mucking around with parameters and settings, to adding indexes to finally realizing the design (the model) is core issue.

Before all the developers suggest this is going back to the waterfall method, rather than Agile, Iterative or Extreme or whatever.
When you are coding a application you have a goal in mind, even if you are building the app up in bits.
The goal is the model, the model is the goal. If the goal moves (updated), the model is updated.

Have Fun

Paul


Add Comment

Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.
BBCode format allowed