Full Volume Database Refresh - Part 2

Doug's Oracle Blog

  • Home
  • Papers
  • Books
  • C.V.
  • Fun
  • Oracle Blog
  • Personal Blog

Aug 6: Full Volume Database Refresh - Part 2

First the good news. "The Refresh from Hell" (because it was) has become "The Slightly Tricky Refresh".

It only took 6.5 hours on Sunday (in time to watch Celtic's first match of the season) and apart from a couple of minor panics along the way, it went well. Anyway, back to the overall process, with additional insight based on the first painful test and yesterday's almost painless one ...

Based on where I left Part 1, let's assume that the Production database has been copied, recovered and started as a new test instance and database.

Actually, let's not assume too much here, because the copy process can be an absolute nightmare, depending on a few factors, including one pointed out in Dominic Delmolino's comment to the last post.

Try to make the filesystems the same -- makes things a lot easier. And really, why would they need to be different?

That comment made me chuckle to myself, because I can hear the voice of painful experience echoing through the internet ;-) As Dominic says, there's no reason not to have the source and target databases on identical filesystems if they're on different servers. At the very least, give me the same number of filesystems that are sized identically. That way, any edits you need to file locations in a control file trace output or RMAN "SET NEWNAME" commands are easier. But look at the start of his comment. Why say "try" if the filesystems are always the same? It's such a no-brainer!

Because they are often completely different! Don't get me wrong, you can shuffle files around to new locations so that they all fit, but it's a monumental pain in the backside. It's almost a rite of passage for a DBA to have tried to solve the database file equivalent of one of these.

There are two main strategies if you have a mish-mash of incompatible filesystems.

  1. Plan meticulously until you are sure that you know where every file from the source database fits on the target filesystems.
  2. Have a stab at it and set the first restore (or copy) running. When it fails, make a few edits and restart the process. Repeat.
Generally I find that option 2 works best for me, as long as the restore operation can skip over the already-restored files very quickly.

One of the main reasons why the most recent practice went so smoothly was that I'd put together all of the SET NEWNAME commands during the last run and there hadn't been any files added to Production since then. So I already have my source-to-target file mapping in place.

Going forward, I need to bear the following in mind.

1) The Production database files will grow.
2) Production files might appear or disappear.

In fact one recent client insisted that all space requests for Production and the Volume Test environment were submitted through the change management process so that we could ensure that the environments were identical, but I haven't often worked in such a well-managed environment.

Why might the file systems be different? All sorts of reasons but usually because small test environments are rapidly converted into full volume. When I say rapid, I mean - 'we need it yesterday' - so compromises are made. Not good, but part of the world I operate in.

Let's just say that, if you're going to perform a regular refresh, it's worth spending time setting up a new set of filesystems on the target server that match the source server. As Dominic said, it 'makes things a lot easier'.

Oh, wow, another part of this blog gone and I haven't even got to the tricky stuff yet! Next time ;-)
Posted by Doug Burns Comments: (10) Trackback: (1)
Defined tags for this entry: Database Refresh
Related entries by tags:
Full Volume Database Refresh - Part 4
Full Volume Database Refresh - Part 3
Full Volume Database Refresh - Part 1

Trackbacks
Trackback specific URI for this entry

Full Volume Database Refresh - Part 3
After the solving the jigsaw puzzle and waiting for tape drives to do their thing, we've finally got a running test instance accessing a full copy of production. (Well, there was some recovery, faffing around with database names, control files, tempfile
Weblog: Doug's Oracle Blog
Tracked: Aug 07, 20:35

Comments
Display comments as (Linear | Threaded)

#1 - koert 2007-08-07 09:49 - (Reply)

I once had to migrate a 12 node OPS with about 60000 raw devices to another machine, single instance / JFS. Needless to say I scripted it, but I was seriously pinching my cheeks when I issued that 'alter database open' command ;-)
Oh - one part of the script appended 512 empty bytes to each datafile. I had to do this because there was a 512byte device header I had to skip, so my datafiles weren't a multiple of the blocksize anymore (otherwise the 'create controlfile' started moaning and bitching ;-) )
That was a refresh from hell :-p

#1.1 - Doug Burns said:
2007-08-07 18:31 - (Reply)

That was a refresh from hell :-p

Oh, yeah! That's hardcore!

This is nowhere *near* as bad ...

#2 - Niall Litchfield said:
2007-08-07 09:57 - (Reply)

1) The Production database files will grow.
2) Production files might appear or disappear

I've lost track of the number of times I've seen this forgotten. A neat variation on this that we are going to run into. Our test/dev etc boxes have 8 copies of production on them. So every time you add a 4gb datafile to production you store up a requirement for 32gb more on dev. As the refresh cycles vary from monthly to 6 monthly or even longer keeping track of this is a royal pain - though to be honest it's probably what excel was invented for.

oh yeah - I don't think I've ever seen production tablespaces dropped (except during a major upgrade perhaps), so you rarely lose production datafiles...

#2.1 - Doug Burns said:
2007-08-07 18:35 - (Reply)

Niall,

oh yeah - I don't think I've ever seen production tablespaces dropped (except during a major upgrade perhaps), so you rarely lose production datafiles...

In general, yes, they never want *less*, do they? ;-)

However, in this particular case, because we're creating a new tablespace for an IOT version of an existing table and then dropping the original later, it does happen ... sometimes! (Oh, and it's for a completely unrelated change)

Essentially, file structures will change, not just sizes, although you're right to point out how rare it is for files to be deleted.

#3 - joel garry 2007-08-07 14:50 - (Reply)

http://www.dilbert.com/comics/dilbert/archive/images/dilbert2045828070807.gif

Serendipity at its finest.

#3.1 - Doug Burns said:
2007-08-07 18:54 - (Reply)

LOL ...

Hey, and when that doesn't work, just try Microsoft Project. Have you *seen* how that can improve productivity?

Not.

#4 - Manish said:
2007-08-08 06:25 - (Reply)

@Doug,
I have been running into this "jigsaw" problem for years and it was especially painful since we did almost a refresh/clone a day for databases ranging from 100G to 1T using either compressed backups or split mirror images. This was very painful and error prone so I scripted it in perl. I wish I could share the code but it belongs to the company I work for but I can share the idea and may be someone can script and share it.

Following process has been used in over 300 clones so far.

The process is pretty generic with two main parts: 1) logic to handle the space requirements and 2) to output the right kind of restore commands. One can handle almost all kinds of restore situations (hot backup, hot backup restored from a remote server using unix pipes, sync-split images, SET NEWNAME script for RMAN and whatever else.

-- continued in next entry --

#5 - Manish said:
2007-08-08 06:27 - (Reply)

1. Generate a colon delimited text file with list of files names, sizes and locations of the datafiles from source.
2. Generate a colon delimited text file with list of mount points available for restore along with useable size available in each of them.
3. Iterate through datafiles deducting the file size from the size available in first mount point, if remainder is >0 then the file would fit so output an approriate restore line (could be a gunzip/scp/set new name).

-- continued --

#6 - Manish said:
2007-08-08 06:28 - (Reply)

4. Once you get a remainder that is less than 0, pick up next mount point and deal with the rest of the files.
5. If you consume all available space with files reamining then you need more space (duh) else you can be confident that all files will fit.
6. Additionally, logic can be built in the same script to not output all datafiles in one big restore script but create multiple files (each restoring approximately same amount/size of datafiles, not the same number of files). This will ensure best possible parallelism of the restore process and restore in least possible amount of time.

A shell script automates step #1, and #2. Step #3 thru #6 are handled by a perl script.

Caveat:
No distinction is made between data and index files. You may divide the whole exercise in two parts, if one is so inclined, dealing with data and index files separately.

HTH,
--
Manish
PS: Please excuse any grammatical mistakes; English is not my mother tongue.

#7 - Doug Burns said:
2007-08-08 06:38 - (Reply)

Manish,

PS: Please excuse any grammatical mistakes; English is not my mother tongue.

I didn't notice any ;-)

If my memory serves me correctly, one of the better DBA teams I worked in had a similar tool. Like you, there would be many refreshes - several a day - so it was well worth the development effort.

Thanks very much for your contribution.

Cheers,

Doug


Add Comment

Standard emoticons like :-) and ;-) are converted to images.
E-Mail addresses will not be displayed and will only be used for E-Mail notifications.
BBCode format allowed
 
 

Statistics on Partitioned Tables

Contents

Part 1 - Default options - GLOBAL AND PARTITION
Part 2 - Estimated Global Stats
Part 3 - Stats Aggregation Problems I
Part 4 - Stats Aggregation Problems II
Part 5 - Minimal Stats Aggregation
Part 6a - COPY_TABLE_STATS - Intro
Part 6b - COPY_TABLE_STATS - Mistakes
Part 6c - COPY_TABLE_STATS - Bugs and Patches
Part 6d - COPY_TABLE_STATS - A Light-bulb Moment
Part 6e - COPY_TABLE_STATS - Bug 10268597

Comments

Doug Burns about 10053 Trace Files - Different Plan in Different Environments
Tue, 02.04.2013 08:57
You're welcome. Now I just nee d to pull my finger out and ac tually come up [...]
Howard Rogers about 10053 Trace Files - Different Plan in Different Environments
Mon, 01.04.2013 23:08
Makes a big difference, so tha nks for that! With two brow ser windows, o [...]
stelioscharalambides.com about 10053 Trace Files
Sat, 30.03.2013 16:28

Upcoming Presentations

Bookmark

Open All | Close All

Syndicate This Blog

  • XML RSS 2.0 feed
  • ATOM/XML ATOM 1.0 feed
  • XML RSS 2.0 Comments
  • Feedburner Feed

Powered by

Serendipity PHP Weblog

Show tagged entries

xml 11g
xml ACE
xml adaptive thresholds
xml ASH
xml Audit Vault
xml AWR
xml Blogging
xml conferences
xml Cuddly Toys
xml Database Refresh
xml DBMS_STATS
xml Direct Path Reads
xml Fun
xml grid control
xml hotsos 2010
xml listener
xml Locking
xml oow
xml oow2009
xml optimiser
xml OTN
xml Parallel
xml Partitions
xml Patching
xml swingbench
xml The Reality Gap
xml time matters
xml ukoug
xml ukoug2009
xml Unix/Shell
xml Useful Links

Disclaimer

For the avoidance of any doubt, all views expressed here are my own and not those of past or current employers, clients, friends, Oracle Corporation, my Mum or, indeed, Flatcat. If you want to sue someone, I suggest you pick on Tigger, but I hope you have a good lawyer. Frankly, I doubt any of the former agree with my views or would want to be associated with them in any way.

Design by Andreas Viklund | Conversion to s9y by Carl