Aug 6: Full Volume Database Refresh - Part 2
It only took 6.5 hours on Sunday (in time to watch Celtic's first match of the season) and apart from a couple of minor panics along the way, it went well. Anyway, back to the overall process, with additional insight based on the first painful test and yesterday's almost painless one ...
Based on where I left Part 1, let's assume that the Production database has been copied, recovered and started as a new test instance and database.
Actually, let's not assume too much here, because the copy process can be an absolute nightmare, depending on a few factors, including one pointed out in Dominic Delmolino's comment to the last post.
Try to make the filesystems the same -- makes things a lot easier. And really, why would they need to be different?
That comment made me chuckle to myself, because I can hear the voice of painful experience echoing through the internet
Because they are often completely different! Don't get me wrong, you can shuffle files around to new locations so that they all fit, but it's a monumental pain in the backside. It's almost a rite of passage for a DBA to have tried to solve the database file equivalent of one of these.
There are two main strategies if you have a mish-mash of incompatible filesystems.
- Plan meticulously until you are sure that you know where every file from the source database fits on the target filesystems.
- Have a stab at it and set the first restore (or copy) running. When it fails, make a few edits and restart the process. Repeat.
One of the main reasons why the most recent practice went so smoothly was that I'd put together all of the SET NEWNAME commands during the last run and there hadn't been any files added to Production since then. So I already have my source-to-target file mapping in place.
Going forward, I need to bear the following in mind.
1) The Production database files will grow.
2) Production files might appear or disappear.
In fact one recent client insisted that all space requests for Production and the Volume Test environment were submitted through the change management process so that we could ensure that the environments were identical, but I haven't often worked in such a well-managed environment.
Why might the file systems be different? All sorts of reasons but usually because small test environments are rapidly converted into full volume. When I say rapid, I mean - 'we need it yesterday' - so compromises are made. Not good, but part of the world I operate in.
Let's just say that, if you're going to perform a regular refresh, it's worth spending time setting up a new set of filesystems on the target server that match the source server. As Dominic said, it 'makes things a lot easier'.
Oh, wow, another part of this blog gone and I haven't even got to the tricky stuff yet! Next time
#1 - koert 2007-08-07 09:49 - (Reply)
I once had to migrate a 12 node OPS with about 60000 raw devices to another machine, single instance / JFS. Needless to say I scripted it, but I was seriously pinching my cheeks when I issued that 'alter database open' command ![]()
Oh - one part of the script appended 512 empty bytes to each datafile. I had to do this because there was a 512byte device header I had to skip, so my datafiles weren't a multiple of the blocksize anymore (otherwise the 'create controlfile' started moaning and bitching
)
That was a refresh from hell :-p
#1.1 - Doug Burns said:
2007-08-07 18:31 - (Reply)
That was a refresh from hell :-p
Oh, yeah! That's hardcore!
This is nowhere *near* as bad ...
#2 - Niall Litchfield said:
2007-08-07 09:57 - (Reply)
1) The Production database files will grow.
2) Production files might appear or disappear
I've lost track of the number of times I've seen this forgotten. A neat variation on this that we are going to run into. Our test/dev etc boxes have 8 copies of production on them. So every time you add a 4gb datafile to production you store up a requirement for 32gb more on dev. As the refresh cycles vary from monthly to 6 monthly or even longer keeping track of this is a royal pain - though to be honest it's probably what excel was invented for.
oh yeah - I don't think I've ever seen production tablespaces dropped (except during a major upgrade perhaps), so you rarely lose production datafiles...
#2.1 - Doug Burns said:
2007-08-07 18:35 - (Reply)
Niall,
oh yeah - I don't think I've ever seen production tablespaces dropped (except during a major upgrade perhaps), so you rarely lose production datafiles...
In general, yes, they never want *less*, do they? ![]()
However, in this particular case, because we're creating a new tablespace for an IOT version of an existing table and then dropping the original later, it does happen ... sometimes! (Oh, and it's for a completely unrelated change)
Essentially, file structures will change, not just sizes, although you're right to point out how rare it is for files to be deleted.
#3.1 - Doug Burns said:
2007-08-07 18:54 - (Reply)
LOL ...
Hey, and when that doesn't work, just try Microsoft Project. Have you *seen* how that can improve productivity?
Not.
#4 - Manish said:
2007-08-08 06:25 - (Reply)
@Doug,
I have been running into this "jigsaw" problem for years and it was especially painful since we did almost a refresh/clone a day for databases ranging from 100G to 1T using either compressed backups or split mirror images. This was very painful and error prone so I scripted it in perl. I wish I could share the code but it belongs to the company I work for but I can share the idea and may be someone can script and share it.
Following process has been used in over 300 clones so far.
The process is pretty generic with two main parts: 1) logic to handle the space requirements and 2) to output the right kind of restore commands. One can handle almost all kinds of restore situations (hot backup, hot backup restored from a remote server using unix pipes, sync-split images, SET NEWNAME script for RMAN and whatever else.
-- continued in next entry --
#5 - Manish said:
2007-08-08 06:27 - (Reply)
1. Generate a colon delimited text file with list of files names, sizes and locations of the datafiles from source.
2. Generate a colon delimited text file with list of mount points available for restore along with useable size available in each of them.
3. Iterate through datafiles deducting the file size from the size available in first mount point, if remainder is >0 then the file would fit so output an approriate restore line (could be a gunzip/scp/set new name).
-- continued --
#6 - Manish said:
2007-08-08 06:28 - (Reply)
4. Once you get a remainder that is less than 0, pick up next mount point and deal with the rest of the files.
5. If you consume all available space with files reamining then you need more space (duh) else you can be confident that all files will fit.
6. Additionally, logic can be built in the same script to not output all datafiles in one big restore script but create multiple files (each restoring approximately same amount/size of datafiles, not the same number of files). This will ensure best possible parallelism of the restore process and restore in least possible amount of time.
A shell script automates step #1, and #2. Step #3 thru #6 are handled by a perl script.
Caveat:
No distinction is made between data and index files. You may divide the whole exercise in two parts, if one is so inclined, dealing with data and index files separately.
HTH,
--
Manish
PS: Please excuse any grammatical mistakes; English is not my mother tongue.
#7 - Doug Burns said:
2007-08-08 06:38 - (Reply)
Manish,
PS: Please excuse any grammatical mistakes; English is not my mother tongue.
I didn't notice any ![]()
If my memory serves me correctly, one of the better DBA teams I worked in had a similar tool. Like you, there would be many refreshes - several a day - so it was well worth the development effort.
Thanks very much for your contribution.
Cheers,
Doug



After the solving the jigsaw puzzle and waiting for tape drives to do their thing, we've finally got a running test instance accessing a full copy of production. (Well, there was some recovery, faffing around with database names, control files, tempfile
Tracked: Aug 07, 20:35