Click here to monitor SSC

Tony Davis

Simple-Talk Editor
News, views and good brews

Virtue in the Virtual

Published Friday, August 06, 2010 5:20 PM

Everyone knows the difference between a live SQL Server database file and a backup file. However, it seems that this distinction is being eroded, gradually, by the third-party tool providers. First, we saw tools that could compare live databases with backups, in order to report the differences in data or structure. Now we are seeing tools that allow a server to quickly mount, or restore a backup, and read the data from the backup file rather than create an entire database file.

Before such tools appeared, you were faced with having to do a full time-consuming and disk-consuming restore of the appropriate backup files, even if you only needed to perform a simple operation such as to check database consistency, or to query a table.

By contrast, the tools that use a file system filter driver take a database backup file and perform a "virtual restore" operation that mounts a "virtual database" directly to the SQL Server instance. The server has no knowledge of the fact that it is reading from a compressed database backup that is probably about 1/10th of the original size: this is done at file-system level. SQL Virtual Restore, for example, allows ACID-compliant modifications of this virtual database, just like a normal database. Suddenly a whole range of chores become easier, from verifying backups to setting up test cells. You can use it to speed up the offloading reporting from the live database, or refresh development servers with production data.

Why might the hard-pressed DBA pause, rather than immediately jump at an obvious means of saving time and resource? It is, perhaps, because DBAs are valued for their prudence. They are the custodians of the data of their organisations; deeply cautious over matters relating to data preservation and integrity. Whatever time and space-related problems may be attached to the traditional backup and restore, they are well-understood and predictable processes, which is the way DBAs like them. The idea of using a backup file as if it were a live database is a new and alien concept and so one must go beyond expounding the benefits, and explain how and why it works, and what the risks are, if any.

This is why we've set Brad McGehee up with a brand new test rig, and a remit to explain and test some of the new generation of database tools to their limits. The first 6500-word instalment is published in this week's Simple-Talk, covering SQL Virtual Restore.

We'd love to hear what you think. How much testing and research do DBAs need to do with a new tool before accepting that it is fit for purpose in their organisation?

Cheers,

Tony.

Comments

 

timothyawiseman@gmail.com said:

The answer to the final question is the infamous, "It Depends."

For a server in full scale production involving data that is vital to the companies operation, the testing must be extensive.  Before I would consider permitting something, it must be thoroughly tested both internally to the company and by the community at large.  For something critical, I would not touch a new piece of software that has not been out for a while and been reviewed by other people in the community I trust.

On the other hand, for a server that is less critical, perhaps an analysis server that is rarely used for time-sensitive projects, I am much more willing to take some limited risks.  I will still do internal testing and research the products before using them, but I am willing to take initial releases that have not been thoroughly analyzed by the community if they come from a known source and seem useful.

Development servers often have technologies we are evaluating for the future, so I am willing to try anything that has a stable release, at least for a limited time until we make up our mind.

Then there is the test machine I have at home.  There I often stay on the bleeding edge playing with evaluation versions just to see what they are like and running early betas so I know what will be out soon.

In short, if the data is important and must be consistent, then prudence is the word and anything new should be viewed warily, but sometimes there is room for more flexibility and experimentation.
August 6, 2010 6:50 PM
 

BuggyFunBunny said:

The mainframe DB2 and COBOL world where I last worked (and others before it), it was quite common to treat "tables" as context independent "files".  Truth be told, if you've no table relational constraints, and that's more common Out There than most will admit, then swapping one version of a "file" for another is No Big Deal.  Of course, what you've actually got then is not a relational database, just a (very expensive) SQL parser.  

And, of course, the application code has to know how to ensure consistency among those "files".  

A very long time ago I worked with a 4GL/database called Progress, which as does Jet, stored all data in a single (logical; it could partition to multiple physical) file; tables were logical constructs defined by the developer, but managed by the engine as bytes in the file.  The notion of the Progress developers was that a Database was an integrated whole, not just a collection of "files".  This was quite enlightened, given that the 4GL was far more COBOL-like (record/row looping) than SQL-like (set semantics).  And it certainly discouraged such playthings.  As relates to the present instance; if you *can* swap tables/files at will without having to jump through multiple hoops, odds are you've not a relational datastore in the first place.
August 6, 2010 8:03 PM
 

Mal Daughtree said:

Interesting.... I've actually evaluated a wonderful Virtual restore from Red Gate's direct competitor in this space and I admit that if I was today given a choise between the two I would have to say that the ability to 'Virtually' restore a database from a backup without having to do a 'real' restore is, without a doubt, the most amazing thing in the DBA sphere in the last decade (because it was available last year.) and I would not hesitate to use the competitors product.   There is of course a level of testing the would be employed but as another way to provide data services to a customer quickly this would add considerable capability to any DBA that has had to beg, borrow and steal disk space in order to carry out a data comparision or recover a lost Stored Procedure or even an entire table or Database.  Careful we are ... Prudent we need to be ... and change we must.
August 9, 2010 6:46 AM
 

BuggyFunBunny said:

@Mal:
and change we must.

There is the famous adage, whose author immediately escapes me:  "All progress requires change, but all change is not progress".  The lemming-esque acceptance of a collection of xml documents as database is so reactionary as to boggle the mind.  But the ignorant are easily swayed.  Hans Christian Andersen wrote about that; it's still worth remembering.
August 9, 2010 1:20 PM
 

fhanlon said:

The business in which I work - Healthcare - like many other business can be severely impacted when technology fails. The difference in Healthcare is that people’s lives may be put in danger with poorly tested changes.  For example imagine a system that alters medical staff when a critical result in a patient monitoring device reaches a certain threshold.  Imagine making a change to the messaging system that causes delays or even non-delivery of messages because some parameters were set incorrectly.   On the other hand some changes introduce much needed or desired functions so the users don’t want to stifle change but they are very concerned that change be well tested to ensure it does what it is supposed to do and does not negatively impact current functionality.

We have several layers of test systems from a development system where testers can introduce changes and verify the change itself works without regard to any impact on any other part of the system.  Assuming things work well the change can be migrated to a test environment where the change will be looked at by systems staff with respect to other changes so the impact can be understood as to what the change might do overall.  Assuming we want to precede the change is then migrated to a User Acceptance test area.  This system contains only changes that the user is ready to accept into production.  The user will use this to verify that everything works as expected and the change doe not break anything else.  When the user is ready to move things into production we do this in 2 steps.  Step 1 involves migrating the change to a staging environment, which is physically identical to production with exactly the same software.  This provides us with several benefits.  First we know the exact process to migrate and gives us one extra migration ‘practice’.  Second we can ensure that hardware or software differences between test and production don’t cause any problems.  Usually, we use SQL Enterprise edition only in production (and staging), SQL developer is used in test or developer systems.  Also the test or developer environments may be virtual servers and production is generally a physical box.  We try to keep the OS identical throughout all environments but production may have more RAM or may be attached to a different SAN array. Test may not even be attached to SAN.   The staging environment also provides us with an identical test bed in the case where something works in test but not in production or even vice versa where something works in production but fails in test.  

Our test process is long but in case of emergency changes we can move fixes through quickly if required.  On the other hand new releases sometimes take a while to get though but we know that they have been well tested and documented by a number of different people each with a different focus.  In terms of DBA tools I may not need to though the same process but that depends on where the tool is to be installed.   For example, a compare tool is really installed on my workstation so I need to verify the tool works the way I want it to and I get the results I expect.  I am not installing this kind of tool on the production server so I don’t need to go through the development, testing , UAT and staging cycles.  On the other hand a backup tool since it is installed on the actual servers will need to go through this process.  The difference is that a tool that is used by the DBA (like database compare) only needs to be verified by the DBAs.  A backup tool is used by SQL Server so we need to verify that the environment we have is working properly with this tool.  Does the tool backup the databases correctly in test with no greater impact that the native tool?  Does this tool provide me with benefits that the native tool does not have.  If it works well I may move it to the next environment and continue to verify that all is well.  Eventually I may move it into the production environment.  Certainly I wouldn’t want to use a different backup tool in test and production so I am testing the tool keeping in mind that either I use this tool or continue to perform this function using native tools.  If there is more that one tool that I want to test I would first do some research as to the comparative advantages and disadvantages of various tools and then select 1 or 2 based on this assessment.  I may want to business to pay for this tool so I would need to convince them that the money spent on this tool allows the DBAs to provide better support in some way.
August 9, 2010 3:57 PM
 

Dave M said:

"a system that alters medical staff..."  could be a good thing if the alterations improve the productivity and reliability of the altered medical practitioners.
August 9, 2010 11:25 PM
 

fhanlon said:

"imagine a system that alters medical staff" -  ok that is funny.  I meant to write  "imagine a system that alerts medical staff  .."

I'm sure medical staff would rather be alerted about something than altered.  Although, perhaps I shouldn't assume this.
August 23, 2010 3:41 PM
You need to sign in to comment on this blog
<August 2010>
SuMoTuWeThFrSa
25262728293031
1234567
891011121314
15161718192021
22232425262728
2930311234
How to Kill a Company in One Step or Save it in Three
 The majority of companies that suffer a major data loss subsequently go out of business. David Wesley... Read more...

Migrating from OCS 2007 R2 to Lync: Part 4
 Having migrated the rest of our users and legacy resources across, and start getting ready to... Read more...

Automated Script-generation with Powershell and SMO
 In the first of a series of articles on automating the process of building, modifying and copying SQL... Read more...

Seth Godin: Big in the IT Business
 Seth Godin has transformed our understanding of marketing in IT. He invented the concept of 'permission... Read more...

Using SQL Test Database Unit Testing with TeamCity Continuous Integration
 With database applications, the process of test and integration can be frustratingly slow because so... Read more...