When Database Mirroring was introduced in SQL Server 2005, it seemed reasonable to assume that log shipping would gradually go out of fashion. Mirroring is a way of introducing high-availability to SQL Server by allowing the secondary server to become the main server instantly, in a way that is invisible to the applications using the database. This, in theory, is the dream solution for the .NET application developer: the application needs to do no work at all to take advantage of this resilience against hardware failure.

Log Shipping has been around for a lot longer, first as a sort of "home-brew" solution that the intrepid DBA could make work with version 6.5 and 7.0, before an official version was built into the enterprise edition of SQL Server 2000. It is a method, based on SQL Server Agent jobs, by which the transaction log backups from a primary server are applied to secondary servers. In this way, one can keep one or more spare 'warm standby' servers in a state of readiness to take over in the event of failure of the primary. It works well, and requires just a few mouse-clicks to set up, but it requires some nimble footwork from the DBA to switch the standby into becoming the primary server, and generally entails 15-30 minutes of downtime. In addition, application client redirection requires some manual changes.

Curiously, despite its many stated advantages and heavy support from Microsoft, Mirroring hasn't replaced Log shipping, as predicted. Aside from the obvious fact that many shops still have SQL 2000 machines and so have no choice but to use log shipping, there are a number of other subtle and not-so-subtle reasons for its continued prevalence.

Many HA setups require more than one standby server, which mirroring cannot provide. Other DBAs require that delay time that is intrinsic in Log Shipping to protect against logical errors that, with mirroring, would have instantly been carried across to the standby. In a sense, Log Shipping also provides "free standbys", since it makes use of backups would need to be made anyway and, to the delight of many DBAs, checks the integrity of the backup for free.

This final point hints at perhaps the biggest reason why log shipping still prevails. Many DBAs regard the High Availability aspect of log shipping as a "happy by-product" of their need to offload reporting functions to a secondary server. In a mirrored setup, the target database is never online, or accessible to the end user, until failure occurs on the source. The only way to get around this is to set up database snapshots on the target, which of course incurs the expense of Enterprise Edition. With log shipping, all you need is Backups, and a few cheap standby servers running SQL Server Express, and you have an effective reporting solution, with "poor man's High Availability" thrown in!

However, although rumours of log shipping's demise have been exaggerated, it may still be only a stay of execution. Microsoft may finally relent on the Enterprise requirement for snapshots. With the advent of server virtualization technologies, it will surely become easier to maintain 'standby' virtual servers which can be deployed even more quickly in the event of either physical or logical failure.

DBAs have stuck with log shipping for good reasons that were un-guessed at by the pundits, but does it have a long-term future as a high-availability strategy?

As always, we'd love to hear your opinion. Add your comments below (you'll need to be signed in), and the best contribution will receive a $50 Amazon voucher.

Cheers,

Tony.

Microsoft's recent policy of Glasnost has meant that we now enjoy direct, technical communications about their products from the people responsible for building them. In the bad old days, when all communication was filtered through the obfuscating glass of the marketing department, one would read through the gobbledygook of the marketing press releases, glean what little sense one could, shrug, and move on. Nowadays, one wades through hundred of blogs, tens of videos and countless twitters, before reaching roughly the same conclusion.

This thought hit me as I tried to make sense of the information regarding the CTP of SQL Server 2008 release 2 (previously codenamed 'Kilimanjaro'). It is due for release in July, to coincide with the Office 2010 CTP, and promises to be the most substantial "point release" ever. Is there a central, consistent theme to the release? Or is it easier to understand R2 as representing a wonderful opportunity to slip all those things into SQL Server 2008 that were 'time-boxed out' by the original deadlines? In amongst the technical buzz from Microsoft, are we just hearing the voice of SQL Server catching up with changes to Windows Server, Visual Studio, and SharePoint/Office 2010?

I started by trying to seek out the reality behind the concept of 'self-service business intelligence', provided via a new Excel add-in (Gemini). These new BI capabilities have been whetting appetites since October 2008, when Donald Farmer first demonstrated how the in-memory, column-oriented processing model allowed users to slice, dice, pivot and sort two million rows of data, right for within Excel. However, relatively little new detail has emerged since about the underlying changes to Analysis Services, or about Gemini's new "DAX expressions", for multidimensional calculations, and what they mean, if anything, for the future of MDX.

What strategies underpinned the move to make IT administration easier? Dashboard viewpoints are a new feature, allowing administrator to manage policies on their servers and "identify consolidation opportunities". However, the central pin in the "manageability" drive seems to be Master Data Services (MDS), a tool by which to centrally manage all critical business data in financial and ERP applications. It's potentially an interesting tool for some people, but seems to have been rather haphazardly shovelled into SQL Server, having originally been announced as part of SharePoint 2010. We'll also have 'Low Latency Complex Event Processing' for capturing and analysing usage patterns in the database and making it easier to detect, for example, fraudulent activity. Again, however, details are sketchy.

Elsewhere, smooth integration with SharePoint 2010 is promised often, as well as better integration with Visual Studio. There will also be improvements to the way that virtualization is done, and an increase in the number of logical processors supported from 64 up to 256, exploiting improvement in the new 64-bit Windows Server 2008 r2. There will be self-service reporting (with Virtual Earth integration), a few minor additions to the TSQL Language, and we'll also get compression of Unicode strings.

No. We're not going to find a unifying theme behind the new release. It is a Muesli release, a disparate mix, with a number of tasty nuts and raisins thrown in. Perhaps Microsoft’s new way of announcing releases acknowledges the huge complexities of the server products. No sound bite will do justice to what will be delivered, but like Muesli, there will be plenty to chew on when SQL Server 2008 R2 arrives.

Cheers,

Tony.

Recently, a local builder handed me an estimate for some construction work on my home. Once my eyes had stopped watering, he calmly explained to me that I wasn't paying him for what he could do, so much as what he knew. He'd been in the trade for 30 years. "That" he declared, pointing at the estimate "is exactly how much it will cost, and exactly how long it will take". Three weeks later the work was complete, exactly as he said it would be, and I gratefully handed over a vast sum of money.

Why do so few IT projects proceed in this way? Why are most developers so poor at estimating projects? The first trick, when estimating, is to break down the project into all of its component parts. Accurately identifying all of these steps is difficult, and is a skill that comes with experience. The second task is to assign time estimates to each step. Again, the more times a developer has been through the mill, the higher the likelihood that a given task is one has been seen before, and the more accurate the estimations become.

Even then, given that software development always tends to be on the cutting edge of technology, there will always be a few steps that even the experienced developer will not have encountered before. It is at this point that a developer is likely, if pressed, to give his "best guess" as to how long the "unknown" step will take. It has been wisely observed that the inaccuracy of these "guesstimates" increases exponentially with time. If the estimate is a day, you can expect it to be finished in around a day. If the estimate is a week, it will probably be finished in 1-2 weeks. If the estimate is a month, then the developer really has no idea how long it will take.

Any step that is estimated at over 1 week is a danger signal. Any step that none of the team has encountered before is an "unknown" and therefore a risk. Instead of "guesstimating" such steps, the developer should apply solid three-valued logic. If a step is "unknown" then a NULL should be inserted into the time column for that step. Of course, thanks to NULL propagation, that means the entire project timeline is NULL. The developer must then insist on having time to prototype these NULL steps, so that estimates can be made based on hard evidence, rather than hunches and guesswork.

Inevitably, there are many "agile" techniques, such as planning poker and velocity measurements, which you can use to produce more accurate project estimates. Most of them are well-documented. However, the biggest problem still remains that the IT industry, unlike the building trade, is notoriously bad at retaining experience. In the main, companies hire and pay developers based on what they can do (write code) rather than on the knowledge that comes with experience, such as effective project estimation and management. If these skills were properly valued and rewarded, the industry would do a better job at retaining experience in the role, and the development team's estimation skills, like fine ales, would refine and mature with age.

Cheers,

Tony.

When the market is slack, nothing succeeds better at tightening it up than promoting serial group-panic within the community. As an example of this, a wave of multi-core panic spread across the Internet about 18 months ago. IT organizations, it was said, urgently had to improve application performance by an order of magnitude in order to cope with rising demand. We wouldn't be able to meet that need because we were at the "end of the road" with regard to step changes in processor power and clock speed. Multi-core technology was the only sure route to improving the speed of applications but, unfortunately, our current "serial" programming techniques, and the limited multithreading capabilities of our programming languages and programmers, left us ill-equipped to exploit it. Multi-core mania gripped the industry.

However, the fever was surprisingly short-lived. Intel's "largest open-source effort ever" to provide a standard tool for writing multi-threaded code, caused little more than a ripple of interest. Various books, rushed out while the temperature soared, advocated the urgent need for new "multi-core-friendly" programming models, involving such things as "software pipelines". Interesting as they undoubtedly are, they sit stolidly on bookshelves, unread.

The truth is that it's simply not a big issue for the majority of people. Writing truly "concurrent" applications in languages such as C# is difficult, as you get very little help from the language. It means getting involved with low-level concurrency primitives, such as lock statements and so on.

Many programmers lack the skills to do this, but more pertinently lack the need. Increasingly, programmers work in a web environment. As long as these web applications are deployed to a load-balanced web farm, then page requests can be handled in parallel so all available cores will be used efficiently without the need for the programmer to be concerned with fine-grained parallelism.

Furthermore, the SQL Server engine behind these web applications is intrinsically "parallel", and can handle and use effectively about as many cores as you care to throw at it. SQL itself is a declarative rather than procedural language, so it is fundamentally concurrent.

A minority of programmers, for example games programmers or those who deal with "embarrassingly parallel" desktop applications such as Photoshop, do need to start working with the current tools and 'low-level' coding techniques that will allow them to exploit multi-core technology. Although currently perceived to be more of "academic" interest, concurrent languages such as Erlang, and concurrency techniques such as "software transactional memory", may yet prove to be significant.

For most programmers and for most web applications, however, the multi-core furore is a storm in a teacup; it's just not relevant. The web and database platforms already cope with concurrency requirements. We are already doing it.

As always, we'd all love to hear what you think. The best contribution to the debate, added as a comment to this blog, will receive a $50 Amazon voucher.

Cheers,

Tony.

Programmers often have an old-fashioned view of their trade. They enter the profession imagining that they will spend most of their time puzzling over complex algorithms, developing dazzlingly creative and compelling applications, writing operating systems in their spare time and secretly working on their own computer language. In fact, of course, most programmers' lives consist mainly of moving sticky pieces of paper from one side of a whiteboard to the other, and making sure the focus moves to the correct control when you press "tab".

It is instructive to examine the desks and book cases of your fellow programmers, especially the slightly older ones. In some cases they will reflect healthy programming aspirations, if not day-to-day reality, holding such lofty titles as "Purely Functional Data Structures" and "Concurrent Programming in Windows", as opposed to "Getting Agile with Post-it Notes".

On many other bookshelves, however, one can find ample evidence of programmers trying to escape programming as fast as they possibly can, by becoming managers. Every time I see a copy of "The Results-Driven Manager" on a programmer's desk, it sends a light shiver down my spine.

However, this obsession with moving into management is perfectly understandable. As an industry we dislike employing older programmers, and provide them with little career progression. You can become a senior developer after five years which means that, if you are a successful graduate, you will get there by the time you are about 26! After a few more years, unless you dye your hair and lie about your age, you begin to feel more or less obliged to start climbing the greasy pole into management. The net result is that the profession as a whole retains little in the way of experience. It is little wonder then that, in Europe for example, the annual cost of IT project failure is a breathtaking $200 billion.

As an industry we have to understand that becoming a manager is not a career progression but a complete change of job. Being a good programmer does not make one a good manager. Often, the reverse is true. Why do we value management skills more highly, anyway, when a good programmer will make just as much of a contribution to an organization as a good manager?

Good programmers should be encouraged to remain programmers, and to continually refine their craft. They should be constantly seeking ways to improve their algorithms, get their applications to scale more efficiently, optimize their memory usage, and so on. Instead, many programmers, by necessity, seem more intent on learning the latest management techniques and buzzwords, and dropping phrases like "bottom line" into conversations. Meanwhile, seven out of eight IT projects fail.

As always, we'd all love to hear what you think. The best contribution to the debate, added as a comment to this blog, will receive a $50 Amazon voucher

Cheers,

Tony.

Over recent years, Agile development and Scrum have been championed by some developers, and various consulting firms, with a quasi-religious fervour. Initially, I was sceptical but Scrum has taken hold among the Red Gate development and testing teams and, as I started to witness their "daily scrum downs", I was moved to act. Why should Simple-Talk be left out? Dismissed as the couch potatoes of the new Agile World? In no time, I had persuaded the Simple-Talk team to form a Scrum and get ready for the next newsletter sprint

OK, so maybe for publishing, a "maul" is more appropriate than a Scrum. And not just any maul but a proper rolling maul, with boots, fists, shouting for blood subs and orange wedges, making agonising, inch-by-inch progress towards the greasy touchline; Phil Factor at the centre of it, holding fiercely onto the ball and Richard Morris blowing madly on his shiny Acme whistle.

And at the end of the match, a total of 15 tries, but just the one conversion.

Cheers!

Tony.

Everyone here knows what a nested loop join is, right? However, I am still willing to bet that if you ask three different people, you will get four subtly different answers.

I admire a person who can take a subject that has apparently been "done to death" and bring a new clarity to it. I think Stephane Faroult manages that with his video, SQL Joins, nested loops and all that in less than 6 minutes. His line in deadpan French humour appealed to me, and the "dance floor" analogy, with prospective dancers needing to choose suitably-sized partners, will stay with me the next time I'm asked to explain a merge join.

My first thought was that it would be great if we could bring this sort of clarity to some of the contentious topics of the day, such as cloud computing, MDX, entity framework, extended events and so on. Arguments are flaring up already and if we can get a short and clear explanation in early, it may save everyone a lot of pain.

Almost immediately, though, I realized the futility of this. What mileage is there in debating the intricacies of MDX until we can agree on something as fundamental as how date intervals work in SQL Server? It seems pointless to dissect complex internal structures of SQL Server until we can agree on how to pronounce SQL.

We need to unite behind a common understanding of some of the absolute basics of our trade. Otherwise how can we expect to be taken seriously? I propose that we compile a "top ten" list of questions to which we want straight answers.

Send in your topic suggestions, along with who you think could provide the guide. If you have a fresh way to present one of the endlessly-debated SQL Server topics, we'll do our utmost to persuade our current Geek of the Week, the famous cartoonist Larry Gonnick, to animate your script (as soon as he's finished work on his latest book!). It will then become the definitive 5-minute guide to which we will henceforth defer on Simple-Talk.

To get us started, I suggest the following candidate topics:

1. SQL: A definitive 5-minute guide to its pronunciation
2. What is NULL: The Final Answer
3. How date intervals work in SQL Server
4. The thinking person's guide to using cursors.
5. Table Variables. Their hidden purpose.
6. Should we be allowed to view Transaction logs?
7. At what level should we stop normalising databases?

We will get these videos made and, as usual, the best suggestions will receive a prize so add them as a comment to this blog!

Cheers,

Tony.

David DeWitt and his team at Microsoft have been exploring the 'next frontier' of architectures for building the parallel and scalable database systems that will be needed to support the "petabyte" data warehouse. The way forward is the "share nothing" grid architecture, which will underpin the likes of Madison, and will offer Commodity SMPs connected with commodity interconnect, and almost "limitless" scalability.

Ever since Oracle introduced RAC, Microsoft has been sniffy and dismissive, in the manner befitting someone with no solution to offer at all. When David presented his "Scalable Architectures" keynote speech at PASS, he could not resist taking several side-swipes at RAC, presenting it as a thoroughly second-rate solution that was "grid in name only".

Oracle had beefed up an existing shared-disk technology for database scale out (Oracle Parallel Server), called it RAC, and marketed it mercilessly. The problem was that it was eye-wateringly expensive and very few people really needed it. Some people were doing creative "out of the box" clustering with standby databases, log shipping, data guard, and so on, but few needed to move up to a full-on "grid" architecture.

However, won't the same be true of Microsoft's 'true grid' offering, when it arrives? According to Andrew Kelly, there are many people who think they need scale out, but in almost all cases, they would be better served by scaling up i.e. by just adding more memory/CPU to their existing box. Unless you either can't afford to be down for more than a few minutes, if a server crashes, or your monstrous CPU/memory demands simply cannot be sated by a single machine, then chances are you can live without clusters, and certainly without "the grid".

Ironically, an old Oracle friend once told me that when you move from a standalone machine to a 2-node cluster, your availability can actually go down slightly. A reliable, standalone SQL Server box, unmolested by human intervention, will just run for 99% of the time. Once you add in more hardware, more software, more complexity, things can and do go wrong a little more frequently, and it's a simple fact that it takes longer to fix a problem on a cluster than it does on a single box.

That's not to say SQL Server people shouldn't be interested in clustering, per se. If you have a valid business reason for building redundancy into your database system, then it is natural that you will want to investigate clustering, replication, database mirroring, distributed partitioned views…or whatever 'high-availability' solution works for you.

Just bear in mind that, with clusters, come extra complexity, extra cost, more manageability issues, and the need for dedicated, trained people to deal with the clustered environment. Multiply each of these many times and you have the same argument for "the grid". When the Microsoft share-nothing grid arrives it will be incredibly impressive, endlessly debated, and fabulously expensive; and almost no-one will need it.

As always, we'd love to hear what you think. Post your comments to this blog, and the best one will receive a prize.

Cheers,

Tony.