Statistics in SQL: Kendall’s Tau rank correlation

Statistical calculations in SQL are often perfectly easy to do. SQL was designed to be a natural fit for calculating correlation, regression and variance on large quantities of data. It just isn't always immediately obvious how. In the second of a series of articles, Phil factor shows how calculating a non-parametric correlation via Kendall's Tau or Spearman's Rho can be stress-free.… Read more

Generating Plots Automatically From PowerShell and SQL Server Using Gnuplot

When you are automating a number of tasks, or performing a batch of tests, you want a way of automating the production of your plots and graphs. Nothing beats a good graphical plot for giving the indications of how the process went. If you are using PowerShell and maybe also SQL Server, it pays to use a command-line plotting tool such as Gnuplot to do all the hard work. It turns out to be handy for a range of data jobs, turning PowerShell into a handy data science tool.… Read more

Doing Fuzzy Searches in SQL Server

A series of arguments with developers who insist that fuzzy searches or spell-checking be done within the application rather then a relational database inspired Phil Factor to show how it is done. When the database must find relevant material from search terms entered by users, the database must learn to expect, and deal with, both expected and unexpected … Read more

String Comparisons in SQL: The Metaphone Algorithm

When exploring the use of the Metaphone algorithm for fuzzy search, Phil couldn't find a SQL version of the algorithm so he wrote one. The Metaphone algorithm is built in to PHP, and is widely used for string searches where you aren't always likely to get exact matches, such as ancestral research and historical documents. It is particularly useful when comparing strings word-by-word. With a SQL version, it is easy to experiment on large quantities of data!… Read more


Over the history of personal computing, it seems that the best software was written by a team of five or fewer programmers. CP/M, the first PC operating system was built by one guy, Gary Kildall, though he got parts from others such as Gordon Eubanks. MSDOS originated in QDOS, again written by one person, Tim … Read more

Lists With, or Without, Ranges in both T-SQL and PowerShell

Whether you are working in a procedural language like PowerShell or in T-SQL, there is something slightly bothersome about having to deal with parameters that are lists, or worse with ranges amongst the values. In fact, once you have a way of dealing with them, they can be convenient, especially when bridging the gulf between application and the database. Phil Factor shows how to deal with them.… Read more

Representing Hierarchical Data for Mere Mortals

Why is it that we use XML, but with so little enthusiasm when it does so much, and is so feature-rich? Phil Factor argues that there are better ways of doing it, more complete than JSON, but easier to read than XML. To try to convince you, he gives a set of flying demos, using PowerShell and his PSYaml module, to illustrate how YAML can let you work faster, and more accurately.… Read more

PSYaml: PowerShell does YAML

PSYaml is a simple PowerShell module that I’ve written that allows you to serialize PowerShell objects to “YAML Ain’t Markup Language” (YAML) documents and deserialize YAML documents to PowerShell objects. It uses Antoine Aubry’s excellent YamlDotNet library To start, you can simply load the PowerShell file and the manifest from its home on GitHub PSYaml … Read more

On Being Economical with the Truth

A while back, I attended a presentation about a suite of software that allowed IT managers to track in some detail the progress of a development project, via static code analysis, a range of source control metrics and other magic. Bathed in these insights, management could instantly track the detail of what was going on … Read more

The SQL of Textonyms

The task of finding textonyms in SQL involves importing a list of common words and doing transformations on every word to convert it into what you'd need to type into the numeric keypad of your mobile phone to get that word. It's not that hard to do, but what is the quickest and most efficient way of doing it? Phil Factor investigates.… Read more

Data Security: Time to be Grown-Up

How do you quickly clear a room-full of application developers, short of shouting ‘fire’ and setting off a smoke-bomb? The answer, of course, is to stand up in front of them and announce that you are giving a talk about database access-control and security. Once, at a conference, I sat through a brilliant talk on … Read more

Measure of a DBA

Marketing people are often surprised when a new version of SQL Server doesn’t quite generate the atmosphere of excited anticipation that they expected. After all, people seem to get themselves in a state of frenzy when a new iPhone is introduced, and suddenly their present phone looks like a museum item. The urge to upgrade … Read more

Unmasking the Dynamic Data Masking

/* Revised 27th June   Dynamic data masking is a great product and solves some niche problems that come if you need to do certain testing with live data at the application level. You should, however, beware of using it as a database-level security device. I haven’t yet used it in testing because I don’t … Read more

How to Build and Deploy a Database from Object-Level Source in a VCS

It is easy for someone who is developing a database to shrug and say 'if only my budget would extend to buying fancy tools, then I could start using version control when developing or maintaining databases'. Phil Factor sets out to show that there are ways of maintaining object-level source control for SQL Server databases just using what Microsoft provides, It may not be perfect, but there are ways of doing it whatever your budget.… Read more