/* SQL Server database engine doesn't have a MEDIAN() aggregate function. This is probably because there are several types of median, such as statistical, financial or vector medians. Calculating Medians are essentially a row-positioning task, since medians are the middle value of an ordered result. Easy to do in SQL? Nope. Joe Celko explains why

The workbench requires SQL Server 2008, but is easily modified to 2005.*/

/*

A Date for an Mock Feud

If you are an older SQL programmer, you will remember back in the 1990’s there were two database magazines on the newsstands, DBMS and DATABASE PROGRAMMING & DESIGN. They started as competitors, but Miller-Freeman bought DBMS from M&T and kept publishing both titles. Later the two titles are merged into INTELLIGENT ENTERPRISE from CMP Publications, which is more management oriented than the two original magazines.

I started a regular freelance column in DATABASE PROGRAMMING & DESIGN in 1990 and switched to DBMS in 1992 as an employee. Chris Date took my old slot back at DATABASE PROGRAMMING & DESIGN. There was a period where we did a version of the famous Fred Allen and Jack Benny mock feud from the classic era of American radio. Chris would run a column with some position on a topic in his column and I would counter it in my next column. It was great fun, it made people think and (most important) it sold magazines and kept me employed.

One of the topics that bounced around the SQL community in those columns was how to compute the Median with SQL. So Chris and I had to get involved in that and toss code around. Today, the Median is easy to do, but it was a good programming exercise in its day.

The 'Simple', or financial Median

The Median is not a built-in function in Standard SQL, like AVG(), and it has several different versions. In fact, most descriptive statistics have multiple versions, which is why SQL stayed away from them and left that job to specialized packages like SAS and SPSS. I will not even mention floating point error handling.

The simple Median is defined as the value for which there are just as many cases with a value below it as above it. If such a value exists in the data set, this value is called the statistical Median. If no such value exists in the data set, the usual method is to divide the data set into two halves of equal size such that all values in one half are lower than any value in the other half. The Median is then the average of the highest value in the lower half and the lowest value in the upper half, and is called the financial Median. In English, sort the data, cut it in the middle and look at the guys on either side of the cut.

The financial Median is the most common term used for this Median, so we will stick to it. Let us use Date's famous Parts table, from several of his textbooks, which has a column for part_wgt in it, like this:

*/

CREATE TABLE Parts

(part_nbr VARCHAR(5) NOT NULL PRIMARY KEY,

part_name VARCHAR(50) NOT NULL,

part_color VARCHAR(50) NOT NULL,

part_wgt INTEGER NOT NULL,

city_name VARCHAR(50) NOT NULL);

INSERT INTO Parts (part_nbr, part_name, part_color, part_wgt, city_name)

VALUES ('p1', 'Nut', 'Red', 12, 'London'),

('p2', 'Bolt', 'Green', 17, 'Paris'),

('p3', 'Cam', 'Blue', 12, 'Paris'),

('p4', 'Screw', 'Red', 14, 'London'),

('p5', 'Cam', 'Blue', 12, 'Paris'),

('p6', 'Cog', 'Red', 19, 'London');

/*

First sort the table by weights and find the three rows in the lower half of the table. The greatest value in the lower half is 12; the smallest value in the upper half is 14; their average, and therefore the Median, is 13. If the table had an odd number of rows, we would have looked at only one row after the sorting.

Date's First Median

Date proposed two different solutions for the Median. His first solution was based on the fact that if you duplicate every row in a table, the Median will stay the same. The duplication will guarantee that you always work with a table that has an even number of rows. Since Chris (and myself) are strongly opposed to redundant data, this was an unusual approach for him to use.

The first version that appeared in his column was wrong and drew some mail from me and from others who had different solutions. Here is a corrected version of his first solution:

*/

go

CREATE VIEW Temp1

AS SELECT part_wgt FROM Parts

UNION ALL

SELECT part_wgt FROM Parts;

go

CREATE VIEW Temp2

AS SELECT part_wgt

FROM Temp1

WHERE (SELECT COUNT(*) FROM Parts)

<= (SELECT COUNT(*)

FROM Temp1 AS T1

WHERE T1.part_wgt >= Temp1.part_wgt)

AND (SELECT COUNT(*) FROM Parts)

<= (SELECT COUNT(*)

FROM Temp1 AS T2

WHERE T2.part_wgt <= Temp1.part_wgt);

go

SELECT AVG(DISTINCT part_wgt) AS median

FROM Temp2;

/*

This involves the construction of a table of duplicated values, which can be expensive in terms of both time and storage space. The use of AVG(DISTINCT x) is important because leaving it out would return the simple average instead of the Median. Consider the set of weights (12, 17, 17, 14, 12, 19). The doubled table, Temp1, is then (12, 12, 12, 12, 14, 14, 17, 17, 17, 17, 19, 19). But because of the duplicated values, Temp2 becomes (14, 14, 17, 17, 17, 17), not just (14, 17). The simple average is (96 / 6.0) = 16; it should be (31/2.0) = 15.5 instead.

Celko's First Median

I found a slight modification of Date's solution will avoid the use of a doubled table, but it depends on a CEILING() function.

*/

SELECT MIN(part_wgt) -- smallest value in upper half

FROM Parts

WHERE part_wgt

IN (SELECT P1.part_wgt

FROM Parts AS P1, Parts AS P2

WHERE P2.part_wgt >= P1.part_wgt

GROUP BY P1.part_wgt

HAVING COUNT(*)

<= (SELECT CEILING(COUNT(*) / 2.0)

FROM Parts)

UNION

SELECT MAX(part_wgt) -- largest value in lower half

FROM Parts

WHERE part_wgt IN (SELECT P1.part_wgt

FROM Parts AS P1, Parts AS P2

WHERE P2.part_wgt <= P1.part_wgt

GROUP BY P1.part_wgt

HAVING COUNT(*) <=

(SELECT CEILING(COUNT(*) / 2.0)

FROM Parts)))

--or using the same idea and a CASE expression:

SELECT AVG(DISTINCT CAST(part_wgt AS FLOAT)) AS median

FROM (SELECT MAX(part_wgt)

FROM Parts AS B1

WHERE (SELECT COUNT(*) + 1

FROM Parts

WHERE part_wgt < B1.part_wgt)

<= (SELECT CEILING (COUNT(*) / 2.0)

FROM Parts)

UNION ALL

SELECT MAX(part_wgt)

FROM Parts AS B

WHERE (SELECT COUNT(*) + 1

FROM Parts

WHERE part_wgt < B.part_wgt)

<= CASE (SELECT (COUNT(*)% 2)

FROM Parts)

WHEN 0

THEN (SELECT CEILING (COUNT(*)/2.0) + 1

FROM Parts)

ELSE (SELECT CEILING (COUNT(*)/2.0)

FROM Parts)

END) AS medians(part_wgt);

/*

The CEILING() function is to be sure that if there is an odd number of rows in Parts, the two halves will overlap on that value.

Date's Second Median

Now the war was on! Did you notice that we had to use a lot of self-joins in those days? That stinks. Date's second solution was based on Celko's Median, folded into one query:

*/

SELECT AVG(DISTINCT Parts.part_wgt) AS median

FROM Parts

WHERE Parts.part_wgt IN

(SELECT MIN(part_wgt)

FROM Parts

WHERE Parts.part_wgt

IN (SELECT P2.part_wgt

FROM Parts AS P1, Parts AS P2

WHERE P2.part_wgt <= P1.part_wgt

GROUP BY P2.part_wgt

HAVING COUNT(*)

<= (SELECT CEILING(COUNT(*) / 2.0)

FROM Parts))

UNION

SELECT MAX(part_wgt)

FROM Parts

WHERE Parts.part_wgt

IN (SELECT P2.part_wgt

FROM Parts AS P1, Parts AS P2

WHERE P2.part_wgt >= P1.part_wgt

GROUP BY P2.part_wgt

HAVING COUNT(*)

<= (SELECT CEILING(COUNT(*) / 2.0)

FROM Parts)));

/*

Date mentions that this solution will return a NULL for an empty table and that it assumes there are no NULLs in the column. But you have to remember Chris Date, unlike Dr. Codd and SQL, does not use NULLs and thinks they should not be in the Relational Model. If there are NULLs, the COUNT(*) will include them and mess up the computations. You will need to get rid of them with a COUNT(part_nbr) instead. Play with some sample data and see the differences.

Murchison's Median

Others joined in the battle. Rory Murchison of the Aetna Institute had a solution that modifies Date's first method by concatenating the key to each value to make sure that every value is seen as a unique entity. Selecting the middle values is then a special case of finding the n-th item in the table.

*/

SELECT AVG(part_wgt)

FROM Parts AS P1

WHERE EXISTS

(SELECT COUNT(*)

FROM Parts AS P2

WHERE CAST(part_wgt AS CHAR(5)) + P2.part_nbr >=

CAST(part_wgt AS CHAR(5)) + P1.part_nbr

HAVING COUNT(*) = (SELECT FLOOR(COUNT(*) / 2.0)

FROM Parts)

OR COUNT(*) = (SELECT CEILING((COUNT(*) / 2.0))

FROM Parts));

/*

This method depends on being able to have a HAVING clause without a GROUP BY clause, which is part of the ANSI standard but often missed by new programmers.

Occurrence Count Table

I came back with a working table with the values and a tally of their occurrences from the original table. Now that we have this table, we want to use it to construct a summary table that has the number of occurrences of each part_wgt and the total number of data elements before and after we add them to the working table.

Philip Vaughan of San Jose, CA proposed a simple Median technique based on a VIEW with unique weights and number of occurrences and then a VIEW of the middle set of weights.

*/

go

CREATE VIEW ValueSet(part_wgt, occurrence_cnt)

AS SELECT part_wgt, COUNT(*)

FROM Parts

GROUP BY part_wgt;

go

/*

The MiddleValues VIEW is used to get the Median by taking an average. The clever part of this code is the way that it handles empty result sets in the outermost WHERE clause that result from having only one value for all weights in the table. Empty sets sum to NULL because there is no element to map the index

*/

CREATE VIEW MiddleValues(part_wgt)

AS SELECT part_wgt

FROM ValueSet AS VS1

WHERE (SELECT SUM(VS2.occurrence_cnt)/2.0 + 0.25

FROM ValueSet AS VS2) >

(SELECT SUM(VS2.occurrence_cnt)

FROM ValueSet AS VS2

WHERE VS1.part_wgt <= VS2.part_wgt) - VS1.occurrence_cnt

AND (SELECT SUM(VS2.occurrence_cnt)/2.0 + 0.25

FROM ValueSet AS VS2) >

(SELECT SUM(VS2.occurrence_cnt)

FROM ValueSet AS VS2

WHERE VS1.part_wgt >= VS2.part_wgt) - VS1.occurrence_cnt;

go

SELECT AVG(part_wgt) AS median FROM MiddleValues;

/*

Median with Characteristic Function

Anatoly Abramovich, Yelena Alexandrova, and Eugene Birger presented a series of articles in SQL Forum magazine on computing the Median (SQL Forum 1993, 1994). They define a characteristic function, which they call delta, using the SIGN() function. The delta or characteristic function accepts a Boolean expression as an argument and returns one if it is TRUE and zero if it is FALSE or UNKNOWN. We can construct the delta function easily with a CASE expression.

The authors also distinguish between the statistical Median, whose value must be a member of the set, and the financial Median, whose value is the average of the middle two members of the set. A statistical Median exists when there is an odd number of items in the set. If there is an even number of items, you must decide if you want to use the highest value in the lower half (they call this the left Median) or the lowest value in the upper half (they call this the right Median).

The left statistical Median of a unique column can be found with this query, if you will assume that we have a column called bin_nbr that represents the storage location of a part in sorted order.

*/

SELECT P1.bin_nbr

FROM Parts AS P1, Parts AS P2

GROUP BY P1.bin_nbr

HAVING SUM(CASE WHEN (P2.bin_nbr <= P1.bin_nbr) THEN 1 ELSE 0 END)

= (COUNT(*) / 2.0);

/*

Changing the direction of the test in the HAVING clause will allow you to pick the right statistical Median if a central element does not exist in the set. You will also notice something else about the Median of a set of unique values: It is usually meaningless. What does the Median bin_nbr number mean, anyway? A good rule of thumb is that if it does not make sense as an average, it does not make sense as a Median.

The statistical Median of a column with duplicate values can be found with a query based on the same ideas, but you have to adjust the HAVING clause to allow for overlap; thus, the left statistical Median is found by

*/

SELECT P1.part_wgt

FROM Parts AS P1, Parts AS P2

GROUP BY P1.part_wgt

HAVING SUM(CASE WHEN P2.part_wgt <= P1.part_wgt

THEN 1 ELSE 0 END)

>= (COUNT(*) / 2.0)

AND SUM(CASE WHEN P2.part_wgt >= P1.part_wgt

THEN 1 ELSE 0 END)

>= (COUNT(*) / 2.0);

/*

Should Parts contain an even number of entries with the (COUNT(*)/ 2) entry not repeated, this query may return FLOOR(AVG(DISTINCT part_wgt)), as happens when SQL Server computes an average of integers. This can be fixed by changing the inner SELECT to:

*/

SELECT (P1.part_wgt * 1.0) AS part_wgt

/*

Notice that here the left and right medians can be the same, so there is no need to pick one over the other in many of the situations where you have an even number of items. Switching the comparison operators in the two CASE expressions will give you the right statistical Median.

**Henderson's Median**

Another version of the financial Median, which uses the CASE expression in both of its forms, is:

*/

SELECT CASE COUNT(*) % 2

WHEN 0 -- even sized table

THEN (P1.part_wgt + MIN(CASE WHEN P2.part_wgt > P1.part_wgt

THEN P2.part_wgt

ELSE NULL END))/2.0

ELSE P1.part_wgt --odd sized table

END AS median

FROM Parts AS P1, Parts AS P2

GROUP BY P1.part_wgt

HAVING COUNT(CASE WHEN P1.part_wgt >= P2.part_wgt

THEN 1

ELSE NULL END)

= (COUNT(*) + 1) / 2;

/*

This answer is due to Ken Henderson. The only instance in which [[2]] is correct is when Parts has an odd number of entries and the middle (COUNT(*) / 2 + 1) entry is not repeated in the data.

Celko's Third Median

Another approach involves looking at a picture of a line of sorted values and seeing where the Median would fall. Every value in column part_wgt of the table partitions the table into three sections, the values that are less than part_wgt, equal to part_wgt or greater than part_wgt. We can get a profile of each value with a tabular subquery expression.

Deriving the query is a great exercise in SQL logic and math. Don't do it unless you just like the exercise. The query finally becomes:

*/

SELECT AVG(DISTINCT part_wgt)

FROM (SELECT P1.part_wgt

FROM Parts AS P1, Parts AS P2

GROUP BY P1.part_nbr, P1.part_wgt

HAVING SUM(CASE WHEN P2.part_wgt = P1.part_wgt

THEN 1 ELSE 0 END)

>= ABS(SUM(CASE WHEN P2.part_wgt < P1.part_wgt THEN 1

WHEN P2.part_wgt > P1.part_wgt THEN -1

ELSE 0 END)))

AS Partitions;

/*

OLAP Median

The new OLAP function in the SQL-99 Standard allow you to replace COUNT() functions with row numberings. Let’s assume we have a table or VIEW named RawData that has only one numeric column. The obvious solution for data with an odd number of elements is:

*/

WITH SortedData (x, sort_place)

AS

(SELECT x, ROW_NUMBER() OVER(ORDER BY x ASC)

FROM RawData)

SELECT x AS median

FROM SortedData

WHERE sort_place = ((SELECT COUNT(*) FROM RawData)/ 2) + 1;

--A slight modification for an even number of elements:

WITH SortedData (x, sort_place)

AS

(SELECT x, ROW_NUMBER() OVER(ORDER BY x ASC)

FROM RawData)

SELECT AVG(x) AS median

FROM SortedData

WHERE sort_place

IN ((SELECT COUNT(*) FROM RawData)/ 2),

(SELECT COUNT(*) FROM RawData)/ 2) + 1);

/*

You can combine both of these into one query with a CASE expression and I leave that as an exercise for the reader.

But there is another question we have not asked about an even number of rows. Assume the data set look like {1, 2, 2, 3, 3, 3} so the usual median would be AVG({2, 3}) = 2.5 as the measure of central tendency. But if we consider where the real trend is, then we would use the set-oriented weighted median with AVG({2, 2, 3, 3, 3}) = 13/5 = 2.6 as the measure of central tendency.

This leads to the final query which can handle odd and even counts and do the weighted Median:

*/

WITH SortedData (x, hi, lo)

AS

(SELECT x,

ROW_NUMBER() OVER(ORDER BY x ASC),

ROW_NUMBER() OVER(ORDER BY x DESC)

FROM RawData)

SELECT AVG(x * 1.0) AS median

FROM SortedData

WHERE hi IN (lo, lo+1, lo-1);

/*

This might be slower than you would like, since the CTE might do two sorts to get the "hi" and "lo" values. At the time of this writing, ROW_NUMBER() is new to all SQL products, so not as much work has gone into optimizing them. A smarter optimizer would map the i-th element of a sorted list of (n) items in ASC order to the (n-i+1)-th element for DESC.

Want to try your own versions?

*/