Click here to monitor SSC
  • Av rating:
  • Total votes: 10
  • Total comments: 0
Richard Morris

Agile Techniques for developing SQL Source Control

03 August 2011

In this interview, Stephanie Herr, Development Manager for SQL Tools at Red Gate, talks about the recent SQL Source Control development project. As a certified Scrum Master, Stephanie was keen to use Agile techniques throughout the development process, and she explains how the team maximized user feedback and ensured that customers got what they wanted, as quickly as possible.

Stephanie has Bachelor of Science degrees in Computer Science and Mathematics and a Master’s degree in Software Engineering with a concentration on database design. She has worked as a software/database engineer for 8 years at a large US government contractor where she got her Green Belt certification in Lean Six Sigma techniques. She has been with Red Gate for two and a half years.

RM:
Stephanie, you were the Project Manager for versions 1 and 2 of SQL Source Control. Can you tell me how you went about developing the latest iteration... What was the reason for the upgrade? What problems were you trying to solve?
SH:
When we were developing SQL Source Control v1.0, we set up a Suggestion Forum, where users could suggest new features or enhancements and vote on these ideas. This was set up for our very first Early Access Release in January 2010. We averaged releasing a new “early access” version every three weeks until the final release was available in June that year, and we tried to take these suggestions into consideration. During this time, we got a lot of great feedback from our users and the Suggestion Forum is still available directly from the tool.
Version 1.0 focused on enabling users to source control their database schemas directly from SQL Server Management Studio (SSMS) to their existing source control systems. The idea was to provide a bridge between the development environment and the source control systems they were already using. Subversion and Team Foundation Server were supported in version 1.0.
The number one requested feature was to add support for source controlling static data (lookup/reference data). Each user can vote for an idea up to 3 times on our Suggestion Forum, and this feature got 262 votes from 107 users. So it was obvious what we should do for v2.0 from our users’ perspective. And it made sense too, because in order to have a complete picture of the application at any given version or point in time, you need to source control the reference data that drives your application alongside the schema and the application code.
There are many other things to consider when deciding what goes into a release, including technical implementation, market, long term goals, conversations with customers at trade shows, things our sales team hear, and so on. I worked very closely with the Product Owner, David Atkinson, and the team to prioritize what goes into the releases. In particular, the team can identify “quick wins”, which may not have a lot of votes, but are easy to implement and can make a big difference to the end product.
The next version, 2.1, also evolved out of our users’ requests. We added support for additional source control systems (Vault, Vault Pro, Git, Mercurial, Perforce, and any source control system with a command line), we made the commit check-in comment larger, and we now treat column ordering differences as a change.
This was well-received, so for our latest release, version 2.2, we once again looked at our users’ suggestions, and this time we added Object Filtering, Browse/Create folders when linking, and ‘Flicky Tabs’.
RM:
The new release is very much a collaboration then, with an early diagnosis, if you will, that SQL Source Control version 2 needed more texture, users asking for additional features, and the team listening and Red Gate delivering? Was there anything in the Forum you didn’t add because not enough people asked for it?
SH:
I don’t think we thought version 2 needed more texture. Each release focused on what we thought we should do next based on input from various sources. The purpose of version 2.0 was to support source controlling your data alongside your schema; version 2.1 then added the support of further source control systems like Vault, Vault Pro, GIT, Mercurial, Perforce, and any system that has a command line; while for version 2.2 we focused on object filtering.
Each release has extended the functionality of SQL Source Control by adding valuable features. There are a lot more ideas on the forum that haven’t been implemented yet, but if we didn’t release until we finished every feature that was requested, we would never have a release.
We are really embracing Scrum and Agile practices at Red Gate. We want to deliver valuable, working software to our users as soon as possible. If we develop a great feature and it’s just checked into our own source control system internally, then it’s not doing our users any good because they can’t start taking advantage of the feature. And it’s also not doing us any good, because we aren’t getting feedback from our users on what they think and how we could enhance the product. We are continuously looking at the feedback, and re-prioritising our backlog as we learn more.

Sometimes we implement a feature that a user suggested on the forum that doesn’t have the highest number of votes. This could be because we think it is important, it’s technically easy to implement, or it could be because it fits with our vision for the tool. Other features will stay open on the forum, and users can keep voting or commenting on them, and we’ll continue to monitor this.
I think it’s great that our releases are so short, because we can reconsider what to do next if circumstances change. We hear great ideas from users every day through our forum, phone conversations, tradeshows, or usability sessions, and we try to implement what we think would be of the most value for the majority of users.
RM:
Do you think SQL Source Control 2.2 is a stronger product than version 2.1? You mentioned it contains 'Flicky Tabs'... what other features does it contain?
SH:
‘Flicky Tabs’ is a performance improvement that reuses information when a user switches between the Commit and Get Latest tabs within SQL Source Control. It’s really great that users don’t have to wait to re-generate the list of changes anymore.
Version 2.2 also contains “Object Filtering” and “Browsing/Creating Folders when linking a database to Source Control”. These are key benefits that also came from users’ suggestions.
Object Filtering allows users to specify objects that shouldn’t be source controlled. This is important if you have some local test schema or a test user that you don’t want to be committed to source control. Before this feature was developed you would have to uncheck these items every time you went to the Commit tab. Now, you can filter them out so they don’t appear on the Commit tab.
Browsing/Creating Folders when linking a database to source control makes it much easier to navigate through your repositories and set up the initial link to the source control system.
RM:
Would you say that developing version 2 was technically, or logistically, more difficult than version 1?
SH:
Yes, version 2 was a little bit more challenging both technically and logistically. We had to ensure that users on v1 could upgrade to v2 without any problems. We had given users of v1 a workaround for putting static data into source control using our SQL Data Compare tool, so we had to make sure that SQL Source Control version 2 would work with the existing repositories that had used this workaround. We also had fewer users taking up the early releases, because they were already using the stable v1 product in their development environments.
From a technical point of view, the team had to become familiar with the SQL Data Compare engine code that is used behind the scenes. We also wanted to ensure that users could commit the schema and data atomically, so that it would appear in the same changeset in their source control system. So, all these challenges had to be addressed.
RM:
Is there a typical customer for version 2.2 over and above those who already use it?
SH:
Any database professionals who are not source controlling their databases, or are not happy with their current process for source controlling their databases, should look at SQL Source Control.
We heard from a lot of users who have to script the SQL file manually to source control their changes, or they have to open a SQL file from a source control system outside of SSMS. These are very manual processes, and so it’s tempting for users to make changes directly to the database and skip source controlling altogether. SQL Source Control integrates source control functionality directly into SSMS where most database professionals are making changes to their databases anyway. And SQL Source Control’s blue indicators in SSMS’ Object Explorer are great reminders for when you have changes that need to be committed.
SQL Source Control also makes it really easy to share changes with others by getting the latest out of the source control system. And it gives users the power to undo changes if they’ve made a mistake and want to get back to a known good state.
One of the features I really like is the history; you can right-click on a database and see who has changed what, when, and why. This history is also available at an object level, so, for example, you can see exactly how a stored procedure has changed over time and get back to a previous version if needed.
RM:
What features in version 2.2 are you most proud of?
SH:
I think I talked a lot about the features already: Object Filtering, Browse/Create folders when linking, and ‘Flicky Tabs.’ I’m proud of them all, and I’m particularly pleased that we’re focusing on things that our users need and really adding value to what SQL Source Control can do.
RM:
Just how much time did you test version 2.2 before release? What processes did you go through?
SH:
We are using Scrum to develop SQL Source Control. We currently have 3 developers, 3 test engineers, a user experience specialist, and a technical author. As each story gets pulled into a sprint, the developers write unit tests to test the lowest level of functionality that they implement. The test engineers do some exploratory testing and write automated integration tests and GUI tests. These tests are added to our test suite, which runs every time a change is committed to our source control system, so we’re continuously checking that new changes don’t break existing functionality. If an issue is found, this is logged in our bug tracking system and a new sticky is added to our team’s Scrum whiteboard, to get fixed and re-tested before a story is considered complete.
We invest a lot of time in testing each story, and automating these tests. This means that when it comes to a release there’s not too much release testing we need to do because we have confidence in our automated tests, but we still do some release testing. We also do code reviews and test code reviews to help identify issues at the lowest level. This is a great way to spot things and learn from each other.
For a release, we have a Release Candidate, which is available through our support forum, and we send out an email to all our Early Access Program users for them to try it out before the final release. For some releases, we also have a Bug Hunt, where we set up laptops and invite everyone in the company to come and try the tool and test it out. And finally, the test engineers run a release test suite before each release, to test the features and issues we’ve hit in the past and any extra risks that we need to test for.
RM:
Red Gate has its ‘relentlessly tested’ strapline, but how far does this go with this version? How fast can you react to problems if someone finds a bug?
SH:
We really are relentless. It’s everyone’s responsibility to make sure the product is stable.
We use Red Gate’s SmartAssembly tool so that users can send error reports back to us if they have a problem when they’re using SQL Source Control. This has been a really powerful tool for making sure we hear from our users about problems they encounter, because it makes it easy for them to report issues – they don’t have to pick up the phone to call us or send us an email. The SmartAssembly report contains the user’s log file too, so it’s much easier for our developers to understand and reproduce the issue quickly.
How fast we can react depends on what the problem is. Recently, an early access user hit a problem with the 2.2 Release Candidate. The issue arises under a very specific set of circumstances, and there was no way we would have caught it beforehand. Within a few working days of receiving the SmartAssembly report, we sent the user a Private Build, which he has confirmed fixed his issue. It’s really amazing how quickly the team turns these issues around.
Of course, there are some issues that we can’t turn around quickly. If a user doesn’t provide their email address when they send the report and we can’t reproduce the issue, then we feel pretty stuck.
RM:
I know everyone at Red Gate is closely involved in the business so people understand how it works. How do you think this dual role helps in developing software? Does it help you understand the needs of other businesses that much better? Is this the key to the success of Red Gate so far?
SH:
It’s so great that Red Gate is so open and details about the business are communicated freely within the company. I think this helps when developing software because we know exactly how what we do makes an impact to the business. This could be shown as downloads of the tools, marketing campaigns around the tools, sales of the tools, things the sales team hear from customers (both positive and negative), and issues that our support team is helping users with.
All in all, this definitely creates an engaging, empowered environment for the development teams to do great work and create great, ingeniously simple software for our users.

You can find out more about SQL Source Control and how quickly you can connect your existing source control system to SSMS, by visiting www.red-gate.com/5min.

You can also read more about the SQL Source Control project, and how Agile development techniques were used to optimize the project, in Regular Rapid Releases: An Agile Tale.

Richard Morris

Author profile:

Richard Morris is a journalist, author and public relations/public affairs consultant. He has written for a number of UK and US newspapers and magazines and has offered strategic advice to numerous tech companies including Digital Island, Sony and several ISPs. He now specialises in social enterprise and is, among other things, a member of the Big Issue Invest advisory board. Big Issue Invest is the leading provider to high-performing social enterprises & has a strong brand name based on its parent company The Big Issue, described by McKinsey & Co as the most well known and trusted social brand in the UK.

Search for other articles by Richard Morris

Rate this article:   Avg rating: from a total of 10 votes.


Poor

OK

Good

Great

Must read
Have Your Say
Do you have an opinion on this article? Then add your comment below:
You must be logged in to post to this forum

Click here to log in.
 

Phil Factor
Searching for Strings in SQL Server Databases

Sometimes, you just want to do a search in a SQL Server database as if you were using a search engine like Google.... Read more...

 View the blog

Top Rated

SQL Server XML Questions You Were Too Shy To Ask
 Sometimes, XML seems a bewildering convention that offers solutions to problems that the average... Read more...

Continuous Delivery and the Database
 Continuous Delivery is fairly generally understood to be an effective way of tackling the problems of... Read more...

The SQL Server Sqlio Utility
 If, before deployment, you need to push the limits of your disk subsystem in order to determine whether... Read more...

The PoSh DBA - Reading and Filtering Errors
 DBAs regularly need to keep an eye on the error logs of all their SQL Servers, and the event logs of... Read more...

MySQL Compare: The Manual That Time Forgot, Part 1
 Although SQL Compare, for SQL Server, is one of Red Gate's best-known products, there are also 'sister'... Read more...

Most Viewed

Beginning SQL Server 2005 Reporting Services Part 1
 Steve Joubert begins an in-depth tour of SQL Server 2005 Reporting Services with a step-by-step guide... Read more...

Ten Common Database Design Mistakes
 If database design is done right, then the development, deployment and subsequent performance in... Read more...

SQL Server Index Basics
 Given the fundamental importance of indexes in databases, it always comes as a surprise how often the... Read more...

Reading and Writing Files in SQL Server using T-SQL
 SQL Server provides several "standard" techniques by which to read and write to files but, just... Read more...

Concatenating Row Values in Transact-SQL
 It is an interesting problem in Transact SQL, for which there are a number of solutions and... Read more...

Why Join

Over 400,000 Microsoft professionals subscribe to the Simple-Talk technical journal. Join today, it's fast, simple, free and secure.