Click here to monitor SSC
  • Av rating:
  • Total votes: 9
  • Total comments: 1
Konrad Lukasik

Configuration Management with PowerShell and XML

03 June 2014
Database Lifecycle Management Patterns & Practices Library Stage 4

DATABASE LIFECYCLE MANAGEMENT PATTERNS & PRACTICES LIBRARY

Release Management

For the rapid delivery of any software application, there must be an effective configuration management system that is scripted. As the application increases in size it becomes more important that configuration information is kept in one place, without repetition, in version control. How to achieve this in practice? Konrad Lukasik explains a working solution.

Introduction

To minimize problems in any software delivery process, it is vital to keep every single script in the repository: Obviously, all aspects of configuration must be scripted, and configuration scripts are a part of this “every”. When I first attempted to apply elements of the Continuous Delivery (CD) process, I struggled with the practical difficulties of structuring and storing configuration information in any enterprise environment that has many interdependent .NET applications.

On the .NET platform, all configuration information is typically kept in XML file(s). Configuration files consist of sections; some of which are standard and control built-in features of the framework, whereas others are custom sections that are created by developers to govern the behavior of the components they write. The .NET framework provides a rich set of classes to read and write such configuration information. In Listing 1, we show a sample XML configuration file.

<configuration>

  <appSettings>

    <add key="webpages:Version" value="3.0.0.0" />

    <add key="webpages:Enabled" value="false" />

    <add key="ClientValidationEnabled" value="true" />

    <add key="UnobtrusiveJavaScriptEnabled" value="true" />

  </appSettings>

  <system.web>

    <authentication mode="None" />

    <compilation debug="true" targetFramework="4.5" />

    <httpRuntime targetFramework="4.5" />

  </system.web>

</configuration>

Listing 1 – Sample .NET configuration file

Much of this configuration information will be different from the development version when the application is deployed into production. For example, there might be debug or logging options to change, and there are likely to be connection strings to be changed to point to different databases.

To make this easier, Visual Studio 2010 introduced Web.config Transformations, a simple and straightforward method of transforming config files during publishing and packaging. Depending on the nature of the build configuration, Debug or Release for example, the appropriate conversion is applied to the XML configuration file. The technique is similar to XSLT but, while XSLT is a general transformation language, Web.config Transformations are optimized for the requirements of deployment. Listing 2 shows a simple transformation that removes an attribute.

<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">

  <system.web>

    <compilation xdt:Transform="RemoveAttributes(debug)" />

  </system.web>

</configuration>

Listing 2 – An example of Web.Config Transformations

However, out-of-the-box, the feature is not really sophisticated enough to meet the needs of complex environments. This is because:

  • It requires a separate transformation for each deployment environment. Even with a couple of such environments, it is hard to maintain the consistency of transformations. How to validate that a new app setting is correctly done for all environments? Manual verification and maintenance is time-consuming and error-prone.
  • It does not allow configuration-sharing between applications. For example where an application has a UI layer that uses web services, there should be only one parameter defining the port on which services are running and it should be used for deployment of both components.

In one of the projects at Objectivity, we dealt with these problems by using Web.config Transformation as a basis for a more sophisticated solution. Let’s look closely at it, as I strongly believe that the structural concepts are broadly universal and can be applied more generally to Windows environments, regardless of deployment language or storage technology.

Types of configuration

Within most .NET-based solutions, we will come across three types of configuration setting:

  • Application settings – used in *.config files to influence behavior of application (Listing 1)
  • Deployment process parameters – necessary to perform installation, for example location of DB backups
  •  Topology– defines servers, application layers, package assignments

I will go through each type and describe in more detail how each type of configuration setting can be organized.

Tokens and Applications settings

Our configuration management system was based on the idea of a token. A token, in this context, is a unique macro substitution string, which does not appear naturally in any config files and therefore can be easily and safely substituted during deployment the with actual, environment-specific, value. For example we chose the following format convention to use to define tokens:

##token_name##

We used these tokens to control configuration items such as:

  • connection strings
  •  names of application pools
  • web sites names
  • ports of HTTP and NET.TCP bindings
  •  notification email addresses
  • paths to DB backups

Building blocks

From the architectural perspective, our solution had three main building blocks:

  •  Web.config Transformations – the Visual Studio feature helped us to convert the developers’ settings into a more generic configuration template
  • XML file(s) – used to store tokens (an example on Listing 3)
  • PowerShell scripts – facilitated remote placement of environment-specific values in config files during deployment

<tokens>

    <token key="##DB_CONNECTION_STRING##" value="server=10.20.3.27;database=prodDb;uid=myProdUser;password=myProdPass;" />

    <token key="##Services_Install_Root##" value="D:\Services" />

    <token key="##INSTALL_ROOT##" value="C:\_DSL\INT" />

    <token key="##Install_Certificates##" value="true" />

</tokens>

Listing 3 – Sample XML with tokens

XML works really well as a format for configuration information, because it is widely known by developers and in addition, it integrates well with PowerShell, a language of deployment automation on Windows platform. Listing 4 demonstrates how easy it is to load a configuration file (defined in Listing 3), iterate over the nodes, accessing the attributes.

$tokens = [xml] (Get-Content "tokens.xml")

foreach ($token in $tokens.tokens.token)

{

    "$($token.key) = $($token.value)"

}

Listing 4 – XML operations in PowerShell

Transformation process

The three transformation stages of the configuration settings of the application, illustrated in this diagram are:

  •  Developers’ Config Files:
    Developers were working on their own version of configuration files, adapted to their local environment and checked it in together with the code changes. Local configuration could be shared due to similarity of environments, for example: the same naming of local DBs, keeping sources in the same folder, etc. That simplified development process and definitely helped us to avoid some merge conflicts.
  •  Generic Config Files:
    Next, during the Continuous Integration (CI) process on TFS, local configuration was converted with Visual Studio Web.config Transformations into generic files having tokens for settings, which are environment-specific. Those generic files were then embedded in MsDeploy packages (ZIP files containing application binaries), which we used for deployment to all environments. This is in line with another Continuous Delivery principle i.e. “Build once, deploy many“.
  • Environment-specific Environment settings:
     Finally, as part of the installation process, PowerShell scripts replaced tokens with values defined for a given environment. This is done on the destination server by using ‘remoting’ or remote execution. The Environment-specific configuration was kept in XML files in the repository, each environment with a separate token file, together with the code.

 This diagram below illustrates how connection string was altered throughout the process:

Connection string value substitution (Click to see full-size)

Tokens’ inheritance

The XML files that stored the values of the configuration Macro tokens were organized hierarchically, so that the default values of the parent tokens from the Root XML could be inherited or overwritten by child tokens for any individual environment. An approach like this minimizes the number of tokens that are kept in environment-specific files and therefore simplifies maintenance.

The Root XML defined:

  • The global list of tokens
  • The description of each token – very important from maintenance perspective. After a few months, nobody remembers what a particular token is used for unless it is documented, no matter how descriptive the name may be
  •  The default values
  • A flag to indicate whether the token must be defined at environment level – this attribute was used for the configuration health check

Local XML files defined each environment-specific token-overrides and were stored in their corresponding environment folder. Here is what the physical structure looked like:

Environments folder

  • RootTokens.XML
  • <TEST> folder
    • Tokens.XML
  • <UAT> folder
    • Tokens.XML
  • <Production> folder
    • Tokens.XML

Sub-tokens

The design I’ve described, using tokens with inheritance, is very flexible and it may seem that it would be sufficient for its’ purpose. However we found that some of configuration tokens were interdependent. For example, the HTTP port on which the WCF service is running was declared the first time for service-deployment and a second time for the UI layer using it. As mistakes with such dependencies usually come to light too late in the testing phase, we wanted to avoid them as much as possible. So we applied DRY principle to configuration files and introduced the concept of sub-tokens. Now a token’s value could refer to other token(s) and so the deployment process was enriched with another step – the token pre-processing phase, which resolved those dependencies.

Here is a token referring to sub-token:

<token key="##INSTALL_ROOT##" value="##Services_Install_Root##\INT" />

Deployment process parameters

Because PowerShell was our language of automation for the deployment process, we could easily access tokens from the deployment code through the global variable. Here is an example showing how to load tokens (for simplicity without support for sub-tokens) and use one of them to run SQL script against DB:

function Get-Tokens($tokenRepo) {  

    $hash = @{}

 

    [xml]$tokenConfig = Get-Content $tokenRepo

    $tokenConfig.tokens.token | ForEach-Object {  

        $hash += @{$_.key = $_.value}

    }

      

    return $hash

}

 

 

$tokens = Get-Tokens -tokenRepo "$environments_dir\$environment\tokens.xml"

 

$dbConnectionStringToken = "##DB_CONNECTION_STRING##"  

 

Write-Log -message "Db initialization started..."

 

# function below checks whether token is present and if not, raises an error

Verify-Token $tokens $dbConnectionStringToken

 

$dbConnectionString = $tokens[$dbConnectionStringToken]

 

$dbConnectionBuilder = New-Object System.Data.SqlClient.SqlConnectionStringBuilder

$dbConnectionBuilder.set_ConnectionString($dbConnectionString)

      

RunSqlFile $dbConnectionBuilder (Join-Path $initDbScriptsPath "UpdateDictionaryTables.sql")

      

Write-Log -message "Db initialization completed."

One could argue whether embedding (and possibly duplicating) token names in PowerShell code is a good practice or not, but we had more pressing concerns on our minds and, in practice, this approach worked for us.

Topology

Describing the variables that affect the topology of the infrastructure via XML is very similar to   using UML Infrastructure Diagrams for the given server environment. It defines things like:

  • The Servers to use (hostnames or IP addresses)
  • The Logical layers to which those servers belong to – this allows us to easily express load-balanced configurations
  • The Packages to install and on which layer

The Topology XML document was, due to its uniqueness, stored in a separate file with its own bespoke format. I think that a sample will explain more than a raw description, so here it is one:

<Platform>

  <Environment>

    <AppTier name="Web">

      <appServer name="10.20.3.32" />

      <appServer name="10.20.3.31" />

    </AppTier>

    <AppTier name="UI">

      <appServer name="10.20.3.30" />

      <appServer name="10.20.3.29" />

    </AppTier>

    <AppTier name="App">

      <appServer name="10.20.3.28" />

    </AppTier>

    <AppTier name="Db">

      <appServer name="10.20.3.27" />

    </AppTier>

  </Environment>

 

  <Packages>

    <Package name="Phoenix.ImportService" appTier="App" />

    <Package name="Phoenix.Queuing.WindowsService" appTier="App" />

   

    <!-- web apps -->

    <Package name="Phoenix.Dealing.Services" appTier="Web" />

    <Package name="Phoenix.ModelPortfolio.Services" appTier="Web" />

    <Package name="Phoenix.B2B2C.Services" appTier="Web" />

    <Package name="Phoenix.Users.Services" appTier="Web" />

    <Package name="Phoenix.OS.Services" appTier="Web" />

    <Package name="Phoenix.SSOBroker.Services" appTier="Web" />

    <Package name="Phoenix.Files.Services" appTier="Web" />

 

    <Package name="Phoenix.Web.FrontOffice" appTier="UI" />

    <Package name="Phoenix.Web.BackOffice" appTier="UI" />

  </Packages>

</Platform>

<

During the deployment process, PowerShell scripts loaded the XML topology document for the target server-environment and used it to find out which MsDeploy packages to install on which servers. An MsDeploy package contains application artifacts such as: DLLs, HTML, CSS or JavaScript files, XML configuration, etc.

In addition, every package was accompanied by a manifest file (there is a sample on Listing 5). It is an XML document that defines the type of application (web, desktop, and windows service) and its related parameters (application pool and web site parameters in case of web application or destination folder in case of desktop app). In our approach, the manifest could also utilize tokens and therefore change parameter values between environments. But let's not dig too deep in details, because the deployment process itself is a topic worth a separate article.

<deployment>

  <service physicalPath="##Services_Install_Root##\Phoenix.Queuing.WindowsService"

           displayName="Phoenix.Queuing.WindowsService"

           description="Phoenix.Queuing.WindowsService"

           username="NT AUTHORITY\NetworkService">

  </service>

</deployment>

Listing 5 – Application’s manifest file

Conclusions and future directions

The configuration structure I’ve described is not perfect but, nonetheless, it proved sufficient in our case and allowed us to successfully deliver three fairly complex sub-systems. Configuration maintenance and deployment automation was part of our ‘Definition-of-Done’ and every developer was encouraged to get familiar with the configuration architecture. This helped make the project more predictable by removing any justification for postponing deployment-related activities till the last minute.

Configuration-completeness was verified during CI builds as the first step of the process (the so-called build framework self-check). Our feedback loop was very fast so, if something basic went wrong, CI caught it right at the beginning.

World and technology inevitably moves forward and these days with gained experience we would do some of the things differently. For example, currently I and my fellow colleagues are exploring whether INI files could be better format for configuration than XML – although XML is powerful and generic, it is also hard for people to read and change and some valid alternatives (like INI, JSON or DSL) exist.

Another technological advance could well affect how we manage configuration in future. In October 2013, Microsoft released version 4 of PowerShell. One of new features is “Desired State Configuration” (DSC) – declarative language extensions that enable deployment and configuration management. What DSC handles perfectly is the topology, i.e. one of discussed configuration types. You would have to add support for tokens on your own, but, as far as I know, none of the deployment products running on Microsoft stack (e.g. “Octopus Deploy” or “Release Management for Visual Studio 2013”) deliver support for  configuration that is as sophisticated. Although PowerShell 4 cannot be installed and used in every environment due to the limited versions of the Windows operating system that it supports, I would definitely give it a try if only possible. Microsoft is investing strongly in PowerShell and already there are several resource kits available giving you the means to: configure IIS, change Active Directory, install SQL Server, manage a cluster or firewall rules. I am sure that the future will bring many improvements in this area.

This article is part of our Database Lifecycle Management patterns & practices.

Return to the library to find more articles on DLM, or visit red-gate.com/dlm for more information about SQL Server products for database lifecycle management.

Konrad Lukasik

Author profile:

Konrad Lukasik is an enthusiast of Microsoft technologies in general and .NET in particular. A professional with ten years of commercial experience. Currently works as a Technical Architect at Objectivity (http://www.objectivity.co.uk/) and helps teams deliver high-quality software. He continuously works on making things “as simple as possible, but not simpler”.

Search for other articles by Konrad Lukasik

Rate this article:   Avg rating: from a total of 9 votes.


Poor

OK

Good

Great

Must read
Have Your Say
Do you have an opinion on this article? Then add your comment below:
You must be logged in to post to this forum

Click here to log in.


Subject: source code powershell
Posted by: kiquenet (view profile)
Posted on: Monday, November 17, 2014 at 12:36 AM
Message: What is

Write-Log
Verify-Token
RunSqlFile

functions?

maybe interesting RunSqlFile for Oracle Database.

PD: great series

 

Top Rated

Building a Customised ALM Platform with TFS and VSO
 The latest versions of Team Foundation Server are not only sophisticated, but extensible. Continue... Read more...

ASP.NET SignalR: Old-fashioned Polling, Just Done Better
 A website often needs to update a page as the underlying data changes. You can, of course, just poll... Read more...

Rethinking the Practicalities of Recursion
 We all love recursion right up to the point of actually using it in production code. Why? Recursion... Read more...

Acceptance Testing with FitNesse: Multiplicities and Comparisons
 FitNesse is one of the most popular tools for unit testing since it is designed with a Wiki-style... Read more...

Acceptance Testing with FitNesse: Documentation and Infrastructure
 FitNesse is a popular general-purpose wiki-based framework for writing acceptance tests for software... Read more...

Most Viewed

A Complete URL Rewriting Solution for ASP.NET 2.0
 Ever wondered whether it's possible to create neater URLS, free of bulky Query String parameters?... Read more...

Visual Studio Setup - projects and custom actions
 This article describes the kinds of custom actions that can be used in your Visual Studio setup project. Read more...

.NET Application Architecture: the Data Access Layer
 Find out how to design a robust data access layer for your .NET applications. Read more...

Calling Cross Domain Web Services in AJAX
 The latest craze for mashups involves making cross-domain calls to Web Services from APIs made publicly... Read more...

10 Reasons Why Visual Basic is Better Than C#
 After having converted a whole lot of training materials based on VB.NET into C#, Andy ‘Wise Owl’ Brown... Read more...

Why Join

Over 400,000 Microsoft professionals subscribe to the Simple-Talk technical journal. Join today, it's fast, simple, free and secure.