Executing SSIS Packages

30 August 2006
by Nigel Rivett

Executing all SSIS packages in a folder: three methods

A common requirement is to execute several SSIS packages as part of one logical process. This is especially true when one is transferring existing data from an external system.

If the indivisual packages are placed in a folder and named in such a way that a batch can be executed via a wildcard filemask, then they can be altered without changing any code, and can be used for different processes by changing the filemask. This article presents three different methods of executing all packages in a folder:

  1. SSIS control package
  2. Stored procedure
  3. DOS batch file

The methods are of differing complexity and flexibility, the SSIS package being by far the most difficult to code, and the DOS command being by far the simplest but also the least flexible.

The folder that contains the packages is d:\TestPackages\.

All the packages to be run have names of the format t1…….dtsx.

SSIS control package

For this we create a package that uses a for each loop task to call an execute package task for all packages in the folder.

The first task is to create an SSIS package with a Foreach Loop container that will loop round the packages in the folder, setting a variable called "PackageToRun" to the file name for each package.

  1. Load Business Intelligence Development Studio and start a SSIS project.
  2. Create a new package.
  3. Add a Foreach Loop container to the package.
  4. Right-click on the Foreach Loop container and select Edit.
  5. Click on Collection.
  6. Set the Enumerator to Foreach File Enumerator.
  7. In the Enumerator configuration:
    1. Set Folder to "d:\TestPackages\"
    2. Set Files to "t1*.dtsx"
    3. Under Retrieve file name select Fully qualified.
  8. Click OK.

  1. Click on Variable Mappings.
  2. Click on the Variable drop-down list and select New Variable.
  3. Set Name to PackageToRun.
  4. Click OK.

Next, we need to add the Execute Package task to the Foreach Loop container so that this task will be executed for each package that we wish to run. We then set the variable value to be the name of the package to be executed by the ExecutePackage task.

  1. Drag an Execute Package task into the Foreach Loop container.
  2. Right-click on the Execute Package task and select Edit.
  3. Select Package.
  4. Set Location to File system

  1. Click on the Connection drop-down list and select <New connection…>.
  2. Set the File to an existing package.

  1. Click OK to save the connection.
  2. Click OK to complete the Execute Package task configuration.

Finally, we configure the connection to use the variable package name:

  1. Right-click on the connection and select Properties.
  2. In the Properties window change the name to PackageToRunConnection.

Note: this will automatically update the connection name in the Execute Package Task.

  1. Select Expressions and add a new expression.
  2. In the property drop-down list select ConnectionString.
  3. Click on the Expression Editor button.
  4. From the Variables tree drag @[User::PackageToRun] into the Expression window.

  1. Click OK twice, to save the expression.

The package should now look like this:

If you now run the package, it will execute each package in the folder with the correct filename mask.

Stored Procedure

This involves the following steps:

  1. Retrieve a list of the package names to run from the folder
  2. Loop through each package name
  3. Execute each package

First of all, we need to get a list of all packages in the folder. The following code executes the DOS command dir /B d:\TestPackages\ t1*.dtsx and places the output in the temporary table #dir. The DOS command gets a list of the filenames in the folder which satisfy the filemask.

Create table #dir (Filename varchar(1000))
Insert #dir
Exec master..xp_cmdshell 'dir /B d:\TestPackages\ t1*.dtsx'
delete #dir where Filename is null or Filename like '%not found%'

The following is a useful technique for looping through values in a table and here we use it to loop through the filenames in #dir:

Declare @Filename varchar(1000)
Select @Filename = ''
While @Filename < (select max(Filename) from #dir)
Begin
   Select @Filename = min(Filename) from #dir where Filename > @Filename
end

Now we just need to create the command to execute the package:

Declare @cmd varchar(1000)
select @cmd = 'dtexec /F " d:\TestPackages\' + @FileName + '''
exec master..xp_cmdshell @cmd

This takes the package name and uses the dtexec command to execute it. Putting this all together, we get:

Declare @Filename varchar(1000)

Declare @cmd varchar(1000)

Create table #dir (Filename varchar(1000))

Insert #dir

Exec master..xp_cmdshell 'dir /B d:\TestPackages\t1*.dtsx'

delete #dir where Filename is null or Filename like '%not found%'

Select @Filename = ''

While @Filename < (select max(Filename) from #dir)

Begin

   Select @Filename =  min(Filename) from #dir where Filename > @Filename

   select  @cmd =  'dtexec /F "d:\TestPackages\' + @Filename + '"'

   exec master..xp_cmdshell @cmd

end

drop table #dir

The above completes the code needed for the requirement but to simplify the implementation I like to keep a utility stored procedure which calls another stored procedure for each file (SSIS package) in a folder. The following utility stored procedure, ProcessAllFilesInDir, uses the above techniques to obtain a list of files from a folder and calls another stored procedure for each one. The file path and mask are input as parameters together with the @ProcSP parameter that defines the stored procedure to be called for each file (in this case this will be the ExecutePackage procedure that executes each SSIS package).

Create procedure [dbo].[s_ProcessAllFilesInDir]

@FilePath varchar(1000) ,

@FileMaskvarchar(100) ,

@ProcSp varchar(128)

as

   set nocount on

  

declare @File varchar(128) ,

      @MaxFile varchar(128) ,

      @cmd     varchar(2000)

  

   create table #Dir (s varchar(8000))

  

   select @cmd = 'dir /B ' + @FilePath + @FileMask

   insert #Dir exec master..xp_cmdshell @cmd

  

   delete #Dir where s is null or s like '%not found%'

  

   select @File = '', @MaxFile = max(s) from #Dir

   while @File < @MaxFile

   begin

      select @File = min(s) from #Dir where s > @File

     

      select @cmd = @ProcSp + ' ''' + @FilePath + ''' , ''' + @File + ''''

     

      exec (@cmd)

     

   end

   drop table #Dir

go

With this, the only coding needed is to create the procedure to execute the package and to call it. The ExecutePackage procedure is as follows:

Create procedure [dbo].[s_ExecutePackage]

@FilePath   varchar(1000) ,

@Filename   varchar(128)

as

Declare @cmd varchar(1000)

   select @cmd =  'dtexec /F "' + @FilePath + @Filename + '"'

   exec master..xp_cmdshell @cmd

go

And the call:

Exec [dbo].[s_ProcessAllFilesInDir]

   @FilePath = 'd:\TestPackages\' ,

   @FileMask = 't1*.dtsx' ,

   @ProcSp = '[dbo].[s_ExecutePackage]'

DOS batch file

The DOS command required is dtexec /F "d:\TestPackages\mypackagename" as above. We can make use of a DOS For loop to execute this command for all files in a folder:

for %1 in (d:\TestPackages\t1*.dtsx) do dtexec /F "%1"

This uses the for loop construct:

  • (d:\TestPackages\t1*.dtsx) returns a list of the files which meet the path (similar to a dir command).
  • For %1 in (d:\TestPackages\t1*.dtsx) loops through the file list setting the variable %1 to the filename.
  • Do dtexec /F % executes dtexec /F with each filename in turn.

Note that you can place this command in a bat file to execute.

Summary

We have introduced three different methods of executing all packages in a folder. The methods are of differing complexity, the SSIS control package being by far the most difficult to code, and the DOS command by far the simplest.

There are a few things missing from the solutions presented which would (or should) prevent them from being used in a production system. At a minimum this should include logging and error handling. The start and end time of each package execution should be logged by the control process and also errors should be detected and logged.

At first sight, it may seem that the DOS command is the one that we should be using, but this method loses a lot of flexibility. There is much you can do with batch files, but not many people have skills in writing such code beyond very simple commands, so you can quickly end up with an un-maintainable system.

The stored procedure solution gives a lot of flexibility – it is easy to add logging and pre/post processing to the process. However, it does require an installation of SQL Server. Although this process would usually be run via SQL Server, it may sometimes need to be run on a client machine (or not involve SQL Server at all); this is quite difficult with a stored procedure as it is server based whereas the other two solutions are client applications.

The SSIS solution keeps the process within SSIS, so you can take advantage of the integration of package processing, e.g. inheritance of properties from the calling package, development and debugging of packages, within the development studio.

Overall, none of these solutions is very difficult to implement, so the decision which to use will probably depend on the experience of the developer, how much flexibility is required, and the environment in which it is to be run.


© Simple-Talk.com