Hello all,
I just wanted to bring to light a recent discovery that I cannot find documented anywhere in MSDN or on the web. Perhaps the well-read simple-talk audience can clarify if this is a bug, "feature", or my own user error.
In an SSIS 2005 package I am reading from five flat files, each being a "partition" of a larger ordered data set. All I want to do is read in all files and join the unique values of their key column to a key column in an Oracle table. Since the contents of the files are sorted, I tried used several Merge transformations to put the files into one sorted buffer, and then fed that to the Aggregate transformation and grouped by the key column. The problem is, the output of the Aggregate transformation is not sorted even though the input is. Has anyone else experienced this? Why even allow the isSorted option on output to be set?
My solution is to use one Union All instead of multiple Merges and then a Sort with the "remove duplicate keys" turned on. This worked and gave a 5x performance boost.
Ryan