Click here to monitor SSC
  • Av rating:
  • Total votes: 2
  • Total comments: 2
Roger Jennings

Testing the StreamInsight Service for Windows Azure

06 February 2013

Getting 'up to speed' with StreamInsight is easier if you take the time to run it and test it out. Roger Jennings expains how to be up and running in nine simple steps, once you've sorted out the prerequisites

In my previous article StreamInsight Service for Windows Azure about the “Project Austin” StreamInsight service, I explained how to provision it on Windows Azure with Visual Studio 2010 or 2012 and the downloadable AustinCtpSample solution. StreamInsight is a Complex event processing (CEP) application that tracks streams of information that combines data from several sources in order to identify meaningful events and respond to them rapidly.

In this article, I’ll be showing you how to test the StreamInsight Service for Windows Azure CTP with the SampleApplication and EventSourceSimulator projects and use the graphical Event Flow Debugger. To do this, you will need the following: (see StreamInsight Service for Windows Azure).

  • Visual Web Developer 2010 Express or Visual Studio Express 2012 for Web or higher version
  • Windows Azure SDK v1.7 (June 2012), which includes the Visual Studio Tools for Windows Azure
  • A Windows Azure trial or paid subscription, preferably dedicated to StreamInsight
  • An invitation to the StreamInsight Service for Windows Azure CTP, the source code and binaries for which you must download and install
  • Completion of all sections of Part 1 except “Deleting a Service Instance Deployment”

Run the AustinCtpSample Solution’s SampleApplication Project

The SampleApplication project’s Program.cs file includes code to connect to and manage a named Project “Austin” instance, and define, deploy and run queries in that instance. There’s also code to deploy (and, optionally, redeploy) the following .NET assemblies created by the SampleApplication project to that instance:

  • StreamInsight.Samples.Austin.EventTypes.dll
  • StreamInsight.Samples.Austin.AzureTableSink.dll

The AzureTableSink class represents an Observer that writes result events into a Windows Azure table in the storage account you created in Part 1. The next section provides more information about event streams and sinks.

To configure and start the SampleApplication program, do the following:

  1. In Visual Studio’s Solution Explorer, right-click the SampleApplication node and choose Set as Startup Project.
  2. Expand the SampleApplication project node, and double-click the App.config node to open the document in the editor window. Replace the AccountName placeholder with the storage account name, oakaustin for this example, the AccountKey placeholder with the value copied from the portal, and the austinDnsPrefix placeholder in three places (see Figure 1).

Replace Accountname, AccountKey and austinDnsPrefix in the three places shown here

Figure 1. Replace Accountname, AccountKey and austinDnsPrefix in the three places shown here.

  1. Double-click the Program.cs item to open the C# code in the editor window, expand the first region and replace austinDnsPrefix in lines 118 and 125 as shown in Figure 2.

Replace austinDnsPrefix on lines 118 and 125 with the Cloud Service name, oakaustin for this example

Figure 2. Replace austinDnsPrefix on lines 118 and 125 with the Cloud Service name, oakaustin for this example.

  1. Press F5 to deploy input and output adapters and a query, and start the query, which displays progress in a console window (see Figure 3):

Type y when asked if you want to (re)deploy the user assemblies by the SampleApplication’s console

Figure 3. Type y when asked if you want to (re)deploy the user assemblies by the SampleApplication’s console.

At this point, the SampleApplication is running in the StreamInsight service and waiting for a stream of events (Observable sequences) to arrive at the ingress endpoint. .NET 4.0’s Reactive Extensions (Rx) library enables the composing of asynchronous and event-based programs using Observable sequences and LINQ-style query operators. To learn more about the .NET Reactive Extensions library, see MSDN’s Reactive Extensions topic and subtopics.

Processing Emulated Events withStream Insight Queries

C# code in the SampleApplication project’s Program.cs file deploys an event sink, AzureTableSink, that passes httpSource events through to a Windows Azure HttpMessages Table. Queries generate average values that populate HighDeviceAverages, DeviceAverages and LowDeviceAverages tables (see Figure 4.)

An event flow diagram for the SampleApplication project

Figure 4. An event flow diagram for the SampleApplication project. Processing events creates the four Windows Azure tables if they don’t already exist; queries add entities to them depending on average values.

The Main program in the SampleApplication project’s Program.cs file defines the LINQ averages and highaverages temporal aggregate queries with the following code:

var averages = from e in deviceStream
               group e by e.DeviceId into gs
               from win in gs.TumblingWindow(TimeSpan.FromSeconds(5))
               select new DevicePayload
               {
                   DeviceId = gs.Key,
                   Value = win.Avg(e => e.Value),
               };var highAverages = averages
                   .Where(e => e.Value > 10);

The EventSourceSimulator project generates an external event stream to validate the service deployment and its interaction with the SampleApplication project. To send one event by the HTTP GET and 60 by HTTP POST methods, follow these steps:

  1. Reprovision the StreamInsight instance and set up the SampleApplication, if you deleted the instance in the preceding section.

Note: Consider changing the IngressProtocol from https to http in line 46 of the ServiceProvisioningproject’s app.config file to minimize potential certificate issues.

  1. If you chose http in Part 1’s step 1, replace https with http in lines 36 and 37 of the EventSourceSimulator project’s Program.cs file (see Figure 5.)

The EventSourceSimulator’s protocol must match that of the provisioned StreamInsight instance

Figure 5. The EventSourceSimulator’s protocol must match that of the provisioned StreamInsight instance.

  1. Press F5 to build and run the EventSourceSimulator project, which sends a total of 61 events to the SampleApplication in about one minute (see Figure 6.)

The EventSourceSimulator project sends a single event by the HTTP POST method

Figure 6. The EventSourceSimulator project sends a single event by the HTTP POST method followed by 30 events each with sequentially increasing values from two simulated devices (X and Y) by the HTTP POST method. The POST payload is in JSON format.

  1. Launch a Windows Azure Storage utility, such as Cerebrata Storage Studio for this example, connect to the storage account and expand the tables list to expose the four tables created and populated by the simulated events (see Figure 7.)

 The entities of theHighDeviceAverage table, shown here in Cerebrata Storage Explorer, alternate between devices X and Y

Figure 7. The entities of theHighDeviceAverage table, shown here in Cerebrata Storage Explorer, alternate between devices X and Y.

Instance-specific tables prefixed siXoakaustinX provide detailed logs of activities involved with deploying and dismantling application objects, such as theHttpEventPipeline (see Figure 8).

Entities in this siX<em>oakaustin</em>XIngressLog table display events for deletion and creation of the HttpEventPipeline for processing events

Figure 8. Entities in this siXoakaustinXIngressLog table display events for deletion and creation of the HttpEventPipeline for processing events.

Enabling Verbose Windows Azure Diagnostics and Monitoring Service Activity

The Austin August CTP enabled Windows Azure Diagnostics (WAD) for tracing StreamInsight Service execution. By default, only the Minimal diagnostics level is enabled by default in the Management Portal’s Cloud Service Dashboard’s Configure page for the instance. To increase diagnostics granularity, click the Verbose and Save buttons (see Figure 9).

Increase tracing granularity

Figure 9. Increase tracing granularity by changing the Monitoring Level from Minimal to Verbose. The warning flag advises that you might incur storage charges. The August CTP enables Windows Azure Diagnostics and adds default Diagnostics Connection Strings.

Verbose diagnostics add Windows Azure tables named WADGUIDPT##XRTable to the storage utility’s display (see Figure 10.)

Enabling Verbose tracing adds detailed WAD tables to your storage account

Figure 10. Enabling Verbose tracing adds detailed WAD tables to your storage account. Table lifetime is fixed at 10 days to prevent excessive storage costs.

Configure and Run the EventFlowDebugger Program

The Austin CTP binaries include a graphical StreamInsight Event Flow Debugger program (C:\Program Files\Microsoft StreamInsight 2.1\Tools\EventFlowDebugger.exe). MSDN’s Using the StreamInsight Event Flow Debugger topic provides details for using the program.

The Debugger has an EventFlowDebugger.exe.configconfiguration file in the same folder for on-premises StreamInsight instances. To reconfigure for Windows Azure deployment and run the debugger, follow these steps:

  1. Rename EventFlowDebugger.exe.configto EventFlowDebugger.exe.orig.
  2. In Solution Explorer, expand the Other resources node and double click the EventFlowDebugger.exe.configitem to open it in the editor window.
  3. Replace two instances of austinDnsPrefix with your service name, oakaustin for this example (see Figure 11.)

Configure the EventFlowDebugger for use with instances deployed to Windows Azure

Figure 11. Configure the EventFlowDebugger for use with instances deployed to Windows Azure.

  1. Chose File, Save EventFlowDebugger.exe.configAs to open the Save File As dialog, navigate to the C:\Program Files\Microsoft StreamInsight 2.1\Tools\ folder and click Save.
  2. Choose Start, All Programs, Microsoft StreamInsight v2.1 x64 (en), StreamInsightEventFlowDebugger to start the debugger.
  3. Choose File, Connect to Server to open the eponymous dialog, select Options to expand it, select the Use Predefined Endpoint option and accept the default MyAustinEndpoint in the list (see Figure 12.)

Specify the connection to the Windows Azure StreamInsight instance

Figure 12. Specify the connection to the Windows Azure StreamInsight instance.

  1. Click OK to open the debugger window and expand the Applications and Processes nodes to display running queries. Right-click a StreamableBinding_1 query item and choose Display Query to open a query graph (see Figure 13.)

 Monitor activity in the Stream Insight Event Flow Debugger

Figure 13. Monitor activity, as well as debug applications and instances in the Stream Insight Event Flow Debugger configured for hosting by Windows Azure Cloud Services. Selecting a query enables the Start Recording Events button; however, you can’t record events of a remote StreamInsight server. Clicking it creates a local *.etf file that contains code to define the graph but no event data.

  1. Right-click the Http Messages Passthrough Process node’s StreamableBinding_1 query item and choose ‘Show Diagnostics to display details of the first emulated event, which used the HTTP GET method (see Figure 14.)

The initial event streamed with the HTTP GET method generates two outgoing events

Figure 14. The initial event streamed with the HTTP GET method generates two outgoing events.

  1. Right-click a ‘High Averages Process node’s StreamableBinding_1 query item and choosehow Diagnosticsto display details of the next 60 emulated events, which used the HTTP POST method (see Figure 15.)

The 60 events streamed with the HTTP POST method generate 164 outgoing events

Figure 15. The 60 events streamed with the HTTP POST method generate 164 outgoing events.

If you want to take advantage of the Event Flow Debugger’s event recording, replay, single-step and breakpoint features, you must install the local server bits with StreamInsight_AustinAugustCTP_x64.msi or StreamInsight_AustinAugustCTP_x86.msi from the August CTP distribution, which is compatible with StreamInsightClient_AustinAugustCTP_x64 or StreamInsightClient_AustinAugustCTP_x86, respectively.

Conclusion

StreamInsight is rapidly maturing with closely spaced CTPs and the latest update for the StreamInsight Service for Windows Azure promises a larger potential customer base. There’s a serious shortage of developers with on-premises or cloud-based StreamInsight skills, as well as for the newly christened HDInsight (Hadoop) Service for Windows Azure and Windows Server. Leverage your .NET chops by learning to write and test LINQ queries for processing fast-moving streams of financial or sensor data to give your career a healthy boost.

Roger Jennings

Author profile:

Roger Jennings is a data-oriented .NET developer and writer, a Windows Azure MVP, a member of Azure Insiders, the principal consultant of OakLeaf Systems, and curator of the OakLeaf Systems blog. He's also the author of 30+ books on the Windows Azure Platform, Microsoft operating systems, databases, .NET data access, Web services and InfoPath 2003. Forbes Magazine ranked him seventh in a "Who Are The Top 20 Influencers in Big Data?" article of February 3, 2012.

Search for other articles by Roger Jennings

Rate this article:   Avg rating: from a total of 2 votes.


Poor

OK

Good

Great

Must read
Have Your Say
Do you have an opinion on this article? Then add your comment below:
You must be logged in to post to this forum

Click here to log in.


Subject: StreamInsight in 2014 and beyond
Posted by: danvanderboom (view profile)
Posted on: Sunday, April 13, 2014 at 7:20 PM
Message: Thanks for all the great information on running StreamInsight in Azure! This has been a big help.

What's new with StreamInsight? I didn't hear of any sessions or announcements at Build, and new articles and blogs on the subject seem to be sparse.

Am I missing something? This technology (plus Reactive Extensions and LINQ along with it) is way too promising and important not to keep moving forward on!

Subject: StreamInsight in 2014 and beyond
Posted by: danvanderboom (view profile)
Posted on: Tuesday, April 15, 2014 at 7:05 PM
Message: Thanks for all the great information on running StreamInsight in Azure! This has been a big help.

What's new with StreamInsight? I didn't hear of any sessions or announcements at Build, and new articles and blogs on the subject seem to be sparse.

Am I missing something? This technology (plus Reactive Extensions and LINQ along with it) is way too promising and important not to keep moving forward on!

 

Top Rated

Data Science Laboratory System – Object-Oriented Databases
 Object-Oriented Databases (OOD) avoid the object-relational impedence mismatch altogether by tightly... Read more...

Tales from a Cloud Software Firm
 Following on from a discussion about how people are using the cloud, the Simple-Talk Editorial Team sat... Read more...

Data Science Laboratory System – Document Store Databases
 A Document Store Database (DSD) is similar to a Relational Database Management system with the... Read more...

Data Science Laboratory System - Instrumentation
 It is sensible to check the performance of different solutions to data analysis in 'lab' conditions.... Read more...

Testing the StreamInsight Service for Windows Azure
 Getting 'up to speed' with StreamInsight is easier if you take the time to run it and test it out.... Read more...

Most Viewed

Windows Azure Virtual Machine: A look at Windows Azure IaaS Offerings (Part 2)
 We continue our introduction of the Azure IaaS by discussing how images and disks are used in the Azure... Read more...

PHPFog and Pagoda Box: A Look at PHP Platforms
 Cloud platforms such as Heroku, AppEngine, PHPFog and Pagoda Box are ideal for companies who just want... Read more...

An Introduction to Windows Azure BLOB Storage
 Azure BLOB storage is persistent Cloud data storage that serves a variety of purposes. Mike Wood shows... Read more...

Managing session state in Windows Azure: What are the options?
 Because you can't maintain session state for ASP.NET applications in Azure using the default in-process... Read more...

Creating a custom Login page for federated authentication with Windows Azure ACS
 Windows Azure Acess Control Service (ACS) provides a way of authenticating users who need to access web... Read more...

Why Join

Over 400,000 Microsoft professionals subscribe to the Simple-Talk technical journal. Join today, it's fast, simple, free and secure.