This is the tenth and final entry in the 10-Series Blog post in regards to Software-Plus-Services: Application Instrumentation, Stress Testing and Production Management.    

Application Monitoring Enables Proactive Efficient Management

Proactive application monitoring provides the insight needed to enable efficient production management. When implementing the monitoring elements discussed in previous posts, production operations staff, whether internal or external, will now have the visibility necessary to manage the Software-plus-Services application at a lower cost and higher value to end customers. The next step is using this proactive intelligence as a selling tool by providing some level of transparency and quality assurance to customers and prospects alike.

When preparing for the launch of an application, it is crucial to understand how it will likely behave under real-world user load. Our experience indicates the best way to plan for growth and avoid surprises is to simulate the anticipated user load with a realistic and well-instrumented load test. Developing a test scenario for each of the application's anticipated user types ensures that all application functions are exercised. Furthermore, by simultaneously loading the application with multiple user types, we can expose complex interactions inside the application such as deadlocks, race conditions, and queue backlogs. Real-world application usage is rarely characterized by searching for the same keyword or logging in as a single user thousands of times. It is extremely important to introduce unique, dynamic user input to the load testing to avoid falsely positive results due to caching of sessions, query results, etc.

Effective load testing, like ongoing operational monitoring, should correlate end-user experience with internal application performance metrics. The load testing then serves as both a direct measurement of performance as well as a means to exercise the application while observing its behavior directly though database, application, and OS metrics. The second question after "What is the capacity of the application?" is always "Where did the bottleneck occur?" With a properly instrumented environment and well-designed load test, we are able to answer both questions.

Stress testing should not be a standalone effort. In an ideal world, it will take advantage of all of the instrumentation previously created and leverage the existing synthetic user scenarios. With System Shepherd®, WebWalk^TM, and StressWalk^TM all of the components tie together to provide maximum reusability. Additionally, data from the stress testing runs should be available for analysis side-by-side with live production data thus enabling correlation and continued enhancement of the stress testing approach.

A holistic view of the application delivery infrastructure is not complete without looking at the performance and availability of the network and storage infrastructure. Access to this data is dependent upon the hosting strategy being employed. Administrator level access to the network and storage infrastructure provides a wealth of data. This data may be collected via SNMP with System Shepherd® agents, or just about any other monitoring solution. If access to the network and storage infrastructure is limited, data collection will require a bit of creativity. One method is to correlate data from ping and port monitoring from various points within the architecture with individual server or virtual machine networking statistics. This provides a view of network behavior. On the storage side, individual server (or virtual machine) physical storage metrics can be used to obtain a partial picture of storage performance.

Real-Time Application Analysis

Once the above recommendations are implemented, data will then be available for analysis. The key is to be able to use the data to improve application delivery from the perspective of the end-user and the business. Regardless of the tools or services used to collect the data, it needs to be accessible in one place to maximize the value of the monitoring solution. The best practice in this area is to provide the technical teams with proactive alerting. Proactive alerting requires configurable alarm and escalation rules and the ability to perform real-time or near real-time analysis on the data in both ad-hoc and standard views.

 For most applications created in the Software-plus-Services paradigm, agent-based monitoring with very low resource consumption is the most reliable and extensible approach available to enable efficient instrumentation. Further, the most efficient instrumentation approach is to use a scalable template-based architecture such as used by System Shepherd®. This method allows for the creation of a class of infrastructure monitoring templates, which may be reused many times over. This methodology provides the ability to capture "one-off" metrics on specific devices while allowing configuration inheritance from the master template to take care of the rest. This will save significant time and resources allowing for a seamless and consistent migration from pre-production environments into production. When looking at designing application backend components, it is important to take instrumentation into account from the onset of development.

A lot of valuable data is available in standard monitoring configuration templates for most products via WMI or PERFLIB including MSMQ and .NET application component stack metrics. Many of the metrics exposed by WMI are counters which can be useful in their raw format for troubleshooting longevity related problems. However, they are often most valuable when converted to rate based metrics which should be inherent in the monitoring solution.

Many of the most important metrics are not available through standard WMI counters. Some application developers choose to create custom WMI counters. The article "Powerful Instrumentation Options in .NET Let You Build Manageable Apps with Confidence" serves as an excellent starting point for creating application specific counters. These counters can then be monitored via custom templates and thresholds within the monitoring solution. An alternate approach, which some development teams find easier to implement, is to log performance metrics and application events to a file. The monitoring solution should also be able to watch these files on a continuous basis and translate the metrics and events into rates using frequency based log analysis. What custom parameters of an application should be monitored?

This is the second entry in a 10-Series Blog post every Wednesday in regards to Software-Plus-Services: Application Instrumentation, Stress Testing and Production Management.

Application instrumentation is the means to monitor and report the performance of an application. This instrumentation is done by using a combination of internal and external observation methods. Internal counters can be provided by the underlying operating system and by the application layer itself. External counters are typically used to measure end-user experience and other external environmental metrics. It is critical to combine internal and external measures into a single repository to maximize the business value of monitoring. Very rarely do organizations look at the operational and business needs during the initial phases of application instrumentation instead they look primarily at the needs of the software engineering team. This is a highly inefficient approach. Based on experience with a number of applications at various stages of development and deployment, a best practices approach has been developed that satisfies all three sets of requirements. A standard method for applying instrumentation allows engineering and operations to "speak" the same support language. This approach allows these groups to work in tandem to proactively resolve service problems much faster and provide better value to the business.

WHAT IS A SOFTWARE-PLUS-SERVICES APPLICATION?

Software-plus-Services should not be confused as another name for SaaS. A SaaS (Software as a Service) application is simply an application delivered as a service; it does not specify what type of application architecture is involved. A Software-plus-Services application combines software that is running in multiple locations together in a unified value proposition for users. A Software-plus-Services application may be sold and delivered as a product, as a service, or as a combination of both. Examples of Software-plus-Services applications include:

• Applications that run on-premise but integrate services delivered from the cloud such as a trading application which pulls real-time data feeds from the stock exchanges;
• Applications that run in a hosted environment over the internet, but integrate back into the IT environments of their customers, such as hosted CRM systems that integrate into their customers' on-premise Active Directory or sales systems;
• Applications that integrate a user experience into clients outside the web browser, such as mobile devices, the Windows desktop or Microsoft Office;
• Combinations of the above.

Regardless of the architecture of your Software-plus-Services application, to deploy and manage it effectively you must be able to monitor its performance. This means you must have the ability to know how any given component of the application is performing at any time, regardless of whether that component is deployed on a customer desktop, on a customer server, on a hosted server, or in a remotely supplied web service. This 10 series blog post will provide you with guidance on how to effectively implement performance monitoring for your Software-plus-Services applications.