When preparing for the launch of an application, it is crucial to understand how it will likely behave under real-world user load. Our experience indicates the best way to plan for growth and avoid surprises is to simulate the anticipated user load with a realistic and well-instrumented load test. Developing a test scenario for each of the application's anticipated user types ensures that all application functions are exercised. Furthermore, by simultaneously loading the application with multiple user types, we can expose complex interactions inside the application such as deadlocks, race conditions, and queue backlogs. Real-world application usage is rarely characterized by searching for the same keyword or logging in as a single user thousands of times. It is extremely important to introduce unique, dynamic user input to the load testing to avoid falsely positive results due to caching of sessions, query results, etc.

Effective load testing, like ongoing operational monitoring, should correlate end-user experience with internal application performance metrics. The load testing then serves as both a direct measurement of performance as well as a means to exercise the application while observing its behavior directly though database, application, and OS metrics. The second question after "What is the capacity of the application?" is always "Where did the bottleneck occur?" With a properly instrumented environment and well-designed load test, we are able to answer both questions.

Stress testing should not be a standalone effort. In an ideal world, it will take advantage of all of the instrumentation previously created and leverage the existing synthetic user scenarios. With System Shepherd®, WebWalk^TM, and StressWalk^TM all of the components tie together to provide maximum reusability. Additionally, data from the stress testing runs should be available for analysis side-by-side with live production data thus enabling correlation and continued enhancement of the stress testing approach.

A holistic view of the application delivery infrastructure is not complete without looking at the performance and availability of the network and storage infrastructure. Access to this data is dependent upon the hosting strategy being employed. Administrator level access to the network and storage infrastructure provides a wealth of data. This data may be collected via SNMP with System Shepherd® agents, or just about any other monitoring solution. If access to the network and storage infrastructure is limited, data collection will require a bit of creativity. One method is to correlate data from ping and port monitoring from various points within the architecture with individual server or virtual machine networking statistics. This provides a view of network behavior. On the storage side, individual server (or virtual machine) physical storage metrics can be used to obtain a partial picture of storage performance.

Real-Time Application Analysis

Once the above recommendations are implemented, data will then be available for analysis. The key is to be able to use the data to improve application delivery from the perspective of the end-user and the business. Regardless of the tools or services used to collect the data, it needs to be accessible in one place to maximize the value of the monitoring solution. The best practice in this area is to provide the technical teams with proactive alerting. Proactive alerting requires configurable alarm and escalation rules and the ability to perform real-time or near real-time analysis on the data in both ad-hoc and standard views.

Some of the most valuable business relevant metrics reside in the front end components of an application served by IIS or other web server components. As with the other layers of the application stack, the standard metrics available through WMI provide a good starting point. However, these metrics do not give visibility into the end-user experience.

The recommended best practice is to combine this information with synthetic user transaction monitoring via a user transaction simulation service such as WebWalk^TM. This layer of monitoring allows for proactive tracking of the end-user experience. Alert thresholds should be set for metrics such as content/functionality validation, transaction execution times, and rates of change for transaction execution times. This component of external monitoring should be replicated behind the load balancer within the application environment to provide visibility into the behavior originating from individual servers or virtual machines. Combining external synthetic transaction monitoring with internal synthetic transaction monitoring enables the operations/delivery organization or outsourced provider to be proactive in managing the application.  

In this tier, the value of the data appearing in access logs should also be monitored. This can be achieved by customizing the logging parameters to expose the user or client associated with each request and the amount of time it took to return the request to the browser. Frequency based log analysis can then be employed to capture access frequency and average/max/min response time metrics associated with the requests on an aggregate basis. This data is highly valuable for isolating application response time issues. Additionally, the raw data can be used as part of the overall data set to identify resource utilization by an individual user or client.

 This is the forth entry in a 10-Series Blog posts every week in regards to Software-Plus-Services: Application Instrumentation, Stress Testing and Production Management.  

 The data access layer is most likely composed of a SQL Server database and one or more in-memory caching services, and external web services calls. Basic database monitoring is fairly straightforward and handled by most products "out of the box". However, when building an application with instrumentation in mind, it is important to consider what additional information the monitoring platform should capture from either the database or external data service calls. It is recommended the following incremental metrics be used:

• Business relevant metrics such as revenue, user conversion, transaction volume, etc. These metrics enable the use of APM data for business analytics. This information is key to supporting continuous performance improvement at the business level.
• Resource utilization by user or client enables costing at the customer or user level as previously discussed.
• Aggregate application usage metrics help to facilitate both long-term capacity planning and feature utilization pattern analysis.
• External service call frequency and duration for data access by user or client provides valuable insight into the delivery characteristics of third party data and services providers.
• Data cache utilization and performance provides the visibility necessary to determine if the cache client software is effectively utilizing the data cache in place.

Please let me know your thoughts by leaving a comment. Please check back next Tuesday, July 14th for the next blog in this series on front end user interface and web services.

 For most applications created in the Software-plus-Services paradigm, agent-based monitoring with very low resource consumption is the most reliable and extensible approach available to enable efficient instrumentation. Further, the most efficient instrumentation approach is to use a scalable template-based architecture such as used by System Shepherd®. This method allows for the creation of a class of infrastructure monitoring templates, which may be reused many times over. This methodology provides the ability to capture "one-off" metrics on specific devices while allowing configuration inheritance from the master template to take care of the rest. This will save significant time and resources allowing for a seamless and consistent migration from pre-production environments into production. When looking at designing application backend components, it is important to take instrumentation into account from the onset of development.

A lot of valuable data is available in standard monitoring configuration templates for most products via WMI or PERFLIB including MSMQ and .NET application component stack metrics. Many of the metrics exposed by WMI are counters which can be useful in their raw format for troubleshooting longevity related problems. However, they are often most valuable when converted to rate based metrics which should be inherent in the monitoring solution.

Many of the most important metrics are not available through standard WMI counters. Some application developers choose to create custom WMI counters. The article "Powerful Instrumentation Options in .NET Let You Build Manageable Apps with Confidence" serves as an excellent starting point for creating application specific counters. These counters can then be monitored via custom templates and thresholds within the monitoring solution. An alternate approach, which some development teams find easier to implement, is to log performance metrics and application events to a file. The monitoring solution should also be able to watch these files on a continuous basis and translate the metrics and events into rates using frequency based log analysis. What custom parameters of an application should be monitored?

This is the second entry in a 10-Series Blog post every Wednesday in regards to Software-Plus-Services: Application Instrumentation, Stress Testing and Production Management.

Application instrumentation is the means to monitor and report the performance of an application. This instrumentation is done by using a combination of internal and external observation methods. Internal counters can be provided by the underlying operating system and by the application layer itself. External counters are typically used to measure end-user experience and other external environmental metrics. It is critical to combine internal and external measures into a single repository to maximize the business value of monitoring. Very rarely do organizations look at the operational and business needs during the initial phases of application instrumentation instead they look primarily at the needs of the software engineering team. This is a highly inefficient approach. Based on experience with a number of applications at various stages of development and deployment, a best practices approach has been developed that satisfies all three sets of requirements. A standard method for applying instrumentation allows engineering and operations to "speak" the same support language. This approach allows these groups to work in tandem to proactively resolve service problems much faster and provide better value to the business.