Organizations worldwide rely on Absolute Performance comprehensive application management and services to ensure that critical applications and data services are performing as expected.
At Absolute Performance, we're experts in monitoring, managing, and optimizing BizFlow and its underlying infrastructure to help you make your workflow and business process management objectives a reality. We can help you address common challenges to performance that affect the application, database, and supporting network infrastructure.
BizFlow Opportunities
Because BizFlow provides an enterprise level workflow that links forms to multiple relational databases, applications, and legacy systems, there are many areas within the technology stack that can affect end-user satisfaction. At Absolute Performance we understand what metrics to monitor, manage, and repair before it affects the user.
Why managed services for BizFlow application performance?
Absolute Performance's ongoing managed services ensures 24x7 operational support with maximum end-user experience so that you can focus on your business priorities.
At Absolute Performance we're experts in monitoring, managing and optimizing BizFlow and its underlying infrastructure to help you make your BPM and workflow objectives a reality. We can help you address common challenges such as event log management, open/close/wait connections and SQL deadlock issues.
Our outsourced BizFlow Management Services include:
- 24x7 response to alerts generated by the System Shepherd® platform
- IT best practices - incident, problem, and change management
- Rapid troubleshooting and resolution of performance issues throughout the BizFlow stack
- Installation and testing of relevant patches and updates
- Capacity planning and recommendations
- Proactive performance analysis and tuning of BizFlow and its underlying infrastructure
- Proactive maintenance of BizFlow application
- Comprehensive database tuning and maintenance optimization
- Web server analysis and tuning
- Application server analysis and tuning
- Change activation and validation in production environment
- Tracking and reporting on performance and availability SLAs to business owners
To optimize the performance, availability, and scalability of your HandySoft BizFlow Business Process Management (BPM) solution, you need deep visibility into the application and its infrastructure.
At Absolute Performance, we've customized our System Shepherd® operations management platform to deliver the detailed data and analysis tools to manage the complex BizFlow environment:
Network Network device basics, including:
| Operating System OS basics and internals, incl.:
| BizFlow Database Database basics and internals, including:
Detailed statistics, including:
|
The System Shepherd® WebWalk™ feature can be customized to confirm that BizFlow services are available and performing as you intended.
WebWalk™ uses synthetic transactions to measure performance from the end-user perspective. IT professionals can then cross-correlate end-user experience data with systems data to pinpoint problems.
With WebWalk™ you can:
- Simulate end-user experience from any location
- Monitor and validate multiple links or steps in your process
- Time each page access or transaction response and validate content
- Receive immediate notification when performance falls below established parameters
Companies around the globe rely on HandySoft’s BizFlow® solution to meet their business process needs. However, in complex deployments, the BizFlow environment itself can pose significant monitoring and management challenges for the IT professionals responsible for its care and feeding.
Over the years, Absolute Performance has worked with a number of clients to optimize their business-critical BizFlow environments. During these engagements, we’ve noticed three common early indicators of problems that can lead to poor performance or outages in complex BizFlow deployments:
- Log Analysis
- Blocking Processes
- Queue Analysis
Keeping tabs on these three "hot buttons" is critical to the health of your BizFlow application and its underlying infrastructure. In this article, we’ll focus on analyzing and cleaning BizFlow error logs.
The Importance of Vigilant BizFlow Log Analysis
BizFlow log analysis is important for two key reasons:
- The BizFlow application generates a massive amount of log entries, and ...
- Many BizFlow performance problems initially present themselves in the log files
Often, people managing BizFlow environments gain a false sense of security by looking at the logs once and, after that, assuming everything is okay. This approach can quickly backfire because:
- Some common BizFlow errors only present themselves under heavy loads – say, at the end of the day or the end of the month
- In some cases, it’s the frequency of the errors over time (not the errors themselves) that matters
While the BizFlow application may behave normally during these time periods, keeping a close watch on the error logs could shed light on a potential problem before it affects the end-user.
Good Housekeeping for BizFlow Log Files
Regular cleaning of the log files is another important BizFlow management task. If one doesn't address errors and periodically purge the log entries, the log files can quickly grow to an enormous size. For instance, the ERA logs in BizFlow 9 can generate 5+MB per minute of data!
When the log files grow unchecked, the activity of logging itself can become a huge drain on the performance of your BizFlow environment. In fact, many IT professionals managing BizFlow environments aren’t even aware that some of these logs exist until the logs grow large enough to fill up a disk.
BizFlow Log Analysis Tips
It’s important to keep tabs on a variety of BizFlow logs, including:
- Individual components logs
- Archiving log
- SOAP logs
- ERA logs
- Form server logs
- Combined debug log
- Application server (Tomcat, etc.) logs
Each component of the BizFlow application generates its own log file. These individual component logs contain both performance and error information. They should be monitored continuously, as some errors only present themselves during heavy loads.
Archiving logs should be monitored to ensure that finished processes are being written to the archive tables and scrubbed from the active tables.
SOAP logs indicate problems with inter-system component communication and contain valuable profiling information. This profiling information can be turned into time-based metrics and used to predict issues and perform capacity planning analysis.
ERA logs track any activity on processes in response to events configured in ERA. Monitoring the ERA logs is critical for analysis of the ERA configuration and for troubleshooting any process problems.
Form server logs indicate any problems encountered in processing a form. Monitoring these logs can help you troubleshoot embedded code inside the form along with the actions the form takes.
Manual vs. Automated BizFlow Log Analysis
Without keeping close tabs on the BizFlow error logs, you can’t proactively manage your BizFlow environment. However, the sheer volume of data makes BizFlow log analysis a tedious and time-intensive proposition.
The typical manual approach to BizFlow log analysis involves opening a log file and searching for errors. However, in addition to being time-consuming, this method can generate numerous false positives – ranging from “normal” errors such as duplicate logons to inaccurate “timeout errors” for processes that completed successfully.
Another alternative to BizFlow log analysis is to narrow your manual search of the logs to look for only truly critical errors. There are two drawbacks to this approach: 1) you must first know what these truly critical errors are; 2) it’s virtually impossible to manually detect them amidst the noise of "normal" errors.
Human monitoring can’t possibly keep up with the volume of log messages generated by BizFlow – it’s like searching for a needle in a haystack. A better solution is to rely on a monitoring tool tailored for the BizFlow environment to filter out the noise and highlight real issues before they cause performance or availability problems.
System Shepherd®: A Platform for Proactive BizFlow Monitoring
Unlike generic application and database monitoring solutions, Absolute's System Shepherd® collects in-depth the BizFlow-specific metrics needed to facilitate meeting aggressive performance and availability SLAs.
System Shepherd continuously monitors BizFlow error logs for critical error messages and message frequency predictors, and alerts the appropriate person when it spots a problem or potential problem. System Shepherd also stores performance profiling information in a central repository, arming you with the data required to proactively analyze and tune your BizFlow environment on an ongoing basis.
Over the years, Absolute Performance has worked with a number of clients to optimize their complex, business-critical HandySoft BizFlow® implementations. During these engagements, we’ve noticed three common early indicators of problems that can lead to poor performance or outages:
- Log Analysis
- Blocked Processes
- Queue Analysis
In our last article, we discussed the importance of persistent automated log analysis in complex BizFlow environments. We will now turn our attention to another significant BizFlow monitoring and management challenge – blocked processes.
What are Blocked Processes?
A blocked process is an issue within the database where one query (BizFlow process) is holding locks that prevent other BizFlow processes from executing. In many cases, these situations will persist until the application is restarted or the problem is remediated in another manner.
In the complex environments of BizFlow, blocked processes quickly become a major issue. A single blocked process can rapidly escalate, degrading end-user performance. If it escalates too far, the blocked processes can eventually lead to an outage that causes the business to grind to a halt.
Blocked Processes in BizFlow: A Vicious Cycle
As a Business Process Management (BPM) solution, BizFlow is often responsible for orchestrating complicated work flows for thousands of business process instances each day. When a process doesn’t run to completion, the typical response of a business user or batch application is to attempt to do the same work over again or proceed to the next work item in the list. Doing the same work over again means attempting to access the locked record. Proceeding to the next work item may require access to physically adjacent records that may call for the same lock already being held.
If the problem is a true deadlock situation, the database engine will detect the problem and eventually kill one of the processes. However, sometimes these are not true deadlock situations. Rather, they are lock contention issues that temporarily behave like deadlocks. In this case, the database will never kill off the blocker. Once processes start stacking up against this lock, the problem compounds itself as the repeated attempts to process the work eat up available database connections. If unresolved, BizFlow performance slows and eventually crashes.
Blocked Processes in SQL Server vs. Oracle
Our experience with real-world BizFlow environments has shown that blocked processes are a more severe issue in environments using Microsoft SQL Server rather than in those relying on Oracle. Why? Each database engine takes a different approach to managing lock escalation, with the SQL Server method being less capable of handling the type of locking issues that we commonly see in BizFlow. While similar problems exist in Oracle, they do not typically escalate to a level that causes an outage.
So, if your BizFlow environment relies on a SQL Server database, it is even more critical that you keep a close watch for blocked processes to avoid unplanned downtime.
Tracking Down Elusive Blocked Processes
When overlooked, blocked processes can lead to longer and more frequent outages – not to mention growing frustration among business users. However, to deal with blocked processes, you have to be able to see them.
A blocked process in BizFlow doesn’t always manifest itself as a deadlock situation, in which case it will be missed by most standard monitoring techniques. Plus, it is often difficult for IT professionals to make the correlation between a BizFlow outage and subtle happenings in the database.
One clear indicator of a blocked process situation can be found by looking for long-held blocking locks keyed off of the DBA_BLOCKERS view in Oracle or the SYSPROCESSES table in SQL Server.
If the blocker is killed by the database engine or manual intervention, the processes stacked up behind the lock will process through and the process instance whose query was killed off will go into one of several stuck flow conditions, including:
- Processes with no running activity
- Running activities with no work items
- Work items that were checked out earlier than today
One approach to identifying the root cause of a blocked process situation is to use log profiling techniques similar to those discussed in our article BizFlow Hot Button #1: Log Analysis. However, identifying process instances and their behavior by hand in a log can be as tedious, time-consuming, and inaccurate as manually tracking error entries.
Another alternative for tracking down root cause is to run the queries for detecting stuck work flows before and after you kill the blocker.
Mitigating Stuck Work Flows
Once you have identified a blocked process, you must clean it up using a two-step approach. The first step is to kill the process in the database that is holding the lock. This action will usually create an orphan or stuck work item inside of BizFlow.
The next step is to resolve the stuck work item. The options for fixing the stuck work item vary based on your company’s customized work flow. For some processes, the remediation may be as simple as someone on the IT team restarting the activity to get the work item to process forward. However, restarting an activity is not the only solution. More complex problems, such as those that involve dependencies on other systems or custom code written against BizFlow tables, may require the intervention of a business power user for resolution.
Absolute Performance has consulted with a number of companies who use HandySoft BizFlow® as their platform for business process management and automated workflow. Through our work with these complex, real-world environments, we have discovered three common problems that can degrade BizFlow performance or cause an outage:
- Log Analysis
- Blocked Processes
- Queue Analysis
In our last two articles, we discussed the importance of BizFlow log analysis and the potentially vicious cycle of blocked processes. In this article, we’ll discuss another key BizFlow monitoring and management challenge – queue analysis and tuning.
BizFlow Queues: A Delicate Balance
Queue analysis and tuning is important in BizFlow for two reasons:
- All work items flow through the BizFlow queues at some level; and
- The volume of work introduced into the BizFlow queues directly affects system performance
In a typical BizFlow deployment, work flows into the queues from another application in the environment, such as an imaging application or a customer-facing order entry system. If the BizFlow queues are not effectively monitored and tuned, they can quickly become a bottleneck in the system.
Controlling the flow of work into BizFlow is a delicate balance. If too much work enters the queues, BizFlow performance rapidly degrades and you risk ending up with blocked processes in the database that can cause an outage. However, if the flow of work is too low, business-user productivity suffers as work items are not processed in a timely manner.
Two Approaches to Controlling Work Item Flow
There are two common approaches to regulating the flow of work items into BizFlow:
- External controls
- Queue tuning
The first alternative is to build appropriate external controls to manage the volume of work items before they reach BizFlow. This approach is preferred because it offers more fine-grained control and the ability for the controls to be dynamic — dynamic controls don’t exist within BizFlow itself.
Another option for regulating the flow of work into the system is to modify the configuration parameters for the BizFlow queues. HandySoft supplies a number of parameters for tuning BizFlow queues, including:
- The total number of queues
- The number of threads associated with each queue (# of jobs each queue can simultaneously process)
- The check interval for each queue
- The maximum amount of time each queue will wait for a job to process
Regardless of which alternative you use to regulate the flow of work into BizFlow, monitoring key data about the BizFlow queues is absolutely critical for making good choices in your parameter settings.
The Challenge of Monitoring BizFlow Queues
In order to effectively tune BizFlow queues, you need to collect detailed statistics about each queue including:
- Elements in Queue
- Queue Activities Added per Minute
- Rate of Queue Processing
- Queue Size
- Queue Processing Rate
- Total Number of Errors
- Rate of Errors
You also need the ability to correlate this queue-specific data with system load on the application server, system load on the database server, and end-user response times to help guide your tuning efforts.
However, BizFlow queues are file system based. Most monitoring solutions don’t offer a mechanism to deal with file content monitoring. Absolute Performance's System Shepherd® platform, on the other hand, equally monitors file content, database records and business processes and is precisely architected to make and visualize correlations between all these diverse data sources.
