| 
Capita had been awarded an outsourcing contract for a major client whose principal function is to process (utilising a 24 x 400 Mhz Unix system) some four million applications each year. Problems with response times had started to manifest themselves very early in the life of the system, resulting in considerable user frustration and an increasing backlog of queued applications.
This was not John Ryland's first experience of this system. Whilst still working for Oracle, he had been responsible for analysing the same system (using Athene) to assess whether the core database application had been implemented to Oracle's standards. He commented: "Trying to get to the root of the real problem with the system was very challenging. There were no metrics that we could use and no methodologies. To start to understand where the problems were I arranged to install Athene. We were able to confirm very quickly that the Oracle application had been correctly installed but that it was totally CPU bound. Although the application was running on a 24 x 400 Mhz CPU system, it was still under great stress. In fact, we had no idea what the real CPU requirement was as it was being 'artificially' constrained. "
Having left Oracle and joining Capita in October 2002, John Ryland was reunited with the by now notorious system and joined a team charged with resolving the critical problems faced by the client. The clerical staff were being required to work shifts to cover 7 x 24 hours, the system was incredibly slow and the backlog of unprocessed applications remained a problem.
His first task was to develop a detailed Capacity Planning and performance methodology which included the extensive use of Athene. The report identified all the deliverables, the pro formas for the reports and the types of reports which would be produced, plus the timeframes. This was accepted by the management of Capita and the client - and Athene was switched on to start monitoring the system.
Given the client's anxiety that running a monitoring tool would impact on the already extremely poor performance of the system, Athene was set at medium granularity, collecting data at thirty-minute intervals.
John Ryland: "Because the client's system was totally saturated, we were not able to carry out any modeling on the actual system. We used the Explorer and Analyst modules of Athene to help us understand which parts of the system were generating the CPU load, and also to help us identify which non-essential processing activities we could remove. Even small changes could give us some service improvements. For example using Analyst I was able to see that there was overuse of the Unix process 'top', there were often over six of these processes running concurrently during peak processing times, these were taking up to 10% of the available CPU. I was able, using this information, to issue guidelines which resulted in a reduction in the concurrency of this job (down to one).
John Ryland: "Reports from Athene indicated that of the three Oracle instances on the live database server, one instance alone was generating the majority of the load - and it was clear that this was where we needed to concentrate." |
Based on these findings, a performance improvement project was instigated. This identified a list of performance changes, weighted in terms of business criticality and the impact upon CPU loading, which needed to be implemented. The project was given a life cycle of twelve months and the improvements to the system were made over the course of this time, scheduled in with other major work which had to be carried out.
In December 2003 one of these changes was implemented, this turned out to be a 'magic bullet'. Average CPU loading immediately reduced from 100% to below 75%. "Users noticed the difference immediately. The number of priority calls to the call centre has been reduced from several every week to just one in the last seven months. Queue lengths at peak times have been reduced from over fifty to less than 5." Work has continued using Athene to improve the performance of the system. CustomDB is being used in collaboration with Explorer and some Oracle tools to produce workload profiles. John Ryland: "I am confident that we now have sufficient spare capacity to deal with surges in user demand. Our focus at the moment is to improve the metrics that we are collecting so that we can in turn provide better SLA measurements."
The importance of the work undertaken by John Ryland and the other architects is widely recognised by Capita, the client and the users. Mr. Ryland now 'polices' the system continuously and any changes in, for example, shift patterns, are reported to him before they are implemented in order that their effect can be modeling using Athene. "When there were a series of high profile football matches on the television, some of the staff who were on flexi-time were changing their shift time to the mornings, which is our busiest time. We could see, using Athene, as soon as this started to happen and were able to alert the staff that this would soon begin to have a detrimental effect on response times.
"The reports produced by Athene have enabled us to produce easy to understand illustrations for the client and the management of Capita of how the system is working, and what effect various changes have made or will make to response times and the performance of the system.
"If we had been faced with all these problems and had not had access to Athene, we would have had to do much of the work manually. This would have included pouring over a phenomenal number of metrics trying to understand them. Using Athene, this has all been carried out automatically. It is very flexible, easy and intuitive, to use."
Next Case Study |
