While software applications are becoming increasingly large and complex due
to the improvement of software development techniques, financial enterprises
rely more than ever on some core software systems. Software quality is extremely
important in financial systems with the goal to handle business transactions
quickly, accurately and reliably, to protect the business privacy and successfully
deal with kinds of exceptions. Though the delivery date, budget and resource
effort can be negotiated most of the time, software quality is a very important
criterion for the customer to accept the product and must not be jeopardized.
Bad quality software applications frequently results in revenue loss, hurts
the user confidence, unable to provide competitive advantage to business and
creates additional workload (Original Software, 2010).
Incremental development model (Mills et al., 1987)
is adopted to handle requirements gradually and develop the system in steps
that accumulate the functionalities. It allows partial utilization of product,
shortens the development time and helps ease the traumatic effect and risk of
introducing completely new system all at once (Sommerville,
2001). While it takes advantage of flexibility in resource utilization (Ruhe,
2005), incremental software development model also brings forth some problems.
First because each increment is developed and integrated into the application
consequently, much effort is required if the previous increments were poorly
designed, hard to be understood and difficult to be enhanced (Xu,
2005). Defects slip from former increments may be mixed and amplified which
may introduce new problems to the following increments and thus increase the
effort to fix them (Xu, 2010). Second, the turnover
of human resource requires knowledge sharing effort and further increases the
effort in subsequent increments development across different increments (Xu
and Pan, 2006). Third, more requirement changes are expected in incremental
development and if not properly managed or controlled, development will be definitely
led into chaos (Xu, 2005; Xu et
al., 2004). Quality management (Rose, 2005) after
each increment or even during one increment is necessary to solve these problems.
There are four main components in quality management: Planning, control, assurance
and improvement (Rose, 2005). It is not only a principle
that ensures the high quality in software products and services but also the
meaning to control processes. Quality management therefore uses quality assurance
in software product and controls processes as well as products to achieve more
consistent quality. Quality management is important to companies to have higher
product performance, increased revenues, better customer satisfaction and smaller
waste (Ahire, 1997). A survey conducted by Original
Software (2010) revealed that the importance of quality management has risen
while most managers are not satisfied with their current quality management
The objective of this study is to clarify the special quality related problems in the incremental financial software development and then enable the management to understand and oversee the quality status of the increments so as to make the best tradeoff in quality management and avoid the insufficient testing and optimize the quality assurance tasks.
There are not many works that focus on quality management in incremental software
development. Hewett (2011) suggested mining software
defect data to support software testing management. A method is proposed on
how to order requirement in incremental development to improve quality management
(Wohlin, 1994) and Unified Process is chosen as the embedded
framework to perform the quality management (Jacobson
et al., 1999; Norbjerg, 2002; Kroll
and Kruchten, 2003; Shuja and Krebs, 2007).
Unfortunately, quality is often been ignored when the project is short of schedule
or budget. The requirement on a specific release date often overrules quality
objectives, for example reliability. A software product may be delivered on
time but not thoroughly tested or verified before release due to delays. In
other words, testing effort is cut off because of the tough schedule which results
in poor quality software released to client (Original Software,
2010). In order to ensure the high quality in the incremental software development,
quantitative quality management is essential to measure the quality and the
risk of low quality. However, the quality characteristics defined in the ISO/IEC
(2001) is not practical to be measured directly, lower-abstraction attributes
of the product should be accessed from ISO/IEC (1998).
With the years of incremental financial software development, the authors found that the poor quality increment resulted endless quality risks, brought more rework effort and continually hurt the team confidence in the following increments before the related defects in the former increments had been finally fixed. A poor-quality-delivery of an increment brings forth lots problems because:
||Defects inside former increments may be amplified in the following
||After-increment defect removal requires much more rework effort and impact
a serial of tasks across requirement, design, coding and testing phases
||Worse quality status gives critical pressure on the schedule management
of consequent increments and would make the following quality status even
In this study, the authors argue that quantitative management techniques should be adopted in software quality management and suggest a series of practical solutions to evaluate, analyze and improve the software quality in incremental financial software development.
QUANTITATIVE SOFTWARE QUALITY EVALUATION
Quantitative quality management framework: Quantitative quality management
framework is a dynamic framework which could be improved during the incremental
software development practice. It starts at evaluation phase, within which the
quality and the quality management will be evaluated with some defined rules
and criteria. After the evaluation, some further analysis could be done on increments
with potential issues. The rules, criteria and the factors could be calibrated
and improved to reduce the total effort in quality management. The whole framework
generates a loop which enables the continuous quality improvement (Fig.
1). All these four steps could be overlapped in practice, for example, the
evaluation can be made during the practice and the improvement can be suggested
after the practice has started.
Indicate the possible quality issues after increment: Bug density can
be a good metric to measure the quality of an increment by comparing the subsequent
increments in an incremental software development project.
|| Quantitative quality management framework
Generally, a lower bug density means High quality while a higher one means
Low quality which will be the subject of later analysis.
In this study, bug density is viewed in another view and used to indicate the possible quality issues after increments. Bug density will be estimated according to some historic project from the benchmark repository. The estimated bug density will be calibrated with the average value during the progress of the incremental software development. We assume that there are some problems in those extremely High or extremely Low quality increments. For example, the testing team had much more time to test the increments and found more defects than the average for some Low quality increments. For the High quality increments, maybe the testing team didnt have sufficient time to test the increments and found far too little defects than the average.
In this study, we focus on the extremely High quality increments. Several criteria are set to indicate the possible quality issues:
||Scale is large but complexity is not lower. Line of source
code (ESLOC) is used to evaluate the scale and scorecard is used to measure
||Test duration is short
||Product management is highly efficient and does an excellent job for resource
allocation and schedule planning
Root cause analysis: Causal analysis and resolution is a CMMI process
at level 5. It contains two Specific Goals which is split into five Specific
Practices (Buglione and Abran, 2006).
Root cause analysis enables the team to find out the weakness of the development so that the team could improve the quality of the increments in the future by avoiding to the similar cause.
An analytical tool from Total Quality Management (TQM) (Deming,
1986; Juran and Gryna, 1993; Crosby,
1979) named the Fishbone diagram (Yu, 1998) is useful
for detecting the root causes of a software defect/problem and for classifying
and prioritizing issues.
For each fault fixed, quality assurance engineers and developers choose a fault
type and one or more root causes. Based on the fault data collected from developers
and quality assurance engineers, most actual faults could be covered. Fishbone
diagram for defect cause analysis is suggested by Buglione
and Abran (2006).
Statistics analysis on defects from root cause perspective: Root causes described in the fishbone diagram can be considered as the multiple-dimensions, for example, SDLC original causes were used as the top categories in this research. Some of such causes can be Requirement misunderstanding, Requirement unclear, Requirement faulty, Detail design failure and etc.
Statistics analysis: Defect report is generated after the root cause analysis. In order to enable the decision making in the quality improvement, the price in bug fixing is estimated in the unit of working hours according to manual experience.
QUANTITATIVE QUALITY IMPROVEMENT
Solutions for quality improvement: There are many solutions for quality improvement while these solutions requires different invest of budget or human resource. For example, financial budget is required to purchase a better testing tool and some training effort is also needed for the test engineers to perform this new testing tool. Such financial budget investment and the training effort are fixed cost which is not dependent on the number of defects.
The improvement solutions include the follows, just to name some of them.
||To reduce defect of Coding:
||Enhance development test
||Involve code review
||New technology training
||To reduce defect of Requirement Misunderstood/ Unclear:
||Requirement review meeting
||Requirement question list
||To reduce defect of Faulty Requirement:
||Involve more clients interactive
||Enhance business analysts role
The solutions for the quality improvement are not limited to solve only one
defect category. For example, the software review may reduce the defects in
the coding and benefit the requirement understanding. In such case, the related
benefit and cost is different for each category.
For each root cause category, there could be several improvement solutions
with different benefit and cost. Typically the benefit is measured in the unit
of the Defect Removal Efficiency (DRE) and the cost refers to both the duration
and effort to avoid such defects through review or similar techniques. The values
of benefit and cost should be benchmarked and calibrated for individual IT projects
according to its local historic experience.
RELATED DEFINITIONS IN THE IMPROVEMENT
Generally, Pareto analysis will be performed to find out the possible defect category to be improved. For each category to be improved, a suitable working plan should be determined according to the number and price of the defects and related benefit and cost of the solutions.
Summarized report of root cause analysis: Summarized report of root
cause analysis can be defined as a matrix SRCA :: = <RC, N, P, BV> where
RC is the top category of the root causes, N is the number of the related defects,
P is the estimated bug fix effort and BV is the business value after fixing
Typically, high quality of software refers to low defect density in the software. Here in this research, the business value is considered as the weight of quality when comparing defects across different categories. In order to improve the quality continuously, the project team should clarify the status of defects and remove the defects efficiently, that is obtain more business value with less bug fix effort.
Hot defect categories refer to the defect categories which should be reduced in the increments so as to enable the high software quality. Business value and bug fix effort will be tradeoff to obtain high quality in the increment. Most of the time, project team needs to enlarge the business value with limited budget for the bug fix effort.
Improvement solution: IS :: = <ID, Name, FixedCost>, where
FixedCost is the cost which is independent from the number of defects or the
time of implementation. The FixedCost is important for the project managers
to make the decision to try a new testing tool when the Fixed Cost is affordable
and the related execution cost is small when compared with some other solution.
Capacity of improvement solution: The capacity can be defined for
the quality improvement towards different defect category. CIS :: = <RC,
IS, DRE, Cost, N, SEffort> where RC is the root cause category, IS is the
improvement solution, DRE is the possible defect removal efficiency to remove
the defect category RC with the solution IS and Cost is the execution effort
for each implementation, for example, the additional requirement review effort.
The execution cost varies from different defect category and different improvement
solution. N and SEffort refer to the number and summarized effort for the defect
in the category of RC which detected by the solution IS. These two items are
prepared for the calibration of the capacity.
The capacity will be impacted by the team capacity, the technique complexity and the project environment. Therefore, capacity of improvement solution will be calibrated and updated with more project experience so as to be shared with multiple projects.
Quality assurance task organization: In some case, one improvement
solution may exposure the defects in multiple categories. For example, the functional
testing may help to find out the requirement, design and coding related defect.
The organization of quality assurance task is defined as QTO :: = <Task,
IS, QTI> where Task is the task category for quality assurance work, IS is
the improvement solution and QTI is queue of <RC, Ratio> where RC is the
defect in root cause category and Ratio is the percentage of effort.
Quality assurance task execution: In the execution of each quality
assurance task, defects in different categories may be detected. The execution
of quality assurance is defined as QTE :: = <Id, Task, IS, Effort, DefectFound>
where Id is the task identification of qte. Task is the task category for quality
assurance work, IS is the improvement solution, Effort is the effort to finish
task qte and DefectFound is queue of <RC, N> where RC is the defect in
root cause category, N is the number of defects found in the execution of qte.
Quality improvement system: The entire quality improvement system
can be defined as QIS:: = < SRCA, IS, CIS, BVexpected, Budget>
where SRCA is the summarized report of root cause analysis, IS is the set of
improvement solutions, BVexpected is the expected business value
in the improvement and Budget is the reserved budget for the bug fixing.
Algorithms for the quality improvement: When there is no sufficient
time for the bug fixing, the defects couldnt be fixed within one increment.
However, the quick bug fix will benefit the product quality and reduce the development
of future increments. In this study, to obtain the maximal business value within
the bug fix budget is used as the criteria. Algorithm 1 is
used to identify the defect categories which should be fixed for the better
quality and the defects in other categories may be considered to be fixed in
the next increment.
|Algorithm 1: Identifying hot defect categories
Algorithm 1 walks through the summarized report of root cause
analysis and identify the hot defect categories to be reduced in the increment.
The defect category is queued in the descending order of business value/bug
fix effort. In such way, the quality improvement could be achieved in the most
efficient way which has the largest return on investment.
The bug fixing effort is limited by the reserved budget and the expected business
value is estimated as the summary of the number and the business value per defect.
Of course, the bug fixing effort contains the re-testing and re-opening effort.
|Algorithm 2: Calibrating of the capacity of improvement solution
Algorithm 2 initializes the capacity of improvement solution
CIS, defect remained DR and root cause analysis report SRCA at first, then update
the summarized defect into CIS, SRCA and DR as well. The effort for the defect
detecting is fetched from the quality assurance task execution QTE. Finally
the DRE and Cost are calibrated in CIS.
Here, DRE is calculated as ΣDefect_found/ (ΣDefect_found+Defect remained) for each defect category. The number of defect remained is the feedback from business side after the increment.
For each phase, there are a serial of development and Quality Assurance tasks. From the perspective of quality assurance, development tasks introduce the defects and Quality Assurance tasks help to explore the defects.
After the benchmark has been prepared, the defects numbers of a new increment
can be estimated according to some predict approach. In Algorithm
3, we suggest an approach to schedule the improvement solutions to the coming
increment. E-SRCA is a matrix E-SRCArc, ph which is estimated according
to the historic data, where rc is the defect category identified by root cause,
ph is the phase number stated from 1 to 4, refer to Inception, Elaboration,
Construction and Transition. E-DR is a vector E-DRrc,ph which is
estimated according to the historic data as well where rc and ph have same meaning
as in E-SRCA but E-DRrc,ph refers to the number of defects in the
category of rc will be remained after phase ph. The structure of scheduled improved
solutions is defined as a vector SISph and each item refers to a
queue of improvement solutions <IS>.
|Algorithm 3: Scheduling improvement solutions
Algorithm 3 is briefly demonstrated with some vector access,
such as the line 5. The member method total number () is used to calculate all
the defects number of a vector. For each phase, the total removed defects number
should be no less than the predefined number in E-DR. SISph. Initial
() at line 4 is used to initial the vector as a empty queue. SISph.
add (is) at line 9 is used to append the improvement solution is to the vector
SISph. The scheduling improvement solution in SIS together with the
E-Effort will be output for the quality assurance task scheduling.
CASE STUDY IN GLOBAL IT CORP
We applied our approach in a financial software company which provides advanced IT solutions to a top financial organization.
As usual, we collected bug density increment by increment as shown in Fig. 2. We found that the increment R003s bug density was fairly lower than the average bug density, even lower than the average minus one sigma. Therefore, we checked the criteria:
||The scale is LARGE but the complexity is NOT lower. The features
in the increment R003 were more complicated than the average based on the
scorecard from development and testing teams
||Test duration is very short comparing with other increments
||As a surprise there was no sufficient product management for R003
As a result, the increment R003 was indicated to have quality issues: Test
cycle was not sufficient based on the stable testing resources across the multi-increments
and the short testing duration.
S-Curve which has been introduced in Section 2.3 was used to validate our assumption. S-Curve was generated as in Fig. 3 to demonstrate the expected and exposed defects number. There was a big gap between expected and exposed defects number for the increment R003: Only half of the expected defects had been found. This was an important evidence to support our assumption.
We verified this assumption with the project team and got the confirmation that the testing for the increment R003 was insufficient. Due to the tight schedule of the increment R003, testing duration was cut short which caused more after-release defects. It was really a good example of improper cutting testing efforts under tight schedule as many defects were found on clients site.
|| Bug density analysis on the increment (released to client)
|| S-Curve analysis on the increment R003
|| Defects distribution by root cause
|| Pareto chart for the root cause report
After we had indicated the increment with high quality issue (though it looked
like an extremely High quality increment), root cause analysis was used to clarify
the defects in the distribution of root cause, as is shown in Fig.
Pareto chart was used to identify the most important root causes that must be eliminated or diminished to improve the quality to an acceptable level. As shown in Fig. 5, it is clear that Requirement Misunderstand/Unclear and Coding were the main two causes for the increment R003.
According to our experience, the possible improvement solutions to reduce the requirement misunderstand/unclear related defects are enhancing the requirement review and establish requirement issue list. We suggested to the project team to enhance the requirement review in the following increments and host additional requirement review meetings for the previous increments to reduce the impact of the quality issue from R003. The decision making for the quality improvement should be made according to the feature of the project and the capability of the project team.
Many software quality metrics have been developed to verify the traceability
(Singh and Saha, 2010), the efficiency of object-oriented
technique (Parthasarathy and Anbazhagan, 2006; Boroni
and Clausse, 2011; Changchien et al., 2002)
and to validate special applications (Pan and Xu, 2010;
Vinayagasundaram and Srivatsa, 2007; Bedi
and Gaur, 2007). However, a single metrics means little sense and additional
information or metrics are required to determine the status of quality. As demonstrated
in Fig. 2, the bug density of R003 is far below the average
of the other increments. Typically it will be regarded as the high quality of
the increment. In this study, we found that the bottom boundary of the bug density
is 2.5686 and wondered if there was some quality issue in R003. With the survey
from the team, we found that the test duration was too short and the product
management was not sufficient. Therefore, we concerned that the testing on R003
is insufficient. Late, some feedback from production testing verified that there
were much more defects remained than other increments. The S-curve in Fig.
3 showed that the speed (efficiency) of the testing was much higher than
other increments. Both defect distribution diagram in Fig. 4
or Pareto chart in Fig. 5 showed that the requirement misunderstand/unclear
and coding accounted for too much ratio of the total defects. This could be
considered as another low quality evidence of this increment.
In this study, we made an assumption that the solution adopted in the testing phase would remove the defects in fixed ratio regardless of the priority or order of the adoption. Suppose that there are two improvement solutions, QA1 and QA2, the defect removal efficiency of QA1 is DRE1 and QA2 is DRE2. The defect removal efficiency of testing queue <QA1, QA2> is same as that testing queue <QA2, QA1> which equals to 100%- (100%-DRE1) *(100%-DRE2).
Assume DRE1 of QA1 is 80% and DRE2 of QA2 is 70% and there are 100 original defects. If QA1 is done at first, then it could explore 80 defects and QA2 could only explore 20*70% = 14 defects. If QA1 is done after QA2 then QA2 could explore 70 defects and QA1 could only explore 30*80% = 24 defects. There are 6 remain defects for both scenarios. However, the improvement solutions implemented in different orders result in different evaluation on the improvement solutions and will definitely yield different optimization results.
Pareto technique has been suggested to value the conditional dependence (Barro,
2009), optimize the service composition and to enhance the evolutionary
algorithm (Zhihuan et al., 2010). In this study,
we used Pareto chart to identify the most significant defect category as in
Fig. 5. However, the solutions could not be limited to only
those most significant categories if we consider the related implementation
cost. Currently we dont have enough evidence to validate this suggestion
but we will continue to gather the project data and it will be further studied
with plenty of project data in our future work.
The framework and approaches suggested in this study can also be used in some other quality-critical incremental software development. The historic project data should be prepared and be used to calibrate the benchmark. This will be the future work of the authors.
In this study, we suggested a quantitative quality management framework with
a series of analysis and evaluation approaches. It has been used in a real project
and the experience showed its value in identifying the quality issue which may
not have been easily found manually with some traditional approaches. In such
way, the poor quality increment may be reworked in time so as to avoid the possible
problems, such as the amplified defects number, more rework work due to late
defect identifying and fixing and low team confidence because of low quality
This work is part of Global Collaborative Software Development research project which is an attempt to improve the dual-shore software development with integrated best practice, software engineering technology and project management methodology. The research project is funded by State Street Corporation, USA. The project is collaboration between Zhejiang University, China and State Street Corporation, USA.