Using Fuzzy Group Analytical Network Process and ISO 9126 Quality Model in Software Selection: A case study in E-Learning Systems

INTRODUCTION

Using computer package in various industries has been increased recently. Many companies use complicated soft wares to reduce cost of their firms and improve the quality of their works and products. Although, Software companies have been produced variety of packages to respond the existing demands. The cost of these packages is very high, so selecting appreciated software is a vital issue in many companies. For example, which strategy must be considered during selection of ERP package? Therefore, selecting a software package that meets the requirements of a company is a difficult task. This has led researchers to investigate better ways of evaluating and selecting software packages (Jadhav and Sonar, 2009). Hence, for each field of industry that uses special software package there are several attempts in software assessment that have been done by many researchers.

In accounting software, Arditi reviewed essential criteria to select suitable accounting software and finally proposed a list of popular packages on the US market and classified them by their respective characteristics (Arditi and Sandeep, 1991). In another research, Adhikari et al. (2004) reported a survey on process of selection and design of international accounting software and investigated the relationships among firm characteristics.

Another software package that mostly has been considered in various researches is Enterprise Resource Planning (ERP). Commercial packages of ERP could not provide a unique business model for every process of all industry (Teltumbde, 2000). Thus, no single ERP Software can meet all company business requirements (Wei et al., 2005). In 2000, Teltimbde used Nominal Group Technique (NGT) and the evaluation methodology adopting the Analytical Hierarchy Process (AHP) for evaluating ERP projects (Teltumbde, 2000).

Wei et al. (2005) proposed a comprehensive ERP system selection framework in which the objective hierarchy is constructed and the appropriate attributes are specified to provide detailed guidance for ERP system evaluation.

In the field of E-learning systems selection, there are many attempts that have been done during more that 10 years. Soong et al. (2001) identified some Critical Success Factors (CSF) for favorable usage of on-line course resources. Wang (2003) used a survey, generating items, collecting data and validating the multiple-item scale for developing a comprehensive model and tools for measuring learner satisfaction with asynchronous E-learning systems. Also, many other researchers used MCDM approach for evaluating E-learning systems. AHP proposed by Saaty (1980), has been widely used among other MCDM techniques for bench marking of E-learning systems (Shee and Wang, 2008; Tzeng et al., 2005; Chao and Chen, 2009; Colace et al., 2006; Mahdavi et al., 2008). Moreover, Jadhav and Sonar (2009) reviewed more than 50 papers including software package selection.

This review study investigated methodologies for selecting software packages, software evaluation techniques, software evaluation criteria and systems that support decision makers in evaluating software packages (Jadhav and Sonar, 2009).

In addition, there are several attempts to identify quality attributes. The ISO-IEC 9126-1 standard provides a hierarchical quality model composed of characteristics and sub characteristics in terms of internal quality, external quality and quality in use (ISO/IEC 9126-1, 2001; Jung, 2007). Many researchers used this model for evaluating software package. Bevan (1999, 2001) described the new framework for software product quality developed for ISO/IEC 9126-1 (2001). Suryn and Laverdiere (2007) presented a solution for methodically evaluating the internal quality of a soft switch. In his research he used measurements of internal quality attributes of software which has been defined by the technical report ISO/IEC 9126-3. Zulzalil et al. (2008), proposed a method to investigate and identify the relationships between ISO-IEC 9126 quality criteria in the development of web-based applications.

In this study, we used some identified relationships between ISO-IEC 9126 that have been demonstrated by prior researchers and created a network structure instead of existing ISO-IEC hierarchical structure quality model. Moreover, we used Fuzzy Group Analytical Network Process (FGANP) for evaluating software package.

SOFTWARE QUALITY CRITERIA

Definition: A lot of work has been done since 1976 by a number of researchers to design a software quality framework. According to ISO 9126-1 (ISO/IEC 9126-1, 2001) quality is defined as a set of features and characteristics of product or service that bears on its ability to satisfy the stated or implied needs. A quality model is defined as set of characteristics and the relationships between them, which provides basis for specifying quality requirements and evaluating the quality (ISO/IEC 9126-1, 2001).

Pioneer software quality models: McCall et al. (1977) proposed a model for software quality. Firstly, he started with 55 quality characteristics which had an important influence on Software quality. After that, McCall reduced the number of characteristics to eleven. Another model that was developed by Boehm and In (1996), added seven characteristics to McCall’s model. Moreover, his model included some considerations about evaluation of software products with respect to utility of the program. However, Boehm’s model was same as McCall’s model in which it represented a hierarchical structure of characteristics and both researchers (McCall and Boehme) included user’s needs. Perry (1987) had shown some interrelationships between Boehme’s criteria which was mostly used in later research. In another research, Dromey’s model demonstrated some relationship between the attributes and the sub-attributes of software quality (Dormey, 1995).

By increasing the number of quality models, the need for a sole standard model was felt. As a result, ISO institute started to develop ISO/IEC 9126 in 1985.It was a part of ISO 9000 as a vital standard for quality assurance. This model has a hierarchical tree structure that contains main characteristic and sub-characteristic. It has 6 main criteria including functionality, reliability, usability, efficiency, maintainability and portability. These criteria have many sub-criteria. Also it provides a framework which can consider some trade-offs between software product specifications (Behkamal et al., 2009) (Fig. 1).

ISO 9126 identifies the internal characteristics and external quality characteristics of a software product and contains various characters. However, it has some disadvantages abut showing very clearly how these aspects can be measured and the proposed criteria are so general. Suryn and Laverdiere (2007) analyzed architecture of a soft switch in the telecom industry in order to identify internal quality characteristics, sub characteristics and related measures critical to its functioning as well as a help to measure them. They have been used ISO/IEC 9126-3 as the fundamental reference for software product quality measurement and evaluation. In another research done by Wolfgang AHP is used to evaluate three different AHP software by considering ISO/IEC 9126 criteria (Ossadnik and Lange, 1999). Bee Chua and Laurel Evelyn Dyson used ISO/IEC as a useful evaluation tool for assessing E-learning soft wares (Chua and Dyson, 2004). In another research on B2B(business to business) application that has been done by Behkamal and Kahani, they Used ISO/IEC criteria as a well known quality model in software quality and customized it by some special characteristic of B2B application (Behkamal et al., 2009).

Fig. 1:	ISO/IEC 9126 software quality structure

Fig. 2:	Existing relationship among dimension of software quality

Fig. 3:	ISO/IEC 9126 Network structure

Interrelation between ISO-IEC9126 criteria and ANP model: As mentioned before, Boehm has been presented a quality model which had a hierarchical structure, but this model did not consider any interrelationship between criteria. However these criteria may have some effects on each other. Perry (1987) showed that there was some interrelationship between quality factors. Perry labeled them as inverse, neutral and direct. Many researchers used Perry’s model as a useful subject. Zulzalil et al. (2008) presented a method to investigate and identify the relationships of quality criteria in the development of web-based applications and conducted this research. They used Experienced-based approach and an online survey to gather the intended relationships; in this case they used ISO/IEC characteristic of software quality and used predefined relationships among these criteria that has been proofed by Perry. Henningsson and Wohlin (2002) conducted a research based on relations between software quality attributes in their research. They used two different methods to identify these relationships. First, they review some literature on this subject and in second approach they conducted interview survey with people from industry. Finally they made a table which shows a summary of relationships ( Boehm and In, 1996). By considering Perry, McCall, Boehm, Knnet, Henningsson and Hazura-Zulzalil researches, Fig. 2 shows a summery of software quality relationships which have been addressed in former papers.

Figure 2 shows that there are some relations between 4 main criteria. For example, efficiency and reliability have effect on maintainability and usability. In this case we did not consider positive or negative impact which was in Perry’s model. So, by considering this similarity between criteria we can found some interrelations between ISO/IEC 9126 main criteria which have been shown in Fig. 3 (Zulzalil et al., 2008).

Hence, during evaluation of software package by ISO/IEC 9126 criteria, relationships between main criteria have been considered. As a result every assessment process in any kind of software package like accounting software, B2B, AHP and others, can consider these relations while using ISO/IEC as a reference model. However, we can not use AHP application in this situation. So, we apply ANP (Analytical Network Process) technique which has been proposed by Saaty (1980,1996).

ANP literature: ANP developed by Thomas Saaty provides a way to input judgments and measurements to derive ratio scale priorities for the distribution of influence among the criteria and groups of criteria in the decision (Saaty, 1996, 2003). The well-known AHP is a special case of the ANP. Both the AHP and the ANP derive ratio scale priorities by making paired comparisons of elements on a common property or criterion (Soheil et al., 2010). Many decision-making problems cannot be structured hierarchically because they have some dependency among their criteria. Hence, structuring a problem with functional dependencies that allows feedback among clusters is considered to be a network system. Over the years, ANP, a comprehensive multi-purpose decision method, has been widely used in solving many complicated decision-making problems. In study by Meade and Sarki on introducing a decision methodology and structure for manufacturing (and organizational) agility improvement, they used ANP to evaluate logistic strategies and to improve production speed (Meade and Sarkis, 1998). Moreover, Yuksel used ANP to demonstrate a process for quantitative SWOT analysis that can be performed even when there is dependence among strategic factors (Yuksel and Dagdeviren, 2007). Mikhailov and Singh (2003) represented a fuzzy ANP application to develop a decision support System. Mulebeke and Li (2006) applied analytical network process on multi attribute strategic decision making approach to help the selection of appropriate software to suit the product development process of a particular product. Soheil et al. (2010) proposed an ANP model for evaluation E-learning system; they used Fuzzy Group Analytical Network Process in evaluation process.

Fuzzy group ANP application

Table 1:	Linguistic value look-up table

Also, for helping the decision makers to enhance their decisions and improving their accuracy we used linguistic variables method as is shown in Table 1 (Cheng et al., 1999):

(1)

As a result, for deriving fuzzy comparision matrix we created matrix of pair wise comparisons (Eq. 1); where, = (1/u_ijk, 1/m_ijk, 1/l_ijk) and = {l_ijk, m_ijk, u_ijk}, also l_ijk, and u_ijk are the lower and the upper bounds Representing the scope of the fuzziness of the fuzzy number and m_ijk is core of the fuzzy number, corresponding to the maximum degree of membership, equal to one. By using concept of α-level sets or α-cuts (Mikhailov and Singh, 2003), for a given α-level, the interval version of the Fuzzy Preference Programming (FPP) method tries to find a crisp priority vector W = (w₁, w₂,..., w)^T, which satisfies approximately all interval constraints: with solving the weighted GFPP model, the crisp priority vector W and the objective function C which is an indicator that measures the overall consistency of the judgments are obtained:

S.t.

(2)

where, Σⁿ_i=1 w_i>0; i = 1,…,n ; λ_k>0; k = 1,..., r; q = 1,... 2 m.

When, the interval judgments are consistent, the maximum value of C is greater or equal to one. For inconsistent judgments, C takes a value between one and zero that depends on the degree of inconsistency and the values of the tolerance parameters d_q. In addition l_k demonstrates the weight of decision makers. Here, we have some pair wise matrices which are enforced by GFPP model.

Fig. 4:	GFPP process

(3)

where, w₁₂ is a vector that contains comparision of criteria respect to goal, w₂₂ is a vector that represents the inner dependence of the ISO/IEC criteria, w₃₂ is a vector that contains comparision of sub criteria respect to main criteria and w₄₃ is a vector that illistrates comparision of Alternatives to sub crieria.

CASE STUDY

In this case, we evaluated some E-learning systems by considering both ISO attributes and of E-leaning success factors. In order to address the problem, our Evaluation process contains five steps as follows:

Step 1 has been widely explained in earlier. In Step 2, for evaluating an E-learning system, many factors must be considered during the assessment process which affect such system (Soheil et al., 2010). In addition, the medium of E-learning is the world wide web and most of E-learning systems transaction does perform on the basis of web. Hence, E-learning quality is related to the quality of the web pages and web services (Behkamal et al., 2009). Moreover, there are several studies in field of determining E-learning success factor (Colace et al., 2006; Chao and Chen, 2009; Tzeng et al., 2005).

Among these researches, Wang (2003) identified 4 main criteria and 12 sub-criteria for them. Also, in 2008 Wang proposed an AHP application based on these factors. Table 2 shows these criteria with their sub criteria.

However, only one of the main criteria in Wang’s model; learner interface; has the same sub criteria in comparison to usability in ISO/IEC 9126 standard (ease of use, user friendly, operational stability and ease of understanding) (Soheil et al., 2010).

Table 2:	Wang’s E-learning factors [34]

So, we added other mentioned criteria such as learning community, System content and Personalization of Wang model to ISO software quality model in order to evaluate E-learning systems. At last, our final model consists of 9 main criteria (contains Wang factors’ and ISO criteria) and 29 sub criteria. Secondly, we conducted a survey to found out pair wise comparisons between criteria, sub criteria and alternative using a Standard AHP approach, thus we evaluated an AHP model for selection of the most appropriate E-learning Software. In following Step, we considered existing relationships in ISO model.

In addition, In December 2009 we evaluated three E-learning systems in Iran that have been used in different well-known university, namely, Iran University of Science and Technology (IUST), AmirKabir University of Technology (AUT), Virtual University of shiraz (SVU) which all of them located in Tehran. All of these university provided high standard E-learning systems for student we have captured some information about their performance by conducting some interviews with students and staffs. This process continue for about one month until January 2010.

AT this step, a survey was carried out. Data were gathered from students, the managers of these systems and also some software developers who were involved in developing E-learning systems. To do this, we conduct another pair wise comparison between criteria. Also, for helping the decision makers to decide between pairs of elements and improving their accuracy, we used linguistic variable weight method that is shown in Table 1.

To identify global priorities in E-learning system with its interdependence, we created a super matrix as mentioned in earlier, Eq. 3. Finally, GFPP method has been applied on each comparison matrix and, final weight of each sub criteria and criteria have been shown in Table 3. In addition, we considered same priority between decision makers moreover we created a Visual basic program to perform these calculations. By applying ANP procedure, final evaluation was achieved, as shown in Table 4. Finally, SVU has the best output in our assessment process.

Table 3:	Weight of criteria and sub criteria

Table 4:	Final evaluation result