Abstract: The main purpose of this study is to apply an uncommon research methodology (Design Research) in the field of IT performance management. This study addresses the questions of what IT performance management field can learn from design approach and whether this perspective can be beneficial to design and implement managerial solutions that are understandable, approachable and effective to the many human beings, groups and organizations who work in IT departments or are affected by them. Thus, a detailed literature review is conducted on Design Research method as the underlying procedure of designing the proposed system. The concept of Design, the relationship between Design and Research and the general methodology of Design Research is investigated as well as the role of that in the field of management which is focused. Then, as a case application of the method and to assess the utility of selected research method in IT management scope, the IT performance management system is suggested, developed theoretically and practically in one of Iranian governmental organizations and finally evaluated. On the other side, the process and steps of Design Research method is reviewed and future research opportunities is recognized to work on soft approaches for designing managerial solutions.
INTRODUCTION
The complexity and challenges of today’s organizational settings has put great pressure on institutions to invent more effective managerial approaches and systems. Indeed, focus on management approaches and systems should be complemented by bringing and fostering expertise in design science to help design solutions to thorny problems (Mohrman, 2007). As Simon (1996) argues, organizations are our own creation and managers cannot stand outside as dispassionate observers trying to understand and explain them as scientific realities and pursue a research program aimed at finding truth. Organizations should be perceived as human designs and artifacts which designed to achieve the goals and purposes of human beings. This requires attention to management systems as useful designs and development of a design science to provide content and methodological knowledge to guide the design process (Mohrman, 2007). However, designing useful artifacts is complex due to the need for creative advances in domain areas in which existing theory is often insufficient (Hevner et al., 2004).
In this regard, Design Research (Design Science Research) can be considered as a methodic design process. In the Design Research paradigm, knowledge and understanding of a problem domain and its solution are achieved in the building and application of the designed artifact, hence it can be called exploring by building and it is inherently a problem solving process (Hevner et al., 2004; Vaishnavi and Kuechler, 2008). In fact, Design Research is yet another lens or set of analytical techniques and perspectives (complementing the positivist and interpretive perspectives) for performing research (Vaishnavi and Kuechler, 2004). On the other side, regarding Kuhn (1996) and Lakatos (1978), research can be very generally defined as an activity that contributes to the understanding of a phenomenon. In the case of Design Research, all or part of the phenomenon may be created as opposed to naturally occurring. Hence, as field studies enable behavioral-science researchers to understand organizational phenomena in context, the process of constructing and exercising innovative managerial artifacts enable design researchers to understand the problem addressed by the artifact and the feasibility of their approach to its solution (Hevner et al., 2004).
On this basis, the combination of the perspectives of IT performance management (as a wide managerial field) and design science and methods is experienced in this study and it is addressed as critical point to provide a knowledge foundation to build organizational practices that are sustainable and meet the needs of IT stakeholders. According to Hevner et al. (2004), during this study, evolving both the design process and the design artifact as part of the research are cognized. Hence, some modifications for the general process of Design Research method to apply in management field are suggested along with presenting designed artifacts of the IT performance management system (ITPMS).
DESIGN RESEARCH METHOD
Design Research as a method of doing scientific researches has been defined in several ways. A primary interpretation of Design Research is that it is concerned with undertaking research into the design process (Van Aken, 2007). Secondary interpretations would refer to undertaking research within the process of design (Vaishnavi and Kuechler, 2008). However, two basic concepts can be identified in the theoretical structure of this method as Design and research. Design as the core concept of the method should be investigated and also the relationships with Research concept should be addressed. Moreover, the steps and outputs of Design Research should be studied as a methodological aspect. And at last, role and history of applying design-based approaches such as Design Research in the field of management can be an interesting focus point.
Design: Designing is a natural, intuitive and creative process. Everyone on an ongoing daily basis is engaged in designing his or her actions regarding problems of everyday life (Van Aken, 2007). Design is both a process (set of activities) and a product (artifact)-a verb and a noun (Walls et al., 1992). This Platonic view of design supports a problem-solving paradigm that continuously shifts perspective between design processes and designed artifacts. The design process is a sequence of expert activities that produces an innovative artifact (Hevner et al., 2004).
According to Simon (1996), Design ... is the core of all professional training; it is the mark that distinguishes the professions from sciences. And in a perspective analogous to considering design as the crafting of an interface between inner (the set of components that make up the artifact and their relationships) and outer (the total set of external forces and effects that act on the artifact) environments, design can be thought of as a mapping from function space-a functional requirement constituting a point in this multidimensional space-to attribute space, where an artifact satisfying the mapping constitutes a point in that space (Takeda et al., 1990; Simon, 1996). Design, then, is knowledge in the form of techniques and methods for performing this mapping-the knowhow for implementing an artifact that satisfies a set of functional requirements (Vaishnavi and Kuechler, 2008). On the other hand, a design is a solution to a problem, but it is a special kind of solution. It is not a solution to a pure knowledge problem but a solution to a field problem. Solving an actual field problem, like bridging a river, not only entails the design of a solution but also realization of designed solution in material or social reality. A design is thus interface between the immaterial domain of thinking and communicating on one hand and the domain of material and social reality on the other. So, designs may refer to entities like actions, structures, processes, or systems in organizational contexts (Van Aken, 2007).
Design and research: The link between design and research is made primarily by the interest in design science. This interest may be traced back to two seminal books, Simon’s (1969, 1996) The Sciences of the Artificial and Schön’s (1983) The Reflective Practitioner. Simon revealed the fundamental difference between the natural sciences (explanatory sciences) and the sciences of the artificial. A natural science is a body of knowledge about some class of things-objects or phenomena-in the world (nature or society) that describes and explains how they behave and interact with each other. A science of the artificial, on the other hand, is a body of knowledge about artificial (man made) objects and phenomena designed to meet certain desired goals. Schön (1983) discussed the use of sciences of the artificial in the knowledge-intensive process of designing solutions to field problems as the core process in the professions.
On this basis, Simon (1969, 1996) beliefs that the primary mission of research in Design Sciences is to develop valid knowledge to support the design of solutions to field problems by competent professionals. In fact, design as a coherent discipline of study has its own things to know and its own ways of knowing them (Archer, 1979). Owen (1997) discusses the relation of design to research with reference to a conceptual map of disciplines (Fig. 1) with two axes: Symbolic/Real and Analytic/Synthetic. The horizontal axis of the map position disciplines according to their defining activities: disciplines on the left side of the map are more concerned with exploration and discovery. Disciplines on the right side of the map are characterized more by invention and making. The map’s vertical division (the symbolic/real axis) characterizes the nature of the subjects of interest to the disciplines-the nature of the phenomena that concerns the research community.
The disciplines that lie predominantly on the synthetic side of the map are either design disciplines or the design components of multi-paradigmatic disciplines. Design disciplines have a long history of building their knowledge base through making-the construction of artifacts and the evaluation of artifact performance following construction. Owen (1997) further presents a general model for generating and accumulating knowledge (Fig. 2) that is helpful in understanding design disciplines and the design science research process: Knowledge is generated and accumulated through action. Doing something and judging the result is the general model... the process is shown as a cycle in which knowledge is used to create works and works are evaluated to build knowledge. While knowledge building through construction is sometimes considered to lack rigor, the process is not unstructured. The channels in the diagram of the general model are the systems of conventions and rules under which the discipline operates.
| Fig. 1: | A map of disciplines (Owen, 1997) |
| Fig. 2: | A general model for generating and accumulating knowledge (Owen, 1997) |
They embody the measures and values that have been empirically developed as ways of knowing as the discipline has matured. They may borrow from or emulate aspects of other discipline’s channels but, in the end, they are special to the discipline and are products of its evolution (Vaishnavi and Kuechler, 2008).
Design research methodology: The development of Design Research has led to the establishment of design as a coherent discipline. Archer (1979) encapsulated the view in stating his belief that there exists a designerly way of thinking and communicating that is both different from common scientific and scholarly ways of thinking and communicating and as powerful as scientific and scholarly methods of enquiry when applied to its own kinds of problems. Regarding Van Aken (2007), Design Research (Design Science Research) is the research within a discipline aimed at developing general substantive and procedural design science to solve the field problems of that discipline. The two defining characteristics of Design Research are its interest in field problems and its solution focus which solves field problems with interventions or systems.
Takeda et al. (1990) have analyzed the reasoning that occurs in the course of a General Design Cycle (GDC) and this can be led to the general methodology of design science research (Fig. 3). Although there are many excellent diagrams of the process of design science research (Hevner et al., 2004; Purao, 2002; Gregg et al., 2001; March and Smith, 1995; Nunamaker et al., 1991), but the presented diagram has been chosen because it emphasizes the knowledge generation inherent in the method and because it originated in an analysis of the processes inherent in any design effort.
| Fig. 3: | The general methodology of design science research (Vaishnavi and Kuechler, 2008) |
Components of this model can be
described by Vaishnavi and Kuechler (2008).
| (i) | Awareness of problem: An awareness of an interesting problem can come from multiple sources: new developments in industry or in a reference discipline. The output of this phase is a proposal, formal or informal, for a new research effort |
| (ii) | Suggestion: The suggestion phase follows immediately behind the proposal and is intimately connected with it, as the dotted line around Proposal and Tentative Design (the output of the suggestion phase) indicates. Suggestion is an essentially creative step wherein new functionality is envisioned based on a novel configuration of either existing or new and existing elements |
| (iii) | Development: The Tentative Design is further developed and implemented in this phase. Elaboration of the Tentative Design into complete design requires creative effort. The techniques for implementation will of course vary, depending on the artifact to be constructed. The implementation itself can be very pedestrian and need not involve novelty beyond the state-of-practice for the given artifact; the novelty is primarily in the design, not the construction of the artifact |
| (iv) | Evaluation: Once constructed, the artifact is evaluated according to criteria that are always implicit and frequently made explicit in the Proposal (Awareness of Problem phase). Deviations from expectations, both quantitative and qualitative, are carefully noted and must be tentatively explained. That is, the evaluation phase contains an analytic sub-phase in which hypotheses are made about the behavior of the artifact |
The results of this phase and additional information gained in the construction and running of the artifact are brought together and fed back to another round of suggestion (cf. the Circumscription arrows of Fig. 3). Indeed, the evaluation of the artifact provides feedback information and a better understanding of the problem in order improve both the quality of the product and the design process. This build-and-evaluate loop is typically iterated a number of times before the final design artifact is generated. So, evaluation is a crucial component of the research process (Hevner et al., 2004; Markus et al., 2002).
The evaluation of designed artifacts in the Design Research method typically uses methodologies which are shown in Fig. 4. The selection of evaluation methods must be matched appropriately with the designed artifact and the selected evaluation metrics (Hevner et al., 2004).
| (v) | Conclusion: This phase is the finale of a specific research effort. Typically, reaching to this phase implies a satisfying output; that is, although there are still deviations in the behavior of the artifact from the revised hypothetical predictions, the results are adjudged good enough |
New knowledge production is indicated in Fig. 3 by the arrows labeled Circumscription and Operation and Goal Knowledge. The Circumscription process is especially important in understanding Design Research because it generates understanding that could only be gained from the specific act of construction.
| Fig. 4: | Design evaluation methods (Hevner et al., 2004) |
Further, the applicability of knowledge can only be determined through the detection and analysis of contradictions. This happens many times-not due to a misunderstanding of the theory, but due to the necessarily incomplete nature of any knowledge base. The design process, when interrupted and forced back to Awareness of Problem in this way, contributes valuable constraint knowledge to the understanding of the always-incomplete-theories that abductively motivated the original design (Vaishnavi and Kuechler, 2004; Takeda et al., 1990).
Even within Design Research communities there is lack of consensus as to the precise objective-and therefore the desired outputs-of Design Research (Vaishnavi and Kuechler, 2004). March and Smith (1995) contrasting Design Research with natural science research, propose four general outputs for Design Research: constructs, models, methods and instantiations. Constructs are the conceptual vocabulary of a problem/solution domain. In fact, constructs provide the language in which problems and solutions are defined and communicated (Schön, 1983). A model is a set of propositions or statements expressing relationships among constructs (March and Smith, 1995). Models use constructs to represent a real world situation, the design problem and its solution space (Simon, 1996). Hence, models aid problem and solution understanding and frequently represent the connection between problem and solution components enabling exploration of the effects of design decisions and changes in the real world (Hevner et al., 2004).
A method is a set of steps (an algorithm or guideline) used to perform a task (Vaishnavi and Kuechler, 2008). Methods are goal directed plans for manipulating constructs so that the solution statement model is realized (March and Smith, 1995). Methods define processes and provide guidance on how to solve problems, that is, how to search the solution space. These can be range from formal, mathematical algorithms that explicitly define the search process to informal, textual descriptions of best practice approaches, or some combination (Hevner et al., 2004).
The final output from a Design Research effort in March and Smith’s (1995) explication is an instantiation which operationalizes constructs, models and methods. It is the realization of the artifact in an environment (Vaishnavi and Kuechler, 2004). Emphasizing the proactive nature of design research, they point out that an instantiation sometimes precedes a complete articulation of the conceptual vocabulary and the models (or theories) that it embodies. Instantiations show that constructs, models, or methods can be implemented in a working system. They demonstrate feasibility, enabling concrete assessment of an artifact’s suitability to its intended purpose. They also enable researchers to learn about the real world, how the artifact affects it and how users appropriate it (Hevner et al., 2004).
Rossi and Sein (2003) and Purao (2002) in an ongoing collaborative effort to promote Design Research have set forth their own list of Design Research outputs. All but one of these can be mapped directly to March and Smith’s (1995) list. Their fifth output, better theories, is highly significant. Design research can contribute to better theories (or theory building) in at least two distinct ways. First, since the methodological construction of an artifact is an object of theorizing for many communities (e.g., how to build more maintainable software), the construction phase of a Design Research effort can be an experimental proof of method or an experimental exploration of method or both. Second, the artifact can expose relationships between its elements. It is tautological to say that an artifact functions as it does because the relationships between its elements enable certain behaviors and constrain others. However, Table 1 shows the outputs that can be obtained from a Design Research effort (Vaishnavi and Kuechler, 2008, 2004).
The role of design research in the field of management: Simon (1969) has chosen to regard the field of management as one of sciences of the artificial. On the other side, Thompson (1956) defined the management field as a design science, explicitly comparing it with engineering.
| Table 1: | The outputs of design research (Vaishnavi and Kuuechler, 2008, 2004) |
Although the main stream of management research has developed as more of an explanatory science (Emory, 1985) and it seems that procedural design views have been neglected largely in performance measurement and management (the works like Neely et al., 2000 are exceptional) and IT performance management researches (Mohrman, 2007), but regarding Van Aken (2007, 2005), this does not mean that there have not been any design approaches in the field of management in general and operations research, system approach, soft system methodology, sociotechnical design, requisite organization and business process redesign, are some examples of those. So, it can be said that design perspectives have a rich tradition in organizational notions (Mohrman, 2007).
Design is also implicit in the broader fields of organizational psychology and sociology, where some streams of research have developed important notions like that knowledge and social order are socially constructed in practice (Berger and Luckmann, 1966; Bourdieu, 1990; Giddens, 1993). In addition, there are studies that imply the need for design is the strategy-based literature on organizational capabilities (Dosi et al., 2000). Moreover, some research in the field of information systems and technology (with managerial orientations) has been focused on using design perspectives (Weber, 1987; March and Smith, 1995; Hevner et al., 2004; Vaishnavi and Kuechler, 2004, 2008; Kuechle et al., 2007) All of these researchers can lead to a corollary notion that organizations require dynamic capabilities and abilities to integrate, build and reconfigure internal and external competencies to address rapidly changing environments (Teece et al., 1997) and this implies the ability to intentionally design contexts, structures, processes and systems (Mohrman, 2007).
However, it has been argued that management and design are inseparable (Orlikowski, 2004) and that design is a discipline that should be fundamental to the professional education of managers (Mohrman, 2007; Mohrman et al., 1997). Also, Boland and Collopy (2004) argue that managers need a design attitude-an attitude that views each project as an opportunity for invention that includes a questioning of basic assumptions and a resolve to leave the world a better place than we found it.
RESEARCH METHODOLOGY
This research was conducted at December 2007 to September 2008 has been structured based on applying the Design Research methodology as a research method to form a new solution (system) in IT performance management field. Given the steps and outputs of Design Research method that are described earlier, a new approached solution for IT performance management is designed, developed and evaluated as a case application of the method to IT management scope. In fact, considering the exploratory approach of this research, it is required to perform at least one case study to assess the utility of selected research method in IT management area and it is therefore a research cycle in which the methods like Design Research should be tested in practice and in turn it helps further adjustments of underlying theories of those.
Thus, after the survey on Design Research method in above, the next step is concerned with developing case of application of the method.
APPLYING DESIGN RESEARCH METHOD TO DESIGN AN ITPMS
IT organizations are struggling to measure their own performance due to the enormous size of their expenditure. And as a result managers have faced growing pressure to measure the performance of IT departments (Gomolski, 2004; Son et al., 2005). In fact, as investments in information technology continues to increase and IT is increasingly becoming crucial to achieve organizational and strategic goals, most of key stakeholders like Boards of Directors, executive management, CIOs audit and regulatory bodies and IT professionals are becoming increasingly focused on the issue of ensuring value is received for IT investments (Saull, 2000; Van Grembergen and Van Bruggen, 1997). On this basis, many models and frameworks have been emerged for managing the performance of IT during the last decades (Hallikainen et al., 2006; Lomerson and Tuten, 2005; Atkinson, 2004; Kang and Bradley, 2002; Van Grembergen et al., 2003; Van Grembergen and Van Bruggen, 1997; Nissen, 1994). But despite significance of addressing all of actual stakeholders in the process of ITPM -as some researcher like Saull (2000) has discussed- most of models and approaches ignore this issue. Investigating the literature has shown that most of models and frameworks have addressed only one or two group of IT stakeholders (often including Business and or Users-Customers). Hence a critical improvement opportunity was detected and has motive us to innovatively bring the concept of IT stakeholders and their expectations and contributions into the subject area of ITPM.
However, exploiting the design view of Design Research method and undertaking research within the process of designing an ITPMS led to first version of noted system in this research. Acting upon the described steps of Design Research method, valuable experiences and lessons learned has been made which are explained in the following.
Awareness of problem and suggestion: Awareness of problem and suggestion as two connected steps in general methodology of Design Research has been conducted in the first phase of the research. Awareness of past researches and studies in the area of ITPM has led to a good familiarity with gaps and research opportunities and has been made the basic framework of the research. The output of this step as it is shown in Fig. 3 was the Proposal.
Suggestion as the second step was concerned with elaborated investigation of IT performance management literature. In this regard, the need for evaluating and managing the performance of information technology has been reviewed and frameworks like IT governance and its related concepts and practices have been addressed (Robinson, 2005; Van Grembergen, 2000). Then, four general approaches in ITPM (Financial, IT Value Management, IT Service Management and IT Balanced Scorecard) have been detected based on a detailed literature survey. This survey showed the key gaps and some problems in design of earlier ITPM models and approaches. In addition, some theoretical foundations has been studied which could help to form the basics of suggested approach to ITPM. Value Based Management (VBM), Stakeholder orientation (Bryson, 2004; Emiliani, 2001) and new generation of performance management approaches like Value Mapping and Performance Prism (Green and Jack, 2004; Jack, 2002; Neely et al., 2001), were issues that have been perused.
Regarding detected areas for improvement and based on considered theoretical subjects, the Tentative Design (as the output of Suggestion step) of the ITPMS was presented which includes three first main outputs of the Design Research method according to Table 1: concepts (constructs), conceptual model (models) and algorithm (methods). Generally, 19 concepts have been addressed in the tentative design of suggested system. The suggested design of conceptual model shows the relationships between concepts while its theme has adopted from kind of relations in formal system model of checkland’s (1999) soft systems methodology (Fig. 5).
| Fig. 5: | The tentative design of conceptual model of the suggested ITPMS |
| Fig. 6: | The tentative design of algorithm of the suggested ITPMS |
Also, the Algorithm which shows the usage method and application steps of suggested system is shown in Fig. 6.
Development: The aim of development step is to create a working and utilizable Artifact as an output. Beside of this, managerial systems are not tangible and physical artifacts like engineering ones. Hence, frameworks, models or algorithms in such systems form the one primary side of the system’s artifact which can be called as Theoretical Artifact. This kind of artifact can be supposed as output of a theoretical development sub step. On the other side, Practical Artifact of these systems can be considered as organizational development instances which are the output of a practical development sub step.
In this research, the theoretical product of suggested system was constructed through a validation process by surveying the viewpoints of 14 experts of the field. For this purpose, an elaborated questionnaire was developed which was covered all aspects of proposed Tentative Design of the system. This questionnaire has included structured questions along with open ones. By statistical analysis of responses to structured questions, the totality of system has been approved. Also, the content analysis of descriptive comments of experts has helped to gather useful points about the partial bugs of Tentative Design and led to some valuable modifications in that. Thus the Theoretical Artifact of the system was achieved and it was prepared for developing in an organizational setting.
As it is shown in Fig. 3, the development step has a key role in circumscription of awareness of problem because of useful experiences which can be achieved through the constructing the proposed design in a real world condition. Therefore, practical development of suggested ITPMS was conducted for the primary aim of acquiring experiences to clear the space of problem and improve the proposed design. One of big governmental organizations in Iran’s industry sector was selected and the system was developed in IT unit of that. This unit encompasses six staffs (the unit’s manager is counted) and serves 142 users in central office. The as-is circumstances of the unit showed a serious need for changing the practices and methods of evaluating and managing the performance. So, initiation of suggested system was stressed by top managers of the organization. All steps of algorithm of proposed ITPMS was completely performed in the unit until it was reached up to step 10. In fact, further development of the system required more time and also some changes in operational programs of the unit which made it impossible to continue. However, this scale of development had its own valuable lessons for next iteration of Design Research effort.
Evaluation: As it is known, Evaluation is a crucial step in the Design Research method that makes a significant part of knowledge and lessons about the designed artifact and the problem area. The designed ITPMS in this research was evaluated through the observational approach and by case study method according to Fig. 4. Indeed, the proposed system has been developed and then studied in depth in an organizational environment. This study has been performed to investigate the operational specifications in comparison with expectations. So, structured interviews have been conducted with 4 staffs of the IT unit in which they have been questioned about many aspects of the ITPMS. Results of these interviews have shown that the whole of system is suitable but some problems of weak presentation and incomplete development of the system must be resolved.
Conclusion: In this research, an iteration of Design Research action has been accomplished. Generally, what that can be addressed as the primary result of doing this iteration, is knowledge and experiences which were extracted from development and evaluation steps. These lessons learned can significantly help to transparently define the problem scope and improve design of the system in next iterations. Although another important result of acting upon the Design Research methodology is first version of system’s Theoretical Artifact.
DISCUSSION AND CONCLUSION
In today’s world there are many indications that we need to do a much better job of understanding and crafting the structures, processes and systems through which organizations determine purposes and objectives; plan the ongoing performance; assemble, organize and use resources. The increasing complexity and interdependence of the global economy, the opportunities and challenges presented by pace of technology advance and pending shortages of resources specially in developing countries (e.g., Iran), are incidents that calls for the design of customized solutions to complex and local problems and affords an unprecedented opportunity for academics and practitioners schooled in organizational and managerial disciplines. On this basis, it would be sound that academics and practitioners who research to manage IT organizations might also benefit from a design attitude and should attend to how knowledge of the academy and the profession can be joined with the knowledge of practice to create innovative solutions.
This study was a new experience of applying a design viewpoint in the field of IT performance management. It led to a creative thinking about gaps and problems in existent approaches and models of ITPM and on this basis a new and more comprehensive design to solve those challenges was proposed. This new design can be matured through more several iterations of Design Research cycle. On the other hand, tangible artifact of the proposed system was constructed via a limited field development which partly showed the practical capabilities of that, by observational evaluation. But this development and evaluation efforts should be continued in the future by performing more iterations of design cycle and studying more cases from a wide range of industries and businesses.
Moreover, although applying design attitude has resulted in developing a new solution for IT performance management but other side of this research has caused to some implications for reviewing the general methodology of Design Research. In fact, designing soft systems like managerial ones is completely different from hard and more tangible engineering artifacts. This requires a new perspective to design steps and processes and may lead to some changes in them; while one of examples of this change is proposed to divide the development step into two sub steps: theoretical development and practical development. Accordingly, further researches are planned to investigate key requirements of designing soft artifacts and systems and to reflect them in a method of design which can be called as: Soft Design Research.