by
Barry L. Myers
Visiting Assistant Professor
University of Southern Colorado
2200 Bonforte Blvd.
Pueblo, CO 81001-4901
(719) 549-2844 Fax: (719) 549-2519
bmyers@uscolo.edu
Leon A. Kappelman
Associate Professor
Associate Director, Center for Quality and Productivity
Business Computer Information Systems Department
College of Business Administration
University of North Texas
Denton, Texas 76203-3677
(940) 565-3110 Fax: (940) 565-4935
kapp@unt.edu
Website:http://www.year2000.unt.edu/kappelma/
Victor R. Prybutok
Associate Professor of Management Science
Director of the Center for Quality and Productivity
Business Computer Information Systems Department
College of Business Administration
University of North Texas
Denton, TX 76203-3677
(940) 565-3110 Fax: (940) 565-4935
prybutok@unt.edu
Versions of this manuscript appear in the Information Resources Management Journal (Winter 1997) and in a forthcoming book, Information Systems Success Measurement, edited by E.J. Garrity & G.L. Sanders (1997, Idea Group Publishing)
This work was funded in part by the University of North Texas Information Systems Research Center and the College of Business Administration
A Comprehensive Model for Assessing the
Quality and Productivity
of the Information Systems Function: Toward a Contingency
Theory for Information Systems Assessment (ISRC-WP-19970101)
Author Biographies:
BARRY L. MYERS is a doctoral candidate in the Interdisciplinary Ph.D. Program in Information Science at the University of North Texas and a visiting assistant professor of computer information systems at the University of Southern Colorado. He received a Master of Science degree in statistics from Oklahoma State University and has extensive experience in the management of information systems, systems analysis and design, and in quality management and improvement His current research interests include MIS evaluation and assessment and data quality management.
LEON A. KAPPELMAN, Ph.D. is an associate professor of Business Computer Information Systems in the College of Business Administration at the University of North Texas, and Associate Director of the Center for Quality and Productivity. His professional interests include the management of information assets; information systems development and implementation; and benchmarking, measurement, evaluation, and assessment. He has published over two dozen journal articles. His work has appeared in the Communications of the ACM, the Journal of Management Information Systems, the DATA BASE for Advances in Information Systems, the Journal of Systems Management, the Journal of Computer Information Systems, Information Week, National Productivity Review, Project Management Journal, the Journal of Information Technology Management, Industrial Management, as well as other journals and conference proceedings. He authored Information Systems for Managers, McGraw-Hill (1993). He is Co-Chair of the Society for Information Management (SIM) Year 2000 Working Group.
VICTOR R. PRYBUTOK, Ph.D. is the Director of the University of North Texas Center for Quality and Productivity and an Associate Professor of Management Science in the Business Computer Information System Department. He has published articles in American Statistician, Operations Research, Economic Quality Control, Quality Progress, as well as other journals and conference proceedings. Dr. Prybutok is a senior member of the American Society for Quality Control (ASQC), an ASQC Certified Quality Engineer, a Certified Quality Auditor, and a 1993 Texas Quality Award Examiner. His current research interests include project management, assessment of quality programs, neural networks, and MIS evaluation and assessment.
A Comprehensive Model for Assessing the Quality
and Productivity
of the Information Systems Function: Toward a Contingency
Theory for Information Systems Assessment (ISRC-WP-19970101)
Abstract: Information Systems (IS) managers are under increasing pressure to justify the value and contribution of IS expenditures to the productivity, quality, and competitiveness of the organization. This paper examines the need for IS assessment and suggests a comprehensive, IS assessment framework linked to organizational performance using existing IS assessment theory as a base and incorporating measurement concepts from other disciplines. The existing models of IS success are updated to include the emerging IS success dimensions of service quality and work group impact and provide a comprehensive method for organizing the various measures of IS success. In addition, many new measures from recent research are presented to supplement the lists supplied by previous research. Additional research is suggested to advance the IS assessment contingency theory. Such a theory has the potential to contribute to the quality and productivity of the IS function and the larger organization by providing feedback to manage and improve the IS function to better meet the needs of the organization.
A COMPREHENSIVE MODEL
FOR ASSESSING THE QUALITY AND PRODUCTIVITY
OF THE INFORMATION SYSTEMS FUNCTION: TOWARD A CONTINGENCY
THEORY FOR INFORMATION SYSTEMS ASSESSMENT (ISRC-WP-19970101)
Information Systems (IS) managers are under increasing pressure to justify the value and contribution of IS expenditures to the productivity, quality, and competitiveness of the organization. IS assessment is not well established and recent studies show that more research is needed (Clark, 1992; DeLone & McLean, 1992; Dickson, Wells & Wilkes, 1988; Saunders & Jones, 1992). This paper examines the need for IS assessment and suggests a comprehensive, IS assessment framework linked to organizational performance using existing IS assessment theory as a base and incorporating measurement concepts from other disciplines. Further, a theoretically-based, comprehensive set of IS assessment measures and a contingency theory for selecting appropriate measures is presented that will provide IS managers with the guidance necessary to develop their own IS assessment systems. These assessment systems have the potential to furnish the feedback required to enhance the quality and productivity of the IS function and thereby, the organization.
IMPORTANCE OF TOPIC
Frequently, information technology is used without a full understanding of its applicability, effectiveness, or efficiency. IS managers often lack the tools they need to decide if they are accomplishing the right activities (Davis & Hamann, 1988). In addition, these managers often fail to learn if they are meeting the needs of their customers. The productivity of the information systems function has proven difficult to define and measure (Scudder & Kucic, 1991). "Assessing the value of the IT infrastructure is perhaps the biggest single problem for the 90s - the information technology organization is running out of credibility and managers are no longer willing to give us the benefit of the doubt" (Rochester & Douglass, 1991, p. 16). "Companies have come to realize they are paying big money for technology that isn't being used" (King, 1991, p. 73). Furthermore, a recent survey of senior executives from 220 Fortune 1000 firms found extremely low satisfaction with returns on corporate technology investments. Over 81 percent of those polled rated their organization's payback on technology spending as minimal or average (Maglitta, 1993).
No single measure of the value of the IS function has appeared (Carlson & McNurlin, 1992a). "Measuring IS effectiveness" is consistently reported in the top 20 on the list of most-important IS issues by the members of the Society for Information Management (SIM), an organization of IS executives (Ball & Harris, 1982; Brancheau & Wetherbe, 1987; Dickson, Leitheiser, Nechis & Wetherbe, 1984; Niederman, Brancheau & Wetherbe, 1991). In fact, effectiveness of the IS function has proven practically impossible to define and measure (Niederman et al.). Many possible explanations for this difficulty are available. For example, the role of the IS function in business performance can be subtle and difficult to differentiate from other factors (Crowston & Treacy, 1986; Niederman et al.). Some companies use weak 'surrogate' measures of IS effectiveness that hide the true value of the IS function(Niederman et al.). Others depend mostly on qualitative rather than quantitative measures (Hartog & Herbert, 1986; Marion, 1992; McLean, Kappelman & Thompson, 1993). Some researchers believe that the lack of evidence of a payoff for the high investment in technology could be interpreted as reflecting serious measurement deficiencies (Baatz, 1994; Berndt & Morrison, 1991; Brynjolfsson, 1993).
Evidence suggests that poor performance of the IS function is a serious inhibitor to good business performance (Carlson & McNurlin, 1992a). Carlson and McNurlin (1992a) also found evidence in several of the organizations they studied that high IS effectiveness is associated with high organizational performance. Others report a clear connection between assessment and productivity (Tayntor, 1994). Better use of information, both internal and external, relates positively to profitability (Strassman, 1990).
Assessment is essential to supply the feedback needed for the effective management and continuous improvement of the IS function. "Just as a human being needs a diversity of measures to assess his or her health and performance, an organization needs a diversity of measures to assess its health and performance" (Drucker, 1989, p. 230). Systematic measurements are needed to guide management action. Without quantitative feedback, managers are dependant upon only experience, intuition, and judgement. As firms become more complex, global, and fast-paced, relying on experience and intuition alone is increasingly problematic (Singleton, McLean & Altman, 1988).
Managers define what is important to the organization and manifest corporate culture in their assessment choices (Eccles, 1991; Strassman, 1990; Tsui, 1994). "What gets measured gets attention" (Eccles, p. 131). The relationship between IS performance and organizational performance should be more carefully explored (Weill & Olson, 1989). It is clear that IS assessment is vital to the organization. Also, IS executives need a comprehensive framework for assessment tied to organizational performance to aid them in developing IS assessment systems.
EARLY WORK
Articles discussing the need to assess the contribution of the IS function to the organization began appearing in the late 1970s (King & Rodriguez, 1978; Matlin, 1977; Rolefson, 1978). Early research into assessing the value of the IS function concentrated on economic considerations and introduced the idea that multiple assessment measures were essential to develop a clear picture (Ahituv, 1980; Bender, 1986; King & Schrems, 1978; Matlin, 1979). Most early attempts at assessing the IS function centered on measures of system availability and performance. For example, Borovits and Neumann (1979) described several indices of performance: capacity, response time, throughput rate, overhead percentage, software time measures, reliability measures, system utilization measures, raw speed, and availability. They and others also presented in-depth procedures for system evaluation (Ein-Dor & Jones, 1985).
McLean (1973) was one of the first to call for a shift from a measurement focus on efficiency to effectiveness; in other words, doing the right thing rather than doing the thing right. To do this would require computer professionals to measure and pursue organizational objectives, in addition to pursuing their internal departmental goals. Efficiency and effectiveness are different and require different measures. An efficient IS function is not necessarily an effective one. Efficiency focuses on internal requirements of the IS function, while effectiveness requires an external focus. An example of an IS function efficiency measure is the number of tasks completed per unit of time. An effective IS function, for example, is concerned about the impact of the information provided in helping users do their jobs. Efficiency is concerned with doing things right; effectiveness is concerned with doing the right things (McLean). Lucas (1972) introduced the idea of including users when assessing the IS function. Others began evaluating various measures of system effectiveness and considering the different viewpoints of the evaluators (Hamilton & Chervany, 1981a; 1981b).
ASSESSMENT METHODS & PROCEDURES
Considerable literature exists that proposes methods and offers recommendations for developing assessment systems. The first and most important point to consider when developing measures is to align all measures of effectiveness with corporate objectives (Thierauf, 1988). This should follow easily once the IS function is aligned with the strategic direction of the corporation (Mendelow, 1983). The goal is to couple vision with performance (Cross & Lynch, 1992) in order to aid the IS function in staying aligned with the corporation in a very complex, ever-changing environment.
Many authors stressed that measures
should be easy to implement and understand (AT&T Quality
Steering Committee, 1990a; 1990b; Blenkinsop & Burns, 1992;
Eccles, 1991; Lefrancois, 1984). For example, Lefrancois (1984)
said any evaluation system should have a basis of measurement
that is "readily understood, simple to implement, easy to
administer, and clearly cost effective" (p. 58). The
AT&T Quality Steering Committee (1990b) suggested the
following criteria for effectiveness measures:
The IS function is in the business of serving customers, the end-users. Customers buy or use a service based on the value or benefit it provides. Creating value for customers requires a thorough understanding of their requirements and expectations, and the ability to translate this understanding into concrete service objectives to drive business activities (AT&T Quality Steering Committee, 1990a, p. 4).
Others discussed the need to balance internal, cost-based measures with process and product measures when developing an assessment system (Anonymous, 1993; Band, 1990b; Thornburg, 1991). LaPlante and Alter (1994) addressed the need to use measures that embody senior general managers' definition of value and to make continual surveys of end users an integral part of the way the IS function is managed. Gatian (1994) tested the question "is user satisfaction a valid measure of IS effectiveness?" and found support for the relationship. While it is important to know that this relationship exists, user satisfaction is just one measure of the effectiveness of the IS function and assessing it alone is not sufficient to determine the overall effectiveness of the IS function. Ensuring that each measure is appropriate for, or that it fits the organization is discussed by many writers (Cameron, 1978; 1980; Goodman & Pennings, 1977; Scott, 1977; Singleton et al., 1988) and will be presented in more detail in the following section on organizational effectiveness.
ORGANIZATIONAL EFFECTIVENESS
Extensive work has been done in attempting to define and measure organizational effectiveness. Steers (1975) reviewed the organizational effectiveness literature and noted a distinction between univariate and multivariate performance measures. He compared seventeen multivariate models in the literature. He found a lack of consensus about what constitutes a useful and valid set of effectiveness measures and very little overlap across the various approaches (Campbell, 1977; Goodman & Pennings, 1977). Cameron and Whetten (1983) asserted that no single, universal model of effectiveness is possible. Based on his analysis, Steers (1975) identified eight general problems in assessing organizational effectiveness: (1) Construct validity; (2) criterion stability; (3) time perspective; (4) generalizability; (5) theoretical relevance; (6) multiple criteria; (7) precision of measurement; and (8) level of analysis. Since effectiveness is often so hard to define and measure, Cameron (1984) suggested a model of organizational ineffectiveness. Its basic assumption is that it is easier, more accurate, more consensual, and more beneficial for organizations to identify ineffectiveness (problems or faults) than it is to identify criteria of effectiveness (competencies). An organization is viewed as having achieved effectiveness when it is free from characteristics of ineffectiveness.
Cameron (1984) also presented a tabular comparison among seven major models of organizational effectiveness, giving definitions for each model and describing the conditions under which each is the most useful. For example, the goal model of effectiveness declares that an organization is effective when it accomplishes its stated goals. This model is only useful when goals are measurable and time-bound. Other models listed include the system-resource model, internal process model, and strategic-constituencies model. Different models of effectiveness are useful in different circumstances and it is important to consider the fit of the model to the organization being measured (Lewin & Minton, 1986; Melone, 1990). Cameron (1980) suggested 6 critical questions that must be considered in assessing effectiveness, subsequently expanded to 7 questions by Cameron (1984) and 7 guidelines for assessing organizational effectiveness by Cameron and Whetton (1983). These 7 guidelines are listed below:
Guide 1: From
whose perspective is effectiveness being judged?
It is important to make explicit who is defining and assessing
effectiveness, since each constituency will use different
criteria.
Guide 2: On
what domain of activity is the judgment focused?
The customer, process, and output/service define the domain being
judged and it's important that this be explicitly stated, since
many different domains exist in organizations and each one should
be judged differently.
Guide 3: What
level of analysis is being used?
Effectiveness judgments can be made at many levels: individual,
subunit, organizational, industry, societal. The appropriateness
of the level depends on the constituency being used, the domain
being focused on, the purpose of the evaluation, etc.
Guide 4: What
is the purpose for judging effectiveness?
The judgment almost always is affected by the purpose(s).
Different data will be available, different sources will be
appropriate, different amounts of cooperation or resistance will
be encountered, different strategies will be necessary based on
differences in purpose. The purposes also help determine
appropriate constituencies, domains, levels of analysis, etc.
Guide 5: What
time frame is being employed?
Long-term effectiveness may be incompatible with short-term
effectiveness, and sometimes effects and outcomes cannot be
detected using the wrong time frame, since they may occur
suddenly in the short term, or incrementally over the long term.
The time frame should be made explicit.
Guide 6: What
type of data are being used for judgments of effectiveness?
Objective data or subjective, perceptual data? Objective data
will tend to be more reliable, more easily quantifiable, and more
representative of the 'official' position. These also limit the
scope and usefulness of the data. Subjective data allows
assessment of a broader set of criteria, but can be biased, and
lack validity and reliability.
Guide 7: What
is the referent against which effectiveness is judged?
Comparing competitors, comparing to a standard, comparing to the
organizational goals, comparing to past performance, or
evaluating on the basis of characteristics the organization
possesses are all possible methods for comparison. Each one will
yield different effectiveness judgments; therefore, the referent
being used should be made clear.
Carefully considering these guidelines "should help clarify the meaning of organizational effectiveness in each type of evaluation and guide evaluators in the selection of appropriate criteria" (Cameron, 1980, p. 79).
Cummings (1977) described a different problem: "Effectiveness may be seen by many successful managers (those who somehow get to the top of their organizations and make the most money within their organizations) as best defined in process. The object is to end up being good at what is measured" (p. 58). This problem is also discussed by Likert (1967). Being able to collect accurate measurements in an organization depends on how the results are used over time. Punitive use of measurements is feared by all levels of the hierarchy, except at the very top, and to protect themselves, employees will do whatever is necessary, especially with end-results measurements, to force the data to look favorable to them.
Anthony (1965) identified three levels of management that occur in all organizations: operational, managerial, and strategic. At each level, different measures are appropriate. At the operational level, efficiency and productivity are the key. At the managerial level, the effectiveness of the organization and management becomes essential.Finally, at the strategic level, the competitiveness of the enterprise itself is of central concern. A balanced assessment system will include measures of each organizational level.
Campbell (1977) provided an extensive list of criterion measures or variables proposed in the literature as indices of organizational effectiveness (see Table 1). He emphasized the need for organizations, as well as researchers, to adopt a theory or model of effectiveness. They must also know the mission of the organization and the organizational objectives for each process or task. These requirements must be met before measures of goal attainment are developed. "If a systematic analysis of task objectives can be made, the measurement problems will be substantially solved" (Campbell, 1977, p. 49).
Table 1 - Measures of Organizational Effectiveness
(Campbell, 1977, pp. 36-39)
-----------------------------------------------------------------------------------------------
1. Overall Effectiveness (general
evaluation by knowledgeable judge)
2. Productivity (can be measured at
individual, group, and total organization via records
or ratings or
both)
3. Efficiency
4. Profit (return on inventory (ROI) or
return on sales (ROS) are sometimes used
as alternative
definitions)
5. Quality
6. Accidents
7. Growth
8. Absenteeism
9. Turnover
10. Job Satisfaction
11. Motivation
12. Morale
13. Control
14. Conflict/Cohesion
15. Flexibility/Adaption
(Adaption/Innovation)
16. Planning and Goal Setting
17. Goal Consensus
18. Internalization of Organizational
Goals
19. Role and Norm Congruence
20. Managerial Interpersonal Skills
21. Managerial Task Skills
22. Information Management and
Communication
23. Readiness (military)
24. Utilization of Environment
25. Evaluations by External Entities
26. Stability
27. Value of Human Resources
28. Participation and Shared Influence
29. Training and Development Emphasis
30. Achievement Emphasis
-----------------------------------------------------------------------------------------------
Cameron and Quinn (1988) presented the various paradoxical situations that exist in successful organizations. A paradox occurs when two mutually exclusive, seemingly incompatible situations exist side-by-side. When measuring performance, they stressed the need to include measures of effectiveness and ineffectiveness to account for paradox. For example, consider defining the condition of excellent physical health or wellness. Some indicators might include low percentage of body fat, low blood pressure, cardiovascular fitness, etc. High scores on these indicators might suggest wellness, but low scores do not necessarily mean illness. Independent criteria are needed to indicate illness, such as bleeding, nausea, fever, etc. Illness and wellness could exist simultaneously in the same person. The same is true for the IS function, as well as the organization; it could be both effective and ineffective simultaneously. Therefore, both conditions should be accounted for in an assessment system.
The point Quinn and Cameron (1988) emphasized about paradox is the importance for organizations to balance the opposing sides of paradox, i.e., to not go too far one way or the other. Paradox is good and aids in organizational effectiveness, but unbalanced paradoxical situations will yield an ineffective organization. A balanced organization has the potential to increase productivity, goal clarity, stability, participation, commitment, morale, innovation, growth, and so on, while taking any of these positive characteristics to the extreme can cause a negative situation. For example, stability and control may turn into habitual perpetuation and ironbound tradition; or innovation and change become premature responsiveness and disastrous experimentation; and a well-ordered hierarchy might become a frozen bureaucracy. "In each case, by pursuing good through too narrow a frame, unintended negative consequences are created" (Quinn & Cameron, 1988, p. 306). They said, as have others, that it is difficult to perform this balancing act and that only a few will develop mastery, and only then through painful experience (Cameron, 1986b; Meyer & Gupta, 1994; Miller, 1992; 1993).
What application of these principles can be made for IS? By applying the precepts of the general system model described by Nolan and Wetherbe (1980) to the IS function, the overall IS function assumes the role of system and the various processes and activities of the IS function are then considered subsystems. Only optimizing one subsystem may harm the system, making it less effective. Also, a set of IS effectiveness measures for the system, covering each subsystem, counterbalance each other. For example, minimum labor per unit, maximum quality, and maximum job satisfaction as measures must be balanced as a group in a given context. Attempting to reduce labor per unit to its lowest possible level will certainly cause reduced quality and job satisfaction.
The literature indicates that the assessment of organizational effectiveness is a difficult task. Yet, using the excellent suggestions of the research presented, organizational effectiveness can be defined and assessment systems developed. From assessment guidelines (Cameron & Whetten, 1983) to extensive lists of possible measures of effectiveness (Campbell, 1977), the organizational effectiveness literature provides substantial support for the establishment of assessment systems of organizational effectiveness.
QUALITY MANAGEMENT
The quality management field is another field that offers extensive advice for the assessment of organizations. The Malcolm Baldridge National Quality Award: 1995 Award Criteria (National Institute of Standards and Technology, 1995) provided a complete set of criteria to be used in developing a quality management system, including leadership, information and analysis, strategic planning, human resource development and management, process management, business results, and customer focus and satisfaction. While these criteria are not specific to the IS function, an organization that prepares itself to compete for the award will likely be well on its way to adequately assessing all areas of the business, including the IS function. In discussing the quality measurement systems of previous Baldrige Award winners, Curt Reimann, the director of the Baldridge program, said that "the winning companies measure all their processes. Companies that aren't doing as well have limited measurements and limited access to comparative measurements" (Lakewood Publications, 1990, p. 4). The Baldridge Award criteria provide an excellent example of an organization-wide, quality assessment system, but it lacks adequate guidance for the development of a comprehensive, IS assessment system.
The measurement of customer satisfaction must become a driving force for every organization serious about quality improvement. This opinion is echoed repeatedly in the literature (Asbrand, 1993; AT&T Quality Steering Committee, 1990a; Davis, 1991; Doll & Ahmed, 1985; Marcolin & Higgins, 1992; Reichheld & Sasser, 1990; Stearns, 1984; Strassman, 1985). "Quality measures represent the most positive step taken to date in broadening the basis of business performance measurement" (Eccles, 1991, p.133). Improving the methods of measurement of customer satisfaction has been addressed by both the quality literature (Feigenbaum, 1983; Garvin, 1988; National Institute of Standards and Technology, 1995; Ross, 1993) and the IS literature (Anonymous, 1993; Bailey & Pearson, 1983; Bakar, 1994; Band, 1990a; 1990b; Baroudi & Orlikowski, 1988; Berry & Parasuraman, 1992; Carey, 1993; Conrath & Mignen, 1990; Doll & Ahmed, 1985; Doll, Raghunathan, Lim & Gupta, 1995; Galletta & Lederer, 1989; Gatian, 1994; Gemoets & Mahmood, 1990; Guimaraes & Gupta, 1988; Ives, Olson & Baroudi, 1983; Mahmood & Becker, 1985; Meachim, 1994; Melone, 1990; Moad, 1989; Parasuraman, Berry & Zeithaml, 1991a; Parasuraman, Zeithaml & Berry, 1994b; Peppers & Rogers, 1995; Raymond, 1987; Richardson, 1994; Rushinek & Rushinek, 1986a; 1986b; Rust, Zahorik & Keiningham, 1995; Torkzadeh & Doll, 1993).
Moore (1992) recommended viewing the organization as a system and developing process interface diagrams. These can be a "tool for communicating, eliminating barriers, understanding the relationships within the organization, planning, measuring processes, and responding to suggestions for improvement from customers, suppliers, and employees" (p. 1). He related the importance of goals to effective measurement when they are continually monitored against organizational objectives, customers' needs, and benchmarking information.
Others also recommend organizational process modeling to aid in measurement and improvement (AT&T Quality Steering Committee, 1990b; 1991; Davis, 1991; Donnell & Dellinger, 1990; Pengelly, Norris & Higham, 1993). Hodgetts (1993) described the benefits that winners of the Baldridge National Quality Award report from their emphasis on incremental improvements via ongoing, quality measurements. The benefits reported include increased quality of output, greater competitiveness, and higher profitability. Relative perceived quality and profitability are strongly related and quality is also related to growth (Buzzell & Gale, 1987).
Another value in developing organizational process diagrams is the ability to benchmark parts of the process against other similar organizations (Buckler, 1994; Camp, 1989; Freedman, 1992; McReynolds & Fern, 1992; Moad, 1994; National Institute of Standards and Technology, 1995; Trowbridge, 1995). "Benchmarking involves identifying competitors and/or companies in other industries that exemplify best practice in some activity, function, or process and then comparing one's own performance to theirs" (Eccles, 1991, p. 133). The information gained from comparing oneself to others is invaluable in a measurement and improvement program. It can show areas where much improvement is needed and where the organization compares quite favorably.
In a recent IS satisfaction survey conducted by Datamation (Meachim, 1994), users reported their top three reasons for choosing a vendor were: quality/reliability of product, product performance, and quality of service/support. They also found that the companies rated the highest by users were also the most profitable. But, quality does not improve unless you measure it (Reichheld & Sasser, 1990; Seymour, 1992). In IS organizations where total quality management (TQM) has been successfully implemented, the TQM approach has served to cut costs, better align the IS function with the organization, ease the transition to change, and strengthen the IS function's service and reputation (Anonymous, 1993). The management of quality is not sufficient to ensure the effectiveness of the IS function. The IS manager needs to install measurement systems that fairly and accurately assess the IS operation from the users' perspectives (Anonymous, 1993) and that include multiple measures of the multiple dimensions of the IS function that link to the overall goals of the organization.
DEVELOPMENT OF IS ASSESSMENT FRAMEWORK
Organizing the Measures
Existing Research Support: While many steps toward the development of an IS assessment framework have been taken, the journey is still in progress. In discussing how upper management wants to measure the IS function by its contribution to the business, Moad (1993) presented a framework for evaluating the IS function, developed by the Ernst & Young Center for Information Technology and Strategy. This framework is a 3-by-3 matrix of 9 different categories of performance of the IS function. One axis contains the sources of the IS function's performance, namely, individual, work group, and business unit. The other axis describes the area of company impact, that is, technology-enabling impact, organizational process outcome, and economic performance. No assistance is offered in developing measurement criteria or in suggesting useful measures for each category. Others have also developed IS assessment frameworks (Beise, 1989; Dickson et al., 1988; Wells, 1987). But the two assessment frameworks described next are the most recent and the most comprehensive.
DeLone and McLean's IS Success Model: DeLone and McLean (1992) created the I/S success model and suggested that researchers should "systematically combine individual measures from the I/S success categories to create a comprehensive measurement instrument" (pp. 87-88). Their model rests on the foundation of the work of Shannon and Weaver (1949) and Mason (1978). DeLone and McLean began with the definition of information as the output of an information system or the message in a communication system and noted that it can be measured at different levels. These levels include the technical level, the semantic level, and the effectiveness level. Shannon and Weaver (1949) used accuracy and efficiency of the system producing the information as the definition of the technical level; the level of success in relating the intended meaning as the definition of the semantic level; and the effect of the information on the receiver as the definition of the effectiveness level.
Mason (1978) extended the Shannon and Weaver (1949) model by relabeling effectiveness as influence and presented this level as a series of events that take place at the receiving end of an information system: receipt of the information; influence of the information on the receiver; and influence of the information on the performance of the system. "The concept of levels of output from communication theory demonstrates the serial nature of information (i.e., a form of communication). . . . In this sense, information flows through a series of stages from its production through its use or consumption to its influence on individual and/or organizational performance" (DeLone & McLean, 1992, p. 61). DeLone and McLean suggested that Mason's extension of communication theory to the measurement of IS implies the need for separate success measures for each level of information. They reviewed the IS literature and collected empirical measures of each of the six dimensions of their model. They emphasized the need for additional research to test their model and for the selection of measures of each IS success dimension. "The selection of measures should also consider the contingency variables, such as the independent variables being researched; the organizational strategy, structure, size, and environment of the organization being studied; the technology being employed; and the task and individual characteristics of the system under investigation" (p. 88).
The DeLone and McLean (1992) IS
success model is an attempt to reflect the interdependent,
process nature of IS success. Their model depicts the
relationships of the 6 IS success dimensions. They contend that
SYSTEM QUALITY
and INFORMATION QUALITY singularly and jointly affect
both USE and USER
SATISFACTION. Additionally, the amount of USE can affect
the degree of
USER SATISFACTION - positively or negatively - as well as the
reverse
being true. USE
and USER SATISFACTION are direct antecedents of INDIVIDUAL
IMPACT; and
lastly, this IMPACT on individual performance should eventually
have some
ORGANIZATIONAL
IMPACT (pp. 83-87).
These IS success dimensions are the foundation for the proposed
framework for assessing the effectiveness of the IS function.
DeLone and McLean's IS success model is the most comprehensive IS
assessment model offered by IS research thus far. Yet they
emphasize that additional research is required to authenticate
the model's validity. Seddon and Kiew (1994) were the first to
publish an empirical test of the DeLone and McLean IS success
model. They examined a slightly modified version of the first
four dimensions of the model and the relationships between them.
The results of their examination provided support for DeLone and
McLean's model.
DeLone and McLean (1992) also suggested that arbitrarily selecting measures from each of the six dimensions of IS success to form an overall IS success instrument is not recommended. Instead, further research should be conducted by systematically combining individual measures from the IS success dimensions to develop a comprehensive measurement instrument, while considering contingency variables, such as the independent variables being researched; the organizational strategy, structure, size, and environment of the study organization; the technology; and the task and individual characteristics of the system being studied. "It is unlikely that any single, overarching measure of I/S success will emerge; and so multiple measures will be necessary, at least in the foreseeable future" (p. 83). Other researchers agree (Cameron, 1986a; Carlson & McNurlin, 1992a; 1992b; Landen & Landen, 1990; Mahoney & Weitzel, 1970; Rockart & Short, 1989; Saunders & Jones, 1992; Scudder & Kucic, 1991).
Saunders and Jones' Model: Saunders and Jones (1992) developed the "IS Function Performance Evaluation Model" which was used to describe how measures should be selected from the multiple dimensions of the IS function relative to specific organizational factors and based on the perspective of the evaluator. Saunders and Jones conducted a Delphi study that examined how IS function performance dimensions were ranked in importance by IS executives, how the IS executives measured performance in each dimension, and the value they placed on the measures. The authors also interviewed several chief executive officers (CEO) of the study organizations to find out the degree of their awareness of and support for IS function activities, and to detect the level of agreement between CEOs and CIOs on the manner in which the IS function is assessed. The highest-ranked dimension was the IS function impact on strategic direction, followed by the integration of the IS function planning with corporate planning, the quality of information outputs, and the IS function's contribution to organizational financial performance. The measures used in the highest-ranked dimensions tended to be weak, surrogate measures and were not as highly valued by the IS executives as the more direct measures of the operational efficiency of the IS function, such as system response time and system availability. The authors suggested that one reason for this contradiction might be the fact that IS operational efficiency has been stressed for years while IS impact on strategic direction is a fairly new dimension and measures are still being developed. They also propose that "as the IS function matures, measures likely change from a structured focus on operational efficiency and user satisfaction to a more unstructured concern for IS impact on strategic direction" (p. 80).
The IS function performance evaluation model offered by Saunders and Jones (1992) provides additional knowledge to the developing theory for IS assessment. By comparing the IS assessment perspectives of the CEO with the CIO, they provide a unique perspective for IS assessment, previously suggested by Cameron (1986a) and others (Hamilton & Chervany, 1981b; Van de Ven & Ferry, 1980; Wilkes, 1987). They also contribute to a better understanding of the important IS success dimensions, the need to balance measures across dimensions, and the need to consider the maturity level of the IS function in an IS assessment model. These factors, evaluator perspective, organizational factors, and maturity of the IS function, provide starting variables for further development of a contingency theory for IS assessment.
Yet their model cannot be considered a comprehensive, IS assessment model for several reasons. Their study sample was relatively small and was taken from firms in only three, selected cities in Texas which leads us to question the generalizability of their results. No consideration is given to the interdependent, process nature of the performance of the IS function (DeLone & McLean, 1992) or to the suggested frequency of assessment. They also provide a very limited and inadequate list of suggested measures for each dimension. Even though the contribution to IS assessment theory by Saunders and Jones (1992) is significant, extension and further improvement is still required to provide the comprehensive model for IS assessment demanded by organizations today. Extension and further enhancements are necessary to provide a more complete and comprehensive set of IS assessments and a method for deciding what is appropriate given specific organizational and environmental factors (i.e. a contingency theory). Different elements of each of these two models will be used as a basis for the development of a more comprehensive model for IS assessment.
A Comprehensive IS Assessment Framework: There is considerable overlap in these two models. Several of the DeLone and McLean (1992) categories of IS success are represented by one or more of the Saunders and Jones (1992) performance dimensions. For example, the Saunders and Jones dimensions "IS impact on strategic direction," "IS contribution to organization's financial performance," "integration of IS and corporate planning," and "integration with related technologies across other organizational units" could all be considered as sub-dimensions of "organizational impact." Also, "quality of information outputs" corresponds to "information quality," "user/management attitudes" corresponds to "user satisfaction," and "adequacy of system development practices" and "IS operational efficiency" roughly correspond to "system quality." "IS personnel development" was replaced as a performance dimension by the Saunders and Jones Delphi group by "ability of IS function to identify and assimilate new technologies." But, this dimension was the lowest ranked, the least used, and the 2 measures used to assess performance on that dimension had the lowest mean values of all measures listed. Therefore, these two dimensions will receive no further consideration.
" IS staff competence" is also not included in the DeLone and McLean model - understandably - since they found no empirical research using measures of IS staff competence as a measure of IS success. Staff competence is not unique to the IS function. Typically, organizations have formal review processes to measure the staff competence of the entire organization. It is an important assessment dimension and should not be neglected by the IS manager. Furthermore, IS staff competence is subsumed by the proposed "service quality" dimension discussed below and will not be included as a separate dimension of this framework for IS assessment.
The comprehensive, IS assessment framework, in addition to the DeLone and McLean (1992) dimensions, will include a "service quality" dimension (Pitt, Watson & Kavan, 1995) and a "work group impact" dimension (Figure 1). The measures selected by the IS manager should be balanced across the dimensions, include indicators of both effectiveness and ineffectiveness (paradox), and be developed in cooperation with the work groups involved. Periodically, key measures from each dimension of the IS assessment system should be benchmarked against the performance of other firms.
Figure 1 - A Comprehensive, IS Assessment Model: Organizing the Measures
Service quality, system quality, and information quality singularly and jointly affect both use and user satisfaction. Also, the amount of use can affect the degree of user satisfaction - positively or negatively - as well as the reverse being true. Use and user satisfaction are direct antecedents of individual impact; this impact on individual performance should have some work group impact for most organizations and in some cases may also directly lead to an organizational impact; and, finally, this impact on work group performance should eventually have some organizational impact.
DeLone and McLean's (1992) extensive literature review and tables of success measures for each dimension will not be duplicated here. Rather, to build on their work, each of the six original dimensions are updated with suggested measures from work in other disciplines or from work published since 1988. Also, the two additional dimensions of IS success are defended as worthy to be included in the model and possible measures are presented. Finally, the beginnings of a contingency theory for IS assessment are suggested to guide senior IS managers in selecting appropriate dimensions and measures for their organizations.
Service quality: A service quality perspective views organizations as a collection of multiple processes with the goal of providing the customer with a high-quality service. Service quality is applicable to the IS function, since IS can be considered a service function that serves the information technology needs of the larger organization. The growth of end-user computing, decentralization, and the available choices for sources of IS services, promotes greater discretion by the customers of the IS function in their use and procurement of IS services. To meet the demands of this increasingly market-driven environment, the IS manager must be sensitive to the expectations of their customers and understand the perceived value placed on their services by their customers (Kettinger & Lee, 1994). Moreover, customers recognize and appreciate quality in service areas such as responsiveness to special needs, reliability, courtesy, and communication just as much as, if not more, than the technical specifications of a product or the appropriateness of the information provided (Ferguson & Zawacki, 1993).
Recognition of the importance of
measuring the service quality of the IS function has only
recently appeared in IS literature (Ferguson & Zawacki, 1993;
Kettinger & Lee, 1994; Pitt et al., 1995). But, the marketing
literature provides considerable help for the IS manager in
knowing how to measure and improve service quality. Parasuraman,
Zeithaml, & Berry (1985) listed the dimensions of service
quality (or "determinants") as:
reliability - consistency of
performance and dependability; service performed right the first
time;
responsiveness -
willingness/readiness of employees to provide service in a timely
manner
(promptness);
competence - possession of the
required skills/knowledge to perform the service;
access - approachability and ease
of contact; convenient hours and location;
courtesy - politeness, respect,
consideration, and friendliness of contact personnel;
communication - keeping customers
informed in language they understand and listening to them;
credibility - trustworthiness,
believability, honesty; having the customers best interests at
heart;
security - freedom from danger,
risk, or doubt;
understanding/knowing the customer
- making the effort to understand the customer's
needs; and
tangibles - physical evidence of
the service, including facilities, personnel appearance, tools or
equipment, etc.
They developed these further and gave examples of survey
questions to measure levels of each determinant in their
subsequent book (Zeithaml, Parasuraman & Berry, 1990). The
development and testing of an instrument to measure service
quality, SERVQUAL, are discussed in additional articles by these
researchers (Parasuraman, Berry & Zeithaml, 1991b;
Parasuraman, Berry & Zeithaml, 1993; Parasuraman, Zeithaml
& Berry, 1988; Parasuraman, Zeithaml & Berry, 1994a;
Parasuraman et al., 1994b). They also give practical advice for
understanding customer expectations of service and for improving
quality of service (Berry & Parasuraman, 1992; Berry,
Parasuraman & Zeithaml, 1984; Berry, Parasuraman &
Zeithaml, 1988; Berry, Parasuraman & Zeithaml, 1994; Berry,
Zeithaml & Parasuraman, 1990; Parasuraman, Berry &
Zeithaml, 1991c; Zeithaml, Berry & Parasuraman, 1988a;
Zeithaml, Berry & Parasuraman, 1993). Other viewpoints on the
nature of service quality are available (Rust & Oliver,
1994).
Babbar (1992) extended the service quality model to include system hardware and networking requirements and the dynamics of system operation and control. Nath (1992) used the work of Parasuraman et al. (1985) to develop a framework to improve service quality. His framework involves the examination of the interfaces between the customer, the employee, and the existing IS applications to detect where applications of information technology will alter the interfaces in a positive way either for the customer or the organization. The effect of the change on the customer should be evaluated in terms of how it influences the ten determinants of service quality listed above. Funston (1992) developed a service quality model and depicted gaps in service quality, communication, delivery, and design where measurement and improvement are possible. Performance evaluation should be linked to service quality at all levels and the customer should be built into these evaluations. Return on quality (ROQ) provides a method for evaluating the financial impact of service quality improvements to the business (Rust et al., 1995).
Considering the IS function as a service and applying the principles of service quality can yield many opportunities to show the value of the IS function to the organization (Remenyi & Money, 1994; Santosus, 1995). But, measuring service quality is difficult and often ambiguous (Cheng & Ngai, 1994); moreover, currently-used measures are problematic (Kappelman, Van Dyke & Prybutok, 1995; Van Dyke, Prybutok & Kappelman, 1995). Further study of how to reliably assess this important IS success dimension is needed. While service quality measures are important for assessing the IS function, using them alone in an assessment system will not provide a thorough understanding of the total contribution of the IS function to the organization.
System quality: In addition to the many measures of the information processing system itself from empirical studies listed by DeLone and McLean (1992), such as, reliability, response time, ease of use, usefulness, flexibility, accessibility, etc., cost benefit analysis was presented as a worthwhile measure of the value of individual information systems (Ford, 1994; Keim & Janaro, 1982; King & Schrems, 1978; Mason & Sassone, 1978; Matlin, 1979; Oman & Ayers, 1988; Thompson & Cannon, 1978). Others recommended post-project evaluations or audits (Ahituv, Even-Tsur & Sadan, 1986; Quinn & Baily, 1994). Rainer and Watson (1995) also examined ease of use as well as the presence of specific functions of the system as measures of system quality in their study of executive information system success.
Information quality: Sääksjärvi and Talvinen (1993) used content, availability, and accuracy as measures of information quality in their study of two specific marketing information systems. In their study of the keys to executive information system success, Rainer and Watson (1995) used accuracy, timeliness, conciseness, convenience, and relevance of the information as measures of information quality.
Use: Information systems can improve the quality and productivity of individuals, groups, and organizations, only if they are actually used. DeLone and McLean (1992) provided a lengthy list of IS use studies. In addition, Markus and Keil (1994) suggest that organizations should approach system development as business process reengineering and ensure that implementability, or use, is built in. Rather than develop an IS to solve organizational problems and mandate its use, Markus and Keil argue that system use is inevitable when the interests of developers and users are aligned, good system design concepts jointly developed by users and developers are used, and system use is encouraged through rewards and incentives. Sääksjärvi and Talvinen (1993) measured use of each subsystem of the two marketing information systems they studied as well as the relative usage of each and the integration of usage of the two systems. Le Blanc and Kozar (1990) found that increases in decision support system (DSS) usage were associated with reductions in marine casualties on the lower Mississippi River.
User satisfaction: As discussed by DeLone and McLean (1992), user satisfaction is probably the most widely used single measure of IS success and they provided a summary of the studies and a list of the measures used in measuring user satisfaction. Bailey and Pearson (1983) presented a 39-item instrument for measuring user satisfaction. Ives, Olson, and Baroudi (1983) added four items to the Bailey and Pearson instrument to measure overall user information satisfaction (UIS) and developed a short form of the UIS instrument. Baroudi and Orlikowski (1988) evaluated the psychometric properties of the short-form UIS instrument and found it to be reasonably valid and reliable. This short-form UIS instrument has seen wide use, but also has been criticized. Melone (1990) questioned its use since the UIS construct had not been integrated with user attitude theory. Galletta and Lederer (1989) found that the short-form UIS instrument did not exhibit test/retest reliability, but that four summary questions of overall satisfaction did behave reliably.
One recent study of the short-form UIS instrument examined the underlying construct for UIS (Doll et al., 1995). The results of their analysis supports the use of the short-form, 13-item instrument as a measure of overall UIS. Kettinger and Lee (1994) compared the SERVQUAL instrument with the UIS instrument and found that they were generally mutually exclusive and complementary. So, both service quality and user satisfaction should be measured. But the reliable measurement of user satisfaction requires further study. Perhaps researchers should go back to Bailey and Peason's original 39-item instrument for further study. Maybe effort is being wasted by concentrating so much study on a subset of the original instrument, when a different subset of questions may better measure the UIS construct.
Conrath and Mignen (1990) found that even though the literature extols the measurement of user satisfaction, very few are actually measuring it. Only 26 percent of their sample of large Canadian firms had any formal mechanism in place to measure customer satisfaction.
Individual impact: Sääksjärvi and Talvinen (1993) used overall benefit of system usage to measure the impact of two specific marketing information systems on the users. Rainer and Watson (1995) measured impact on executive work with the operational variables: improve executive efficiency, enable executives to make higher-quality decisions, improve communications, improve operational control, and improve executives' mental model of the firm. Dickson, Senn, and Chervany (1977) provided a summary of research programs administered between 1970 and 1975 that were conducted to examine the significance of various information system characteristics on decision activity. These experiments used a variety of measures as dependent variables, including decision quality, decision time, decision confidence, user evaluation, and estimated outcomes.
Work group impact: DeLone and McLean's (1992) model addresses the impact of the IS function on individual and organizational performance, and assumes a flow of influence from the individual through intermediate stages to the organization. The impact of the IS function on work group performance is an important intermediate stage between the individual and the organization. The current organizational environment of many firms places a greater emphasis on the role of teams in the workplace (Alavi & Keen, 1989; Grohowski, McGoff, Vogel, Martz & Nunamaker, 1990) and therefore, a corresponding emphasis on work group-level performance. In fact, Barua, Kriebel, and Mukhopadhyay (1995) found "that the most significant contributions of IT investments occur at low organizational levels where they are implemented" (p. 20). They also confirmed that the intermediate level contributions positively affected organizational performance measures such as return on assets (ROA) and market share.
As discussed before, Moad (1993) presented the Ernst & Young framework for evaluating IS. It is a 3-by-3 matrix that list the sources of impact as technology-enabling, organizational process outcome, and economic performance on the individual, work group, and business unit. The work group level is explicitly included as an important unit of measure for the evaluation of the impact of IS. The work group is also included in the levels of analysis between individual and organization by Bakos (1987).
Electronic meeting systems (EMS) have been used to support strategic management planning groups successfully as measured by improved equality of participation, reduced production blocking, reduced evaluation apprehension, and improved communication across the hierarchy (Tyran, Dennis, Vogel & Nunamaker, 1992). In a study of negotiating groups using EMS, the measures of success included effectiveness of the original solutions and solution quality, efficiency in terms of total comments and file size, and satisfaction with the group process, ideas generated, evaluations, and overall, and general questions "remain in group?" and "how much fun?" (Nunamaker, Dennis, Valacich & Vogel, 1991a). They also found that larger groups were able to function effectively using the EMS than were possible with no support. Others report similar results (Dennis, Heminger, Nunamaker & Vogel, 1990; Grohowski et al., 1990). Herniter, Carmel, and Nunamaker (1993) reported that EMSs could improve the efficiency of the negotiation process during union bargaining. Two companies that used EMS during bargaining for tasks like writing proposals and tracking agreements, ratified contracts with their unions more quickly and with fewer disputes than in previous sessions.
In another study of EMS implementation in a large corporation, researchers found increased participation, fewer meetings over less time are required to solve problems, participants stay focused on task, pre-planning of meetings takes on increased importance, post-meeting distribution of the session data is crucial, low levels of participant computer competence have not deterred effective use, meeting room environment should match the characteristics of the group, software systems must be flexible to meet the variety of group applications, an infrastructure of staff and support is crucial to EMS success, EMSs help provide an organizational memory concerning related meetings, EMSs provide structure and control mechanisms for the meeting, and the propensity to use the EMS reveals its value (Grohowski et al., 1990). All of these factors are potential measures of success for IS impact on work groups.
Jarvenpaa, Rao, and Huber (1988) used meeting thoroughness, meeting equality, meeting equity, meeting quality, and participants' satisfaction as dependent measures in comparing conventional and two types of meeting support technology. The various studies evaluating group decision support systems (GDSS) and EMSs were organized and summarized and a taxonomy of environments for EMSs were developed by Dennis, George, Jessup, Nunamaker, and Vogel (1988). This work provides an extensive source of success measures for evaluating IS impact on the work group. A comprehensive overview of EMS development, theoretical foundations, applications, effects, and benefits is available (Nunamaker, Dennis, Valacich, Vogel & George, 1991b).
In a survey of potential users of work group computer-support tools, Satzinger and Olfman (1995) found that support for group work between meetings was perceived to be more useful than either support for face-to-face or electronic meetings, and traditional single-user tools were perceived to be more useful than multi-user group tools. Dean, Lee, Orwig, and Vogel (1994) studied the task of business analysis using EMS versus a single-user tool. They established that the EMS-based modeling tool allowed a greater number of individuals to participate efficiently in model development and models were developed between 175 percent and 251 percent faster with the EMS than with the traditional approach. They also incorporated measures of model quality into their evaluation. Gallupe and DeSanctis (1988) compared GDSS-supported and non-supported decision-making groups and found significantly better decision quality in those groups that received GDSS support. Implementers of GDSSs should be cautioned, though, since in this study, the decision confidence and satisfaction with the decision process of the group members were lower in the GDSS-supported groups than in the non-supported groups.
Organizational impact: In their study of the Information Week 500, Brynjolfsson and Hitt (1995) found that two broad strategic goals for IS investments emerged: Some focused on costs savings and improved management control while others had a customer orientation and made investments in quality, customer service, flexibility, and speed. Even though their productivity analysis was based on hard numbers such as revenue, labor costs, and capital costs, the customer-oriented companies had significantly better productivity performance and also achieved higher profits. Kelley's (1994) findings show that with programmable automation technology, manufacturers can produce the same output in about three-fifths of the time it would ordinarily take on conventional machinery. Even greater reductions in production time on the new technology are found with greater experience, more extensive use of the technology, and changes in the organization of work. She was successful in showing the value of investment in IT by focusing on the process innovated by the IT application as the unit of analysis (rather than the entire plant or organization) and by using a time-based indicator of productivity, unit production hours, while controlling for product attributes and eleven other factors.
Quinn and Baily (1994) interviewed over 100 top managers in all major service industries that were heavy users of IT, such as banking and financial services, transportation, communications, retailing, etc. Whenever possible, these managers attempted to quantify the return on investment (ROI) for each IT investment decision. In most cases, benefits were practically impossible to estimate. They often decided to purchase based on intuitive and non-financial judgments. Maintaining market share, avoiding catastrophic losses, creating greater flexibility and adaptability, improving responsiveness for new product lines, improving service quality, enhancing quality of work life, increasing predictability of operations, post-project evaluations or audits, and benchmarks, were all mentioned as possible ways to measure the benefits of IT investment to the firm, but except for the last two, were almost impossible to quantify.
Cost benefit analysis may also be used to quantify the impact of the IS function on the organization in an overall ROI calculation, but doing so is often difficult do to the inability to adequately quantify intangible or qualitative benefits (Ford, 1994; Keim & Janaro, 1982; King & Schrems, 1978; Mason & Sassone, 1978; Matlin, 1979; Oman & Ayers, 1988; Semich, 1994). Some of the overlooked benefits of IT investments include turning over accounts receivables faster, shortening the monthly general ledger closing cycle, performing "what-if" analysis in real time during the financial planning cycle, reducing system support costs, reducing the time and cost of preparing budgets, business plans, and proposals due to the increased availability of business data in real time, and reducing the cost of generating quarterly and annual statements (Semich). Other economic-type value measures include information economic analysis (Semich) and data envelopment analysis (DEA) (Chismar & Kriebel, 1985).
Harris and Katz (1991b) found evidence that firm performance in the home office operation of systems technology leaders in the life insurance industry was linked to the level of IT investment intensity. Their longitudinal analysis showed that the firms with the most improvement in their organizational performance exhibited higher growth in the ratio of IT expense to total operating expense and larger reductions in the ratio of IT costs to premium income. Neumann, Ahituv, and Zviran (1992) developed a measure for determining the strategic relevance of IS to the organization and included the following operational variables: fit of IS applications portfolio to the organization's critical success factors (CSFs), IS contribution to the organization's competitiveness, IS support to tactical management, IS support to operations, IS contribution to improving profitability, IS contribution to financial gains through improved operations, and perceived overall criticality of IS to the organization.
Sethi and King (1994) developed a multidimensional measure of competitive advantage called Competitive Advantage Provided by an Information Technology Application (CAPITA). The CAPITA dimensions consist of efficiency, functionality, threat, preemptiveness, and synergy. They suggest CAPITA might be used for competitive assessment, including justifying and evaluating applications and acting as dependent variables in empirical competitive advantage research. Mahmood and Mann (1993) summarized the research on IT impacts on organizational performance in a handy table with a list of IT investment measures and corresponding organizational performance measures. Using the data on one hundred firms reported in the Computer World "Premier 100" in 1989, Mahmood and Mann conducted a canonical correlation analysis to discover relationships among organization performance and information technology investment variables. They found that when evaluating the impact of IT investment on organizational performance, performance measures such as sales by total assets, market value to book value, return on investment, sales by employee, and return on sales should be considered. The measures to be considered for use as measures of IT investment include IT budget as a percentage of revenue, percentage of IT budget spent on IT training, number of PC's and terminals as a percentage of total employees, and estimated IT value as a percentage of revenue.
Le Blanc and Kozar (1990) found that increases in decision support system (DSS) usage were associated with reductions in marine casualties on the lower Mississippi River. Palvia, Perkins, and Zeltman (1992) reported on the impacts of a self-named "organizational effectiveness system (OES)" developed and used by the Federal Express Corporation, called the PRISM system. It is an advanced, multi-technology system and includes core personnel functions, expanded personnel and organizational functions, and extensive external interface features. Organization impacts and benefits of the PRISM system consist of strategic benefits (organizational flexibility), impact on personnel division, impact on management, impact on employees, and extra-organizational relationships. The financial impact of implementing electronic data interchange (EDI) on the Chrysler Corporation was estimated from reduced inventory holding costs, obsolete inventory costs, transportation costs, and premium freight costs, as well as savings that arose from preparing and processing documents electronically rather than manually. The results estimate the savings to be over $100 per vehicle (Mukhopadhyay, Kekre & Kalathur, 1995).
Carlson and McNurlin (1992a) discussed various measurement models for measuring IT value, including the Kaplan and Norton "balanced scorecard" (Kaplan & Norton, 1992). It consists of four views (customer, internal business, financial, innovation and learning) and suggested measures for each. Carlson and McNurlin called the balanced scorecard a "simple, yet elegant, measurement framework for integrating the diverse kinds of metrics that are important to management" (p. 8). The Ernst and Young framework for evaluating IS called the Value Management Framework (Moad, 1993) discussed earlier suggested that measures of IS impact on the organization should include the technology-enabling impact, organizational process outcome, and economic performance.
Contingency Theory Development
Selecting the Dimensions and Measures: The list of measures for each IS success dimension provided here, supplemented with the lists collected by DeLone and McLean (1992), supply the IS manager with an abundant resource for selecting measures for his or her organization. But several questions remain unanswered: What are the appropriate IS success dimensions that should be assessed for each organizational and external environmental context? Once the appropriate dimensions are selected, what are the appropriate measures to assess performance in each dimension, again, given the context of the organization and external environment? Finally, how should these IS success dimensions and measures be selected?
The purpose of considering a contingency theory for IS assessment stems from the goal of providing guidance for an IS assessment selection strategy that neither dictates a universal solution that is unrealistic for most organizations nor advocates a situation specific view that provides no assistance beyond the given context. Contingency theories propose that different strategies are appropriate for each competitive business setting. They differ from the universal view by emphasizing "it all depends" and they differ from the situation specific view by asserting that there are classes of settings for which strategic generalizations can be made (Hambrick & Lei, 1985).
To build a contingency theory for IS assessment, the relevant contingency variables must be listed (Hambrick & Lei, 1985). Hofer (1975) listed 54 variables that he thought should affect choices of strategies and theorized that product life cycle was the most crucial contingency variable. Hambrick and Lei (1985) reduced the list to 10 significant variables in their study: stage of product life cycle, consumer versus industrial user sector, product differentiability, technological change, concentration rate, purchase frequency, industry imports, share instability, demand instability, and dollar importance to customer. The three most significant variables in their study were consumer versus industrial user sector, purchase infrequency, and stage of product life cycle. These lists of variables were reduced by keeping those contingency variables that were the most significant in moderating the effects of key strategic variables on performance. In the absence of empirical studies to assist in the selection of the most significant contingency variables for IS assessment, all relevant factors should be identified and grouped into broad categories, followed by empirical prioritization (Zeithaml, Varadarajan & Zeithaml, 1988b). The broadest categories of relevance to the IS function appear to be organizational and external environmental.
Organizations are embedded in their environment, however they are not so tightly fixed as to totally restrict strategic maneuvers (Hambrick, 1981). DeLone and McLean (1992) mentioned the importance of considering other organizational and environmental factors, such as those listed by Saunders and Jones (1992) (mission, size, industry, top management support, IS executive hierarchical placement, competitive environment, size of IS function, maturity of IS function, evaluator perspective) when selecting appropriate measures for each dimension. In developing their model for IS research, Ives, Hamilton, and Davis (1980) presented environmental variables that "define the resources and constraints" of the IS function. For example, "the external environment includes legal, social, political, cultural, economic, educational, resource and industry/trade considerations. . . . The organizational environment is marked by the organizational goals, tasks, structure, volatility, and management philosophy/style" (p. 916).
As previously discussed, the organizational effectiveness literature underscored the need to define a theory or model of organizational effectiveness for the organization before developing measures of effectiveness (Cameron & Whetten, 1983; Goodman & Pennings, 1977). Many researchers assert that the perspective of the evaluator must be considered and that there are often multiple perspectives to consider, such as the CEO and the CIO (or IS executive) (Cameron & Whetten, 1983; Hamilton & Chervany, 1981b; Saunders & Jones, 1992; Wilkes, 1987; Wilkes & Dickson, 1987). Zmud (1979) analyzed the empirical literature regarding the influence of individual differences upon IS success and found a clear indication that individual differences do exert a major force in determining IS success. As described before, the perspective of the evaluator is also significant. Often, incongruent perceptions of the definition of IS success exist between the CEO and CIO. The CEO is consistently focused on external factors such as market share and customer satisfaction. The CIO usually defines success by focusing on internal measures. This lack of agreement holds for the issue of how to measure the IS function. Many CIOs tend to concentrate on system and network up-time, reports delivered on time, number of errors, and control over expenses. While these are important, the CIO should also be assessing the IS function using many of the same factors as the CEO when measuring corporate performance, including market share, customer satisfaction, margin, and return on investment (Plewa & Lyman, 1992).
Figure 2 - A Comprehensive, IS Assessment Model and Contingency Theory (Selecting the Measures)
IS organizational maturity was found to be significantly related to user satisfaction (Mahmood & Becker, 1985) and as cited before, Saunders and Jones (1992) suggested that IS organizational maturity may impact on the relevance and usefulness of various measures to the IS manager. Corporate level strategy, organization structure, industry, organization size, business strategy, work group interdependence, culture, incentive system, information intensity of products and/ or services, IT management expertise, IT end-user skills, strategic role of IT, size of IS organization, IS budget size, user participation/involvement, history of organization, individual characteristics, task, climate, and location of the responsible executive are presented as potential contingency variables (Brown & Magill, 1994; Davis & Hamann, 1988; Ein-Dor & Segev, 1978; Harris & Katz, 1991a; Mahmood & Becker, 1985; McKeen, Guimaraes & Wetherbe, 1994; Premkumar & King, 1994; Scott, 1977; Weill & Olson, 1989; Wetherbe & Whitehead, 1977; Zmud, 1979). Figure 2 depicts the contingency theory for IS assessment by showing the IS assessment model within the context of the organization and the external environment, but does not list the variables within the external and organizational environments mentioned above. Rather than attempt to list every variable and measure mentioned in this review in a table, Figure 3 summarizes the contingency theory for IS assessment using selected measures for each assessment dimension and selected organizational and external environmental variables.
Figure 3 - IS Assessment Selection Model (adapted from Saunders & Jones, 1992, p.66)
CONCLUSION
The progress toward the development of a comprehensive framework for IS assessment is significant, yet much work remains to be done. What are the dimensions of IS success that should be assessed? The dimensions critical to the success of the IS function are service quality, system quality, information quality, use, user satisfaction, individual impact, work group impact, and organizational impact. What are the measures for assessing the performance of the IS function in each dimension? This work provides everything needed to create comprehensive, IS assessment systems. The existing models of IS success were updated to include the emerging IS success dimensions of service quality and work group impact and provide a comprehensive method for organizing the various measures of IS success. In addition, many new measures from recent research were presented to supplement the lists supplied by previous research. To help answer the question of what dimensions and measures might be appropriate, all of the IS success dimensions and measures were placed in the context of the organization and environment and the important variables to consider in these contexts were listed, providing the start of the development of a contingency theory for IS assessment.
The algorithm for selecting the appropriate dimensions and measures has yet to be developed and will require empirical research. How should the IS manager select the appropriate IS success dimensions and measures for each given their organizational and environmental context? How should the dimensions and measures be combined? What is working in practice in successful organizations? Research studies to answer these questions should be both quantitative and qualitative, to capture the broad, cross-sectional applications of IS assessment systems and the in-depth, complex nature of the subject. Cross-sectional studies might include Delphi groups or large-scale surveys of IS managers to discover if these are the right IS success dimensions and how these or other dimensions are being assessed and what measures are being used. Such data could then be examined to determine what organizational and environmental contexts and groups of dimensions and measures actually seem to work best. These studies would be complemented by the use of in-depth interviews and content analysis of organizational documents to learn the details of IS assessment system implementations. With the foundation supplied by the model and contingency theory introduced here, and the support of the body of research yet to come, a theory for IS assessment will be achieved. Such a theory has the potential to contribute to the quality and productivity of the IS function and the larger organization by providing feedback to manage and improve the IS function to better meet the needs of the organization.
Acknowledgments:
This work was funded in part by the
University of North Texas (UNT) Information Systems Research
Center (ISRC) and the College of Business Administration. We
appreciate the constructive comments from three anonymous
reviewers and their considerable contribution to improving our
work.
REFERENCES
Ahituv, N. (1980). A Systematic Approach Toward Assessing the Value of an Information System. MIS Quarterly, 4(4), 61-75.
Ahituv, N., Even-Tsur, D., & Sadan, B. (1986). Procedures and Practices for Conducting Post evaluation of Information Systems. Journal of Information Systems Management, 3(2), 61-65.
Alavi, M., & Keen, P. G. W. (1989). Business Teams in an Information Age. Information Society, 6(4), 179-195.
Anonymous. (1993). TQM in IS: Defining user expectations. I/S Analyzer, 31(5), 1-14.
Anthony, R. N. (1965). Planning and Control Systems: A Framework for Analysis. Boston, MA: Harvard Business School.
Asbrand, D. (1993, Feb 15). User satisfaction is essential to successful projects. InfoWorld, 15, 50.
AT&T Quality Steering Committee. (1990a). Achieving Customer Satisfaction. Indianapolis, Indiana: Author.
AT&T Quality Steering Committee. (1990b). PQMI: Tips, Experiences, & Lessons Learned. Indianapolis, Indiana: Author.
AT&T Quality Steering Committee. (1991). Reengineering Handbook. Indianapolis, Indiana: Author.
Baatz, E. B. (1994). Altered stats. CIO, 8(2), 40-46.
Babbar, S. (1992). A Dynamic Model for Continuous Improvement in the Management of Service Quality. International Journal of Operations & Production Management, 12(2), 38-48.
Bailey, J. E., & Pearson, S. W. (1983). Development of a Tool for Measuring and Analyzing Computer User Satisfaction. Management Science, 29(5), 530-545.
Bakar, R. A. (1994). Quality in information technology: Incorporating the market-driven quality concepts. Asia Pacific Journal of Quality Management, 3(3), 54-61.
Bakos, J. Y. (1987). Dependent variables for the study of firm and industry-level impacts of information technology. Paper presented at the Eighth Annual International Conference on Information Systems (ICIS), Pittsburgh, PA.
Ball, L., & Harris, R. (1982). SMIS Members: A membership analysis. MIS Quarterly, 6(1), 19-38.
Band, W. (1990a). Create a Customer Satisfaction Index (CSI) to Improve Your Performance. Sales & Marketing Management in Canada, 31(7), 58-59.
Band, W. (1990b). Performance Metrics Keep Customer Satisfaction Programs on Track. Marketing News, 24(11), 12.
Baroudi, J. J., & Orlikowski, W. J. (1988). A short form measure of user information satisfaction: A psychometric evaluation and notes on use. Journal of Management Information Systems, 4(4), 44-59.
Barua, A., Kriebel, C. H., & Mukhopadhyay, T. (1995). Information technologies and business value: An analytic and empirical investigation. Information Systems Research, 6(1), 3-23.
Beise, C. M. (1989). Assessment of the Information Systems Function: A Study of the IS/Organizational Interface (Doctoral Dissertation, Georgia State University). Dissertation Abstracts International, 50, 4010A.
Bender, D. H. (1986). Financial Impact of Information Processing. Journal of Management Information Systems: JMIS, 3(2), 22-32.
Berndt, E. R., & Morrison, C. J. (1991, December 9). Computers Aren't Pulling Their Weight. Computerworld, 25, 23, 25.
Berry, L. L., & Parasuraman, A. (1992). Prescriptions for a Service Quality Revolution in America. Organizational Dynamics, 20(4), 5-15.
Berry, L. L., Parasuraman, A., & Zeithaml, V. A. (1984). Synchronizing Demand and Supply in Service Businesses. Business, 34(4), 35-37.
Berry, L. L., Parasuraman, A., & Zeithaml, V. A. (1988). The Service-Quality Puzzle. Business Horizons, 31(5), 35-43.
Berry, L. L., Parasuraman, A., & Zeithaml, V. A. (1994). Improving service quality in America: Lessons learned Executive commentary. Academy of Management Executive, 8(2), 32-52.
Berry, L. L., Zeithaml, V. A., & Parasuraman, A. (1990). Five Imperatives for Improving Service Quality. Sloan Management Review, 31(4), 29-38.
Blenkinsop, S. A., & Burns, N. (1992). Performance Measurement Revisited. International Journal of Operations & Production Management, 12(10), 16-25.
Borovits, I., & Neumann, S. (1979). Computer Systems Performance Evaluation. Lexington, Massachusetts: D. C. Heath and Company.
Brancheau, J. C., & Wetherbe, J. C. (1987). Key Issues in Information Systems Management. MIS Quarterly, 11(1), 23-45.
Brown, C. V., & Magill, S. L. (1994). Alignment of the IS functions with the enterprise: Toward a model of antecedents. MIS Quarterly, 18(4), 371-403.
Brynjolfsson, E. (1993). The productivity paradox of information technology. Communications of the ACM, 36(12), 67-77.
Brynjolfsson, E., & Hitt, L. (1995, September 18). The productive keep producing. InformationWeek, 38-43.
Buckler, G. (1994). Do you measure up? I.T. Magazine, 26(1), 14-17.
Buzzell, R. D., & Gale, B. T. (1987). The PIMS Principles: Linking Strategy to Performance. New York: The Free Press.
Cameron, K. (1986a). A Study of Organizational Effectiveness and Its Predictors. Management Science, 32(1), 87-112.
Cameron, K. S. (1978). Measuring organizational effectiveness in institutions of higher education. Administrative Science Quarterly, 23, 604-632.
Cameron, K. S. (1980). Critical Questions in Assessing Organizational Effectiveness. Organizational Dynamics, 9(2), 66-80.
Cameron, K. S. (1984). The effectiveness of ineffectiveness. Research in Organizational Behavior, 6, 235-285.
Cameron, K. S. (1986b). Effectiveness as Paradox: Consensus and Conflict in Conceptions of Organizational Effectiveness. Management Science, 32(5), 539-553.
Cameron, K. S., & Quinn, R. E. (1988). Organizational paradox and transformation. In R. E. Quinn & K. S. Cameron (Eds.), Paradox and Transformation: Toward a Theory of Change in Organization and Management (pp. 1-18). Cambridge, MA: Ballinger.
Cameron, K. S., & Whetten, D. A. (1983). Some conclusions about organizational effectiveness. In K. S. Cameron & D. A. Whetten (Eds.), Organizational Effectiveness: A Comparison of Multiple Models (pp. 261-277). New York: Academic Press.
Camp, R. C. (1989). Benchmarking: The Search for Industry Best Practices that Lead to Superior Performance. Milwaukee, WI: ASQC Quality Press.
Campbell, J. P. (1977). On the nature of organizational effectiveness. In P. S. Goodman & J. M. Pennings (Eds.), New Perspectives on Organizational Effectiveness (pp. 13-55). San Francisco, CA: Jossey-Bass.
Carey, D. (1993). Nine managers discuss: Measuring and delivering value. I.T. Magazine, 25(6), 14-24.
Carlson, W., & McNurlin, B. (1992a). Basic Principles for Measuring IT Value. I/S Analyzer, 30(10), 1-16.
Carlson, W. M., & McNurlin, B. C. (1992b). Do You Measure Up? Computerworld, 26(49), 95-98.
Cheng, T. C. E., & Ngai, E. W. T. (1994). Management support systems for service quality management: A research direction. International Journal of Quality & Reliability Management, 11(8), 44-56.
Chismar, W. G., & Kriebel, C. H. (1985). A method for assessing the economic impact of information systems technology on organizations. Paper presented at the Sixth Annual International Conference on Information Systems (ICIS), Indianapolis, IN.
Clark, T. D., Jr. (1992). Corporate Systems Management: An Overview and Research Perspective. Communications of the ACM, 35(2), 60-75.
Conrath, D. W., & Mignen, O. P. (1990). What Is Being Done to Measure User Satisfaction with EDP/MIS. Information & Management, 19(1), 7-19.
Cross, K. F., & Lynch, R. L. (1992). For Good Measure. CMA Magazine, 66(3), 20-23.
Crowston, K., & Treacy, M. E. (1986). Assessing the Impact of Information Technology on Enterprise Level Performance. Paper presented at the Seventh International Conference on Information Systems (ICIS), San Diego, CA.
Cummings, L. L. (1977). Emergence of the instrumental organization. In P. S. Goodman & J. M. Pennings (Eds.), New Perspectives on Organizational Effectiveness (pp. 56-62). San Francisco, CA: Jossey-Bass.
Davis, G. B., & Hamann, J. R. (1988). In-context information systems assessment: A proposal and an evaluation. In N. Bjorn-Anderson & G. B. Davis (Eds.), Information Systems Assessment: Issues and Challenges (pp. 283-296). North-Holland, NY: Elsevier Science.
Davis, T. R. V. (1991). Information Technology and White-Collar Productivity. Academy of Management Executive, 5(1), 55-67.
Dean, D. L., Lee, J. D., Orwig, R. E., & Vogel, D. R. (1994). Technological support for group process modeling. Journal of Management Information Systems, 11(3), 43-63.
DeLone, W. H., & McLean, E. R. (1992). Information systems success: The quest for the dependent variable. Information Systems Research, 3(1), 60-95.
Dennis, A. R., George, J. F., Jessup, L. M., Nunamaker, J. F., Jr., & Vogel, D. R. (1988). Information Technology to Support Electronic Meetings. MIS Quarterly, 12(4), 591-624.
Dennis, A. R., Heminger, A. R., Nunamaker, J. F., Jr., & Vogel, D. R. (1990). Bringing Automated Support to Large Groups: The Burr-Brown Experience. Information & Management, 18(3), 111-121.
Dickson, G. W., Leitheiser, R. L., Nechis, M., & Wetherbe, J. C. (1984). Key information system issues for the 1980s. MIS Quarterly, 8(3), 135-148.
Dickson, G. W., Senn, J. A., & Chervany, N. L. (1977). Research In Management Information Systems - the Minnesota Experiments. Management Science, 23(9), 913-923.
Dickson, G. W., Wells, C. E., & Wilkes, R. B. (1988). Toward a derived set of measures for assessing IS organizations. In N. Bjorn-Anderson & G. B. Davis (Eds.), Information Systems Assessment: Issues and Challenges (pp. 129-147). North-Holland, NY: Elsevier Science.
Doll, W. J., & Ahmed, M. U. (1985). Documenting Information Systems for Management: A Key to Maintaining User Satisfaction. Information & Management, 8(4), 221-226.
Doll, W. J., Raghunathan, T. S., Lim, J.-S., & Gupta, Y. P. (1995). A confirmatory factor analysis of the user information satisfaction instrument. Information Systems Research, 6(2), 177-188.
Donnell, A., & Dellinger, M. (1990). Analyzing Business Process Data: The Looking Glass. Indianapolis, Indiana: AT&T Quality Steering Committee.
Drucker, P. F. (1989). The New Realities. New York: Harper & Row.
Eccles, R. G. (1991). The Performance Measurement Manifesto. Harvard Business Review, 69(1), 131-137.
Ein-Dor, P., & Jones, C. R. (1985). Information Systems Management. New York: Elsevier.
Ein-Dor, P., & Segev, E. (1978). Organizational context and the success of management information systems. Management Science, 24(10), 1064-1077.
Feigenbaum, A. V. (1983). Total Quality Control. New York: McGraw-Hill.
Ferguson, J. M., & Zawacki, R. A. (1993). Service quality: A critical success factor for IS organizations. Information Strategy: The Executive's Journal, 9(2), 24-30.
Ford, J. C. (1994). Evaluating investment in IT. Australian Accountant, 64(11), 23-28.
Freedman, D. (1992). Those Who Can, Teach. CIO, 5(17), 46-51.
Funston, R. (1992, April). Great Expectations: Improving Customer Service. CMA Magazine, 66, 24-28.
Galletta, D. F., & Lederer, A. L. (1989). Some Cautions on the Measurement of User Information Satisfaction. Decision Sciences, 20(3), 419-438.
Gallupe, R. B., DeSanctis, G., & Dickson, G. W. (1988). Computer-Based Support for Group Problem-Finding: An Experimental Investigation. MIS Quarterly, 12(2), 277-296.
Garvin, D. A. (1988). Managing Quality: The Strategic and Competitive Edge. New York: The Free Press.
Gatian, A. W. (1994). Is user satisfaction a valid measure of system effectiveness? Information & Management, 26(3), 119-131.
Gemoets, L. A., & Mahmood, M. A. (1990). Effect of the Quality of User Documentation on User Satisfaction with Information Systems. Information & Management, 18(1), 47-54.
Goodman, P. S., & Pennings, J. M. (1977). Perspectives and issues: An introduction. In P. S. Goodman & J. M. Pennings (Eds.), New Perspectives on Organizational Effectiveness (pp. 1-12). San Francisco, CA: Jossey-Bass.
Grohowski, R., McGoff, C., Vogel, D., Martz, B., & Nunamaker, J. (1990). Implementing Electronic Meeting Systems at IBM: Lessons Learned and Success Factors. MIS Quarterly, 14(4), 369-383.
Guimaraes, T., & Gupta, Y. P. (1988). Measuring top management satisfaction with the MIS department. Omega: International Journal of Management Science, 16(1), 17-24.
Hambrick, D. C. (1981). Environment, Strategy, and Power Within Top Management Teams. Administrative Science Quarterly, 26(2), 253-275.
Hambrick, D. C., & Lei, D. (1985). Toward an Empirical Prioritization of Contingency Variables for Business Strategy. Academy of Management Journal, 28(4), 763-788.
Hamilton, S., & Chervany, N. L. (1981a). Evaluating Information System Effectiveness -- Part I: Comparing Evaluation Approaches. MIS Quarterly, 5(3), 55-69.
Hamilton, S., & Chervany, N. L. (1981b). Evaluating Information System Effectiveness -- Part II: Comparing Evaluator Viewpoints. MIS Quarterly, 5(4), 79-86.
Harris, S. E., & Katz, J. L. (1991a). Firm Size and the Information Technology Investment Intensity of Life Insurers. MIS Quarterly, 15(3), 333-352.
Harris, S. E., & Katz, J. L. (1991b). Organizational performance and information technology investment intensity in the insurance industry. Organization Science, 2(3), 263-295.
Hartog, C., & Herbert, M. (1986). 1985 Opinion Survey of MIS Managers: Key Issues. MIS Quarterly, 10(4), 351-361.
Herniter, B. C., Carmel, E., & Nunamaker, J. F., Jr. (1993). Computers improve efficiency of the negotiation process. Personnel Journal, 72(4), 93-99.
Hodgetts, R. M. (1993). Blueprints for Continuous Improvement: Lessons from the Baldridge Winners. New York: American Management Association.
Hofer, C. W. (1975). Toward a contingency theory of business strategy. Academy of Management Journal, 18(4), 784-810.
Ives, B., Hamilton, S., & Davis, G. B. (1980). A Framework for Research in Computer-Based Management Information Systems. Management Science, 26(9), 910-934.
Ives, B., Olson, M. H., & Baroudi, J. J. (1983). The measurement of user information satisfaction. Communications of the ACM, 26(10), 785-793.
Jarvenpaa, S. L., Rao, V. S., & Huber, G. P. (1988). Computer Support for Meetings of Groups Working on Unstructured Problems: A Field Experiment. MIS Quarterly, 12(4), 645-666.
Kaplan, R., & Norton, D. (1992). The balanced scorecard: Measures that drive performance. Harvard Business Review, 70(1), 71-79.
Kappelman, L. A., Van Dyke, T. P., & Prybutok, V. R. (1995). Concerns on the Use of SERVQUAL Difference Scores to Measure Information Systems Service Quality (BCIS Working Paper BCIS - WP - 951101). Denton, TX: University of North Texas, Business Computer Information Systems (BCIS) Department.
Keim, R. T., & Janaro, R. (1982). Cost/Benefit analysis of MIS. Journal of Systems Management, 33(9), 20-25.
Kelley, M. R. (1994). Productivity and information technology: The elusive connection. Management Science, 40(11), 1406-1425.
Kettinger, W. J., & Lee, C. C. (1994). Perceived service quality and user satisfaction with the information services function. Decision Sciences, 25(5,6), 737-766.
King, J. (1991, November 11). "Hello, Help Desk? Help!". Computerworld, 25, 73-75.
King, J. L., & Schrems, E. L. (1978). Cost-Benefit Analysis in Information Systems Development and Operations. Computing Surveys, 10(1), 19-34.
King, W. R., & Rodriguez, J. I. (1978). Evaluating Management Information Systems. MIS Quarterly, 2(3), 43-51.
Lakewood Publications. (1990, May). Quality improvement can be demonstrated, not delegated from the executive suite. Total Quality Newsletter, 1, 3-5.
Landen, D. L., & Landen, G. A. (1990). Understanding - And Improving - Organizations Through Cybernetics. National Productivity Review, 10(1), 5-17.
LaPlante, A., & Alter, A. E. (1994, October 31). IT all adds up. Computerworld, 28, 76-77.
Le Blanc, L. A., & Kozar, K. A. (1990). An Empirical Investigation of the Relationship Between DSS Usage and System Performance: A Case Study of a Navigation Support System. MIS Quarterly, 14(3), 263-277.
Lefrancois, R. (1984). A Challenge for the 1980s: Productivity-Oriented University Management. Cost & Management, 58(1), 55-59.
Lewin, A. Y., & Minton, J. W. (1986). Determining Organizational Effectiveness: Another Look, and an Agenda for Research. Management Science, 32(5), 514-538.
Likert, R. (1967). The Human Organization: Its Management and Value. New York: McGraw-Hill.
Lucas, H. C., Jr. (1972). Performance evaluation and the management of information services. Data Base, 4(1), 1-8.
Maglitta, J. (1993, October 18). Executives: Where's the IS payback? Computerworld, 27, 15.
Mahmood, M. A., & Becker, J. D. (1985). Effect of organizational maturity on end-users' satisfaction with information systems. Journal of Management Information Systems, 2(3), 37-64.
Mahmood, M. A., & Mann, G. J. (1993). Measuring the organizational impact of information technology investment: An exploratory study. Journal of Management Information Systems, 10(1), 97-122.
Mahoney, T. A., & Weitzel, W. (1970). Managerial models of organizational effectiveness. Administrative Science Quarterly, 14, 357-365.
Marcolin, B. L., & Higgins, C. A. (1992). Effectively dealing with IS user complaints. Journal of Systems Management, 43(12), 29-36.
Marion, L. (1992). The Trouble with Techies. CFO: The Magazine for Senior Financial Executives, 8(7), 28-33.
Markus, M. L., & Keil, M. (1994). If we build it, they will come: Designing information systems that people want to use. Sloan Management Review, 35(4), 11-25.
Mason, R. M., & Sassone, P. G. (1978). A Lower Bound Cost Benefit Model for Information Services. Information Processing & Management, 14(2), 71-83.
Mason, R. O. (1978). Measuring information output: A communication system's approach. Information & Management, 1(5), 219-234.
Matlin, G. (1979). What is the value of investment in information systems? MIS Quarterly, 3(3), 5-34.
Matlin, G. L. (1977). How to Survive a Management Assessment. MIS Quarterly, 1(1), 11-17.
McKeen, J. D., Guimaraes, T., & Wetherbe, J. C. (1994). The relationship between user participation and user satisfaction: An investigation of four contingency factors. MIS Quarterly, 18(4), 427-451.
McLean, E. R. (1973). Assessing returns from the data processing investment. In F. J. Gruenberger (Ed.), Effective vs. Efficient Computing (pp. 12-25). Englewood Cliffs, NJ: Prentice-Hall.
McLean, E. R., Kappelman, L. A., & Thompson, J. P. (1993). Converging end-user and corporate computing. Communications of the ACM, 36(12), 79-92.
McReynolds, J. S., & Fern, R. H. (1992). Improve Operations by Benchmarking "Best-Practice" Companies. Corporate Controller, 4(5), 20-24.
Meachim, N. (1994, June 15). Users to vendors: Quality, quality, quality. Datamation, 40, 38-46.
Melone, N. P. (1990). A Theoretical Assessment of the User-Satisfaction Construct in Information Systems Research. Management Science, 36(1), 76-91.
Mendelow, A. L. (1983). Setting Corporate Goals and Measuring Organizational Effectiveness - A Practical Approach. Long Range Planning, 16(1), 70-76.
Meyer, M. W., & Gupta, V. (1994). The performance paradox. Research in Organizational Behaviour, 16, 309-369.
Miller, D. (1992). The Icarus Paradox: How Exceptional Companies Bring About Their Own Downfall. Business Horizons, 35(1), 24-35.
Miller, D. (1993). The architecture of simplicity. Academy of Management Review, 18(1), 116-138.
Moad, J. (1989). Asking Users to Judge IS. Datamation, 35(21 (International Edition)), 93-100.
Moad, J. (1993). New rules, new ratings as IS reengineers. Datamation, 39(21), 85-87.
Moad, J. (1994). Successful benchmarking with the team approach. Datamation, 40(3), 53-56.
Moore, G. P. (1992). Process Interface Diagram: Viewing the Organization as a System. American Association of Cost Engineers Transactions, 2, I.1.1-I.1.6.
Mukhopadhyay, T., Kekre, S., & Kalathur, S. (1995). Business value of information technology: A study of electronic data interchange. MIS Quarterly, 19(2), 137-156.
Nath, R. (1992). A framework to identify applications of information technology to improve service quality. Behaviour & Information Technology, 11(1), 24-31.
National Institute of Standards and Technology. (1995). Malcolm Baldridge National Quality Award: 1995 Award Criteria. Washington, DC: author.
Neumann, S., Ahituv, N., & Zviran, M. (1992). A Measure for Determining the Strategic Relevance of IS to the Organization. Information & Management, 22(5), 281-299.
Niederman, F., Brancheau, J. C., & Wetherbe, J. C. (1991). Information Systems Management Issues for the 1990s. MIS Quarterly, 15(4), 474-500.
Nolan, R. L., & Wetherbe, J. C. (1980). Toward a comprehensive framework for MIS research. MIS Quarterly, 4(2), 1-19.
Nunamaker, J. F., Jr., Dennis, A. R., Valacich, J. S., & Vogel, D. R. (1991a). Information Technology for Negotiating Groups: Generating Options for Mutual Gain. Management Science, 37(10), 1325-1346.
Nunamaker, J. F., Dennis, A. R., Valacich, J. S., Vogel, D. R., & George, J. F. (1991b). Electronic Meeting Systems to Support Group Work. Communications of the ACM, 34(7), 40-61.
Oman, R. C., & Ayers, T. (1988). Productivity and Benefit-Cost Analysis for Information Technology Decisions. Information Management Review, 3(3), 31-41.
Palvia, P. C., Perkins, J. A., & Zeltmann, S. M. (1992). The PRISM System: A Key to Organizational Effectiveness at Federal Express Corporation. MIS Quarterly, 16(3), 277-292.
Parasuraman, A., Berry, L. L., & Zeithaml, V. A. (1991a). Perceived Service Quality as a Customer-Based Performance Measure: An Empirical Examination of Organizational Barriers Using an Extended Service Quality Model. Human Resource Management, 30(3), 335-364.
Parasuraman, A., Berry, L. L., & Zeithaml, V. A. (1991b). Refinement and Reassessment of the SERVQUAL Scale. Journal of Retailing, 67(4), 420-450.
Parasuraman, A., Berry, L. L., & Zeithaml, V. A. (1991c). Understanding Customer Expectations of Service. Sloan Management Review, 32(3), 39-48.
Parasuraman, A., Berry, L. L., & Zeithaml, V. A. (1993). More on improving the measurement of service quality. Journal of Retailing, 69(1), 140-147.
Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1985). A Conceptual Model of Service Quality and Its Implications for Future Research. Journal of Marketing, 49(4), 41-50.
Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1988). SERVQUAL: A Multiple-Item Scale for Measuring Consumer Perceptions of Service Quality. Journal of Retailing, 64(1), 12-40.
Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1994a). Alternative scales for measuring service quality: A comparative assessment based on psychometric and diagnostic criteria. Journal of Retailing, 70(3), 201-230.
Parasuraman, A., Zeithaml, V. A., & Berry, L. L. (1994b). Reassessment of expectations as a comparison standard in measuring service quality: Implications for further research. Journal of Marketing, 58(1), 111-124.
Pengelly, A., Norris, M., & Higham, R. (1993). Software process modeling and measurement: a QMS case study. Information & Software Technology, 35, 375-380.
Peppers, D., & Rogers, M. (1995). A new marketing paradigm: Share of customer, not market share. Managing Service Quality, 5(3), 48-51.
Pitt, L. F., Watson, R. T., & Kavan, C. B. (1995). Service quality: A measure of information systems effectiveness. MIS Quarterly, 19(2), 173-187.
Plewa, J. M., & Lyman, H. T. (1992). Keeping the Customer Satisfied: The New CIO Mandate. Information Systems Management, 9(1), 54-61.
Premkumar, G., & King, W. R. (1994). Organizational characteristics and information systems planning: An empirical study. Information Systems Research, 5(2), 75-109.
Quinn, J. B., & Baily, M. N. (1994). Information technology: Increasing productivity in services. Academy of Management Executive, 8(3), 28-48.
Quinn, R. E., & Cameron, K. S. (1988). Paradox and transformation: A framework for viewing organization and management. In R. E. Quinn & K. S. Cameron (Eds.), Paradox and Transformation: Toward a Theory of Change in Organization and Management (pp. 289-308). Cambridge, MA: Ballinger.
Rainer, R. K., Jr., & Watson, H. J. (1995). The keys to executive information system success. Journal of Management Information Systems, 12(2), 83-98.
Raymond, L. (1987). Validating and Applying User Satisfaction as a Measure of MIS Success in Small Organizations. Information & Management, 12(4), 173-179.
Reichheld, F. F., & Sasser, W. E., Jr. (1990). Zero Defections: Quality Comes to Services. Harvard Business Review, 68(5), 105-111.
Remenyi, D. S. J., & Money, A. H. (1994). Service quality and correspondence analysis in determining problems with the effective use of computer services. European Journal of Information Systems, 3(1), 2-12.
Richardson, T. A. (1994). Measuring customer satisfaction with IS services. Capacity Management Review, 22(9), 1-6.
Rochester, J. B., & Douglass, D. P. (1991). Critical Issues in Information Systems Management, 1991-1995. I/S Analyzer, 29(1), 1-16.
Rockart, J. F., & Short, J. E. (1989). IT in the 1990s: Managing Organizational Interdependence. Sloan Management Review, 30(2), 7-17.
Rolefson, J. F. (1978). The DP Check-up. Journal of Systems Management, 29(11), 38-48.
Ross, J. E. (1993). Total Quality Management: Text, Cases and Readings. Delray Beach, Florida: St. Lucie Press.
Rushinek, A., & Rushinek, S. F. (1986a). The influence of troubleshooting, education, and documentation on computer user satisfaction. IEEE Transactions on Systems, Man, and Cybernetics, 16(1), 165-168.
Rushinek, A., & Rushinek, S. F. (1986b). What makes users happy? Communications of the ACM, 29(7), 594-598.
Rust, R. T., & Oliver, R. L. (Eds.). (1994). Service Quality: New Directions in Theory and Practice. Thousand Oaks, CA: Sage.
Rust, R. T., Zahorik, A. J., & Keiningham, T. L. (1995). Return on quality (ROQ): Making service quality financially accountable. Journal of Marketing, 59(2), 58-70.
Saaksjarvi, M. V. T., & Talvinen, J. M. (1993). Integration and effectiveness of marketing information systems. European Journal of Marketing, 27(1), 64-79.
Santosus, M. (1995, October 15). Perception is reality. CIO, 9, 38-48.
Satzinger, J. W., & Olfman, L. (1995). Computer support for group work: Perceptions of the usefulness of support scenarios and end-user tools. Journal of Management Information Systems, 11(4), 115-148.
Saunders, C. S., & Jones, J. W. (1992). Measuring performance of the information systems function. Journal of Management Information Systems, 8(4), 63-82.
Scott, W. R. (1977). Effectiveness of organizational effectiveness studies. In P. S. Goodman & J. M. Pennings (Eds.), New Perspectives on Organizational Effectiveness (pp. 63-95). San Francisco, CA: Jossey-Bass.
Scudder, R. A., & Kucic, A. R. (1991). Productivity Measures for Information Systems. Information & Management, 20(5), 343-354.
Seddon, P., & Kiew, M.-Y. (1994). A partial test and development of the DeLone and McLean model of IS success. Paper presented at the Fifteenth Annual International Conference on Information Systems (ICIS).
Semich, J. W. (1994, January 7). Here's how to quantify IT investment benefits. Datamation, 40, 45-48.
Sethi, V., & King, W. R. (1994). Development of measures to assess the extent to which an information technology application provides competitive advantage. Management Science, 40(12), 1601-1627.
Seymour, D. T. (1992). On Q: Causing Quality in Higher Education. New York: Macmillan.
Shannon, C. E., & Weaver, W. (1949). The Mathematical Theory of Communication. Urbana, IL: University of Illinois Press.
Singleton, J. P., McLean, E. R., & Altman, E. N. (1988). Measuring Information Systems Performance: Experience with the Management By Results System at Security Pacific Bank. MIS Quarterly, 12(2), 325-337.
Stearns, E. W. (1984, June). The information center: The best answer? Infosystems, 31, 46-48.
Steers, R. M. (1975). Problems In the Measurement Of Organizational Effectiveness. Administrative Science Quarterly, 20, 546-558.
Strassman, P. A. (1985). Information Payoff: The Transformation of Work in the Electronic Age. New York: Free Press.
Strassman, P. A. (1990). The Business Value of Computers: An Executive's Guide. New Canaan, CT: Information Economics Press.
Tayntor, C. B. (1994). Partners in excellence: Metrics and productivity programs. Information Systems Management, 11(1), 81-83.
Thierauf, R. J. (1988). Effective Information Centers: Guidelines for MIS and IC Managers. New York: Quorum Books.
Thompson, G. B., & Cannon, W. G. (1978). Hospitals, like industry, must apply cost accounting techniques. Hospitals, J. A. H. A., 52, 129-132.
Thornburg, L. (1991). Performance Measures That Work. HRMagazine, 36(5), 35-38.
Torkzadeh, G., & Doll, W. J. (1993). The place and value of documentation in end-user computing. Information & Management, 24(3), 147-158.
Trowbridge, D. (1995). Benchmarks should be as individual as your fingerprints. Computer Technology Review, 15(3), 1, 12+.
Tsui, A. S. (1994). Reputational effectiveness: Toward a mutual responsiveness framework. Research in Organizational Behavior, 16, 257-307.
Tyran, C. K., Dennis, A. R., Vogel, D. R., & Nunamaker, J. F., Jr. (1992). The Application of Electronic Meeting Technology to Support Strategic Management. MIS Quarterly, 16(3), 313-334.
Van de Ven, A. H., & Ferry, D. L. (1980). Measuring and Assessing Organizations. New York: Wiley.
Van Dyke, T. P., Prybutok, V. R., & Kappelman, L. A. (1995). Cautions on the Use of the SERVQUAL Measure to Assess the Quality of Information Systems Services (BCIS Working Paper BCIS - WP - 950701). Denton, TX: University of North Texas, Business Computer Information Systems (BCIS) Department.
Weill, P., & Olson, M. H. (1989). An assessment of the contingency theory of management information systems. Journal of Management Information Systems, 6(1), 59-85.
Wells, C. E. (1987). Evaluation Of the MIS Function In an Organization: an Exploration Of the Measurement Problem (Doctoral Dissertation, University Of Minnesota). Dissertation Abstracts International, 48, 2390A.
Wetherbe, J. C., & Whitehead, C. J. (1977). A contingency view of managing the data processing organization. MIS Quarterly, 1(1), 19-25.
Wilkes, R. B. (1987). Assessment Of the Information Systems Organization: an Empirical Investigation Of Assessor Perspectives (Doctoral Dissertation, University Of Minnesota). Dissertation Abstracts International, 48, 173A.
Wilkes, R. B., & Dickson, G. W. (1987). Assessment of the Information Systems Organization: An Empirical Investigation of Assessor Perspectives. Paper presented at the Eighth Annual International Conference on Information Systems (ICIS), Pittsburgh, PA.
Zeithaml, V. A., Berry, L. L., & Parasuraman, A. (1988a). Communication and Control Processes in the Delivery of Service Quality. Journal of Marketing, 52(2), 35-48.
Zeithaml, V. A., Berry, L. L., & Parasuraman, A. (1993). The nature and determinants of customer expectations of service. Journal of the Academy of Marketing Science, 21(1), 1-12.
Zeithaml, V. A., Parasuraman, A., & Berry, L. L. (1990). Delivering Quality Service: Balancing Customer Perceptions and Expectations. New York: The Free Press.
Zeithaml, V. A., Varadarajan, P., & Zeithaml, C. P. (1988b). The Contingency Approach: Its Foundations and Relevance to Theory Building and Research in Marketing. European Journal of Marketing, 22(7), 37-64.
Zmud, R. W. (1979). Individual Differences and MIS Success: A Review of the Empirical Literature. Management Science, 25(10), 966-979.
Last Modified: Monday, 10-Feb-97 00:00:00