Skip to main content
ARS Home » Research » Publications at this Location » Publication #67416
Title: SUBJECTIVE EVALUATION OF DECISION SUPPORT SYSTEMS USING MULTIATTRIBUTE DECISION MAKING (MADM)

Author
item Ascough Ii, James
item DEER-ASCOUGH, LOIS - COLORADO STATE UNIVERSITY
item Shaffer, Marvin
item Hanson, Jonathan

Submitted to: Multiple Objective Decision Support Systems for Land, Water, and Environment
Publication Type: Book / Chapter
Publication Acceptance Date: 8/1/1997
Publication Date: N/A
Citation: N/A

Interpretive Summary: This paper presents a generic framework for considering decision support system (DSS) evaluation issues and a multipurpose approach for selecting categories of DSS evaluation techniques. The paper suggests MADM decision support evaluation hierarchy as a formal structure for combining subjective effectiveness measures (SEMs) into a single overall assessment of DSS utility. A case study was performed to assess the usefulness of MADM in evaluating symbolic and schematic GUI's for the USDA-ARS GPFARM DSS. Evaluation was based on judgments concerning the input screens, output screens, help screens, miscellaneous characteristics, and operational concerns associated with the two GUI's. A simple additive weighting (SAW) value function was then used to calculate the overall utility of each GUI. It was concluded that the evaluation team preferred the symbolic GUI over the schematic GUI on the basis of a higher overall utility calculated for the symbolic GUI. The case study, although conceptually simple, is significant in that it represents a first step in conducting a reliable evaluation for USDA-ARS sponsored DSS. Future, more complex evaluations of GPFARM will increase the probability that it will be used and will be beneficial. GUI (score or utility of 0.647) on the basis of the higher overall utility

Technical Abstract: Structured evaluation of decision support system (DSS) development stages may be found in development plans, however, formal and systematic evaluations of "real world" decision support systems are rarely performed throughout the creation process. As a result, we relinquish the chance to gain valuable information about what potential customers think about the system, how well the code is written, the extent to which the system actually supports decision making, etc. Multiattribute decision making (MADM) is presented herein as a mechanism to account for different attributes or system effectiveness measures (SEMs) considered important when evaluating a decision support system and as a means for combining the SEMs into a comprehensive measure of value, that is, a single overall assessment of system utility. MADM is an appropriate tool for evaluating decision support systems in that it is capable of distinguishing between satisfactory and unsatisfactory systems, or components of a system. A cas study is presented showing how MADM was used to evaluate symbolic and schematic Graphical User Interfaces (GUI's) for the USDA-ARS Great Plains Framework for Agricultural Resource Management (GPFARM) DSS. It was concluded that the evaluation team preferred the symbolic GUI(score or utility of 0.703) over the schematic GUI(score or utility of 0.647) on the basis of the higher overall utility calculated for the symbolic GUI.