Evaluating Expert System Tools

This report summarizes the results of a study undertaken to develop criteria for evaluating and selecting tools used to build expert systems. The authors used an evaluation framework composed of five elements: (1) application characteristics, which describe the problem and the project to be undertaken; (2) tool capabilities, the capabilities that the tools support; (3) metrics, the quantitative and qualitative measures of merit for expert system tools; (4) assessment techniques, specific ways of applying metrics to tools; and (5) contexts, which describe the ways in which the evaluation criteria depend on the development phases targeted by a project. Many of the study's conclusions relate to software engineering aspects of the expert system endeavor. Robustness, reliability, portability, integrability, database access, concurrent access, performance, and user interface all appear to be increasingly important requirements for tools, as well as eventual requirements for the expert systems that will be produced with those tools. In addition, the expert system paradigm seems to have had a significant and beneficial effect on software engineering itself.