ESSCOTS For Learning

This paper discusses ESSCOTS systems—Educational Support Systems based on Commercial-Off-The-Shelf software. ESSCOTS systems are developed by selecting an existing commercial software product (the "COTS") around which we build an educational software environment (the "ESS"). Our initial experiences indicate that the ESSCOTS approach to developing educational software has several advantages over developing systems from scratch. We begin with an overview of a specific ESSCOTS system based on ARC/INFO, a geographical information system. We then describe the formative results of piloting the ARC/INFO ESSCOTS system with middle- and high-school students, focusing on some especially impressive inquiries that the ESSCOTS system enabled. The second half of the paper discusses more broadly the potential benefits of developing educational software as ESSCOTS systems rather than "from scratch". We argue that ESSCOTS systems can suggest project-based curricula and curriculum reform ideas, stimulate connections between classrooms and work, and profitably couple science, business, and education. We also suggest that ESSCOTS systems can provide important dual-use applications of technology, consistent with the Technology Reinvestment Project.

Introduction

Over the past several years we have been developing a collection of microworlds that support learning through student-centered inquiry or discovery (see Lewis, McArthur, Bishay & Chou, 1992; Lewis, Bishay & McArthur, 1993; McArthur & Lewis 1991a, 1991b; and McArthur, Lewis & Bishay, 1993). Of course, interest in inquiry-based learning is not new. Among others Cuban (1986) and Cohen (1988) have chronicled several cycles in which inquiry has waxed and waned in popularity, at least among academics and educators. But they note the overall trend-line in the classroom has remained firmly away from inquiry-based learning and towards instruction-based pedagogy. In developing microworlds, our main goal is to demonstrate that new information technologies can empower inquiry-based learning and—for the first time—make it a practical means to acquire knowledge.

The important questions our research addresses include:

• Can we develop intelligent AI-based software tools that support learning by inquiry?
• By changing the model of learning from instruction to inquiry, can AI-based environments support the learning of new and interesting topics in mathematics, not just the algorithmic skills commonly taught by intelligent tutoring systems?
• Does the quality or quantity of learning differ in instruction-based and inquiry-based environments? How can we measure these differences?
• Can inquiry skills themselves be learned by students in a supportive environment?

The study of students' inquiry-skills themselves is a critical aspect of our research. At present, there is little cognitive theory describing how students learn through inquiry (see Shute & Glaser, 1990, and van Berkum & de Jong, 1991 for a few important exceptions). We have been attempting to develop such a theory as we build our microworlds for two reasons. First, an understanding of the specific cognitive processes students engage in inquiry—especially characteristic problems or "bugs" (Brown & VanLehn, 1980; Sleeman, 1982)—can inform the design of pedagogical tools embedded in the microworlds. An effective set of tools will empower processes that are generally strong, but may more actively coach those that are weak. Second, such a theory also helps to formulate general principles for designing powerful inquiry learning environments. Although each microworld may have only a limited life, these principles can represent enduring research contributions.

In this paper we discuss the most recent microworld in our collection. While it retains most of the goals and philosophy of previous systems, it is novel in one respect: Instead of building it "from scratch", we constructed it as an ESSCOTS system—Educational Support Systems based on Commercial-Off-The-Shelf software. ESSCOTS systems are developed by selecting existing commercial software products (the "COTS") around which we build educational software environments (the "ESS"). Our initial experiences indicate that the ESSCOTS approach to developing educational software has several advantages over developing systems from scratch. Thus, while we will describe some preliminary results in fielding our first ESSCOTS microworld, the main intent of this paper is to use the microworld to illustrate some of the potential benefits of building educational software on top of commercial systems.

The first sections of the paper will define the ESSCOTS approach to developing microworlds and educational software in general. In that context we introduce our ESSCOTS software based on ARC/INFO, a geographical information system. We then describe the initial results of piloting the ARC/INFO ESSCOTS software with middle- and high-school students. The final sections discuss more broadly some potential benefits of this approach to educational software development, drawing on the ARC/INFO ESSCOTS system to highlight general points.

ESSCOTS Systems and Microworlds

Each of our previous microworlds comprise several parts. First, the application software defines the graphical interface for the microworld and its particular topics of inquiry. For example, in the Polygons microworld (where students explore the geometry of regular polygons), the application software includes all object-oriented procedures for creating, drawing, and managing regular polygons (McArthur & Lewis, 1991a). Second, each microworld also includes generic inquiry tools that students use to gather and represent data about objects and their properties, represent and manage conjectures, and relate conjectures to confirming and disconfirming data. For example, in Graph Theory (where students explore graphs in discrete mathematics) data about properties like planarity or the degree of vertices might be represented in tables or Cartesian graphs (McArthur, Lewis & Bishay, 1993). Third, the microworlds include formative versions of smart agents. While generic tools provide passive supports for inquiry, smart agents play a more active role. For instance, while passive hypertext cards might passively record conjectures, active agents can find data that bear on a conjecture (e.g., in Graph Theory finding a graph that has vertices of only even degree) or determine if a datum is consistent with a conjecture.

In the past we have developed our microworlds largely "from scratch". Beginning with only an object-oriented dialect of Lisp (CLOS) and powerful windowing and interface-building tools (Common Windows), we have built all the application software, generic inquiry tools, and smart agents comprising each new microworld. Later microworlds cost less to develop in part because generic inquiry tools (e.g., tables of values, Cartesian graphs, active connection between representations) were common across microworlds, as were some smart agents. However, the cost of developing application software for each new microworld remained high.

We are now beginning to pursue a different method for building our future inquiry microworlds. Instead of building new microworlds from scratch we are developing ESSCOTS systems. Each ESSCOTS system is developed around an existing commercial software product. The COTS provides all application software for a microworld, and possibly many of the generic inquiry tools as well. Our "value added" (the "ESS") is the software that turns the powerful commercial tool into a powerful educational one.

We distinguish ESSCOTS systems from other educational uses of existing software. Strictly speaking, educational systems are rarely built from scratch, because they are programmed in higher-level languages, not machine code. Indeed over the past decade, specialized tools—including graphical user interfaces, intelligent tutoring system "shells", and authoring languages (e.g., Anderson & Pelletier, 1991; Forbus, 1991; Sleeman, 1987)—have greatly simplified the construction of intelligent tutoring systems. However, these are generic software construction tools, whereas in ESSCOTS software the kernel COTS is part of the educational system, not just a means to build it.

Attempts to use "systems thinking" software in education come closer to the ESSCOTS approach. Stella (Richmond, 1993), for example, is a simulation package that permits users to understand the dynamic behavior of complex systems. Because it has an excellent user interface, Stella can be used directly by students. Students can create sophisticated models without writing any computer code, and they can easily interpret results through Stella's simple visualization tools. An ESSCOTS system also begins with such a commercial system; however additional software is added to turn a professional application into an educational one. In short, the ESSCOTS approach extends the use of commercial software in education several ways. As we elaborate below, our intention in developing ESSCOTS systems is two-fold. First, we are developing specific microworlds for learning built on COTS. Second, we are compiling and evaluating general principles that help identify existing COTS which might be the basis for powerful educational environments, and that describe what characteristics ESSes require to make these COTS into effective ESSCOTS systems.

The ARC/INFO ESSCOTS System

Our first ESSCOTS system was built using ARC/INFO as the COTS. ARC/INFO is a geographic information system (GIS) used primarily by cartographers, demographers, urban planners, and resource policy analysts. ARC/INFO combines factual data with cartographic information for geographically based applications and analysis.

The intent of our ARC/INFO ESSCOTS system was not simply to help students become expert ARC/INFO users nor to become amateur urban planners. Although ARC/INFO is widely used today, it will probably be obsolete within a few years, superseded by much more powerful environments. We believe it is much more important for students to acquire a general familiarity with the concepts of GIS, and with the symbolic and visual reasoning skills needed to conduct productive inquiries using massive databases to answer realistic questions in demographics, urban planning, and social policy.

Our pedagogical strategy for the ARC/INFO ESSCOTS system was similar to the strategy we used with our previous microworlds. We encouraged students to conduct self-directed inquiries by examining patterns of variables—this time using large ARC/INFO databases rather than mathematical objects like polygons or graphs. We asked students to formulate issues (e.g., What are the correlates and possible social causes of high teen pregnancy rates?), generate conjectures (e.g., Teen pregnancy rates might be related to ethnicity, education, and income), and use visualization tools, such as maps, as well as tables and graphs to disconfirm conjectures (Teen pregnancy is not strongly related to ethnicity) and to confirm them (Teen pregnancy is strongly related to education and income).

To facilitate these inquiries we developed an educational support system (ESS) to interface with ARC/INFO. Normally (without our ESS), ARC/INFO uses a complicated command line language to perform its functions. Even simple things such as drawing a map and placing a legend require the user to type in several commands and keep track of several variable and dataset names. The company that produces ARC/INFO (ESRI) offers week-long courses on the use of its system, so clearly the standard ARC/INFO system is not easy for professionals to use, and definitely not suited for students.

Fortunately, ARC/INFO includes the ARC/INFO Macro Language (AML) which lets a user group series of commands into subroutines that can be called with a single command. AML also has capabilities for designing a customized, menu-driven interface. We used both of these AML capabilities to create our ESS. We also created a dataset we thought would be interesting to students by combining pieces of the commercially available "demographic and health" and "socioeconomic" ArcUSA datasets. Figure 1 shows the interface to our ARC/INFO ESSCOTS system that students see when they first encounter the system, and a list of variables the students can examine.

Our ESS provided some functionalities not available in ARC/INFO and restructured interaction with the system in important ways. As the sample inquiry below makes clear, the ESS provided several kinds of generic inquiry tools to structure students' inquiries, gather and represent data, and manage conjectures. For example, because ARC/INFO is primarily designed for geographically based inquiries, its map making capabilities are much stronger than its graphing or statistical capabilities. Hence, we needed to create several graphing and statistical subroutines in AML to give students the ability to look at the data in multiple representations

More importantly, we designed our ESS to dynamically adapt to the students' level of expertise and needs. The ESS interface contained a "main menu" (at the top of Figure 1) that was simple, providing just enough functionality so students could quickly focus on learning with the system, not about it. But the interface also permitted incremental increases in complexity and functionality through options in the "custom menu" (see the bottom of Figure 2) which could be accessed through a button on the main menu. In the custom menu, the students could increase the number of variables in their dataset or increase the size of the region they were investigating (from East Coast to the entire United States, for example). The custom menu also contained the more complicated inquiry tools such as the scatter plot regression and Pearson's R which are useful for an advanced student but might be confusing for a student using the system for the first time. Finally, the custom menu included tools (hotspot, text, statistics tools for example) that were analogous ones in the main menu but which gave students many more options from which to choose. The examples below will show how these tools structured and facilitated student inquiries.

A sample inquiry

In general, because we began with a simple interface, and permitted students to dynamically add complexity on an as-needed basis, students were able to master the tools in the first session, and thus quickly concentrated on inquiries rather than software. The following is representative of student inquiries after a couple of sessions.

At the beginning of this inquiry the students scrolled through a list of possible variables (see Figure 1) and decided to investigate teenage pregnancy, P_BIR_TEEN (percentage of births to mothers under 20). In the early sessions the students were only told about the dataset that contained a short list of variables. As the students became better at using the system and at managing inquiries they were given the option of using a richer dataset that contained many more variables. After choosing a variable, students would usually make a "hot spot" map to see the geographical distribution of the variable. A hotspot map shows the values of a variable (here P_BIR_TEEN) partitioned into quantiles (intervals containing a constant-sized range of values). Figure 2 shows the "hotspot" map the students created (see Table 1 for a definition of all main menu buttons). In the map, each county in the eastern seaboard states^[1] is colored to reflect its value—red for very high teen pregnancy, down to blue for very low rates.^[2] A legend interprets the meaning of each color for the students. The hotspot map shows visually and immediately what numerical statistics struggle to capture: The percentage of teenage pregnancy is much higher in the south than the north.

After looking at the map as a whole, the students often would "zoom in" on a particular part of the map. In figure 2, the students have zoomed in on the Washington DC area. By zooming in on a small part of the map, students would often notice a county that seemed to be very different from its neighbors and use the "identify county" option in the main menu to find out the name of the county and information about it. For example, while investigating the distribution of doctors in America, the students noticed that a county in Minnesota had a very high doctor per capita ratio. By identifying the county, they found out it was Rochester, the location of the Mayo Clinic.

Show features and annotate buttons:

• Outline states:Include outlines of states
• Outline counties: Include outline of counties
• Show cities: Show all major cities, including names
• Text

Manipulate visual display buttons:

• Save & Clear: Save maps and tools and clear the screen
• Refresh: Redraw the display
• Delete: Delete the last element added to the display
• Move: Move the last element added to the display

Identify element buttons:

• Identify county: Point to a county on a map; get all its variables
• Identify city: Point to a city on a map; get its name
• Find county: Type in partial county name; it is highlighted on map
• Find city: Type in partial city name; it is highlighted on map

Display data and statistics buttons:

• Hotspot: Color-coded presentation of county data
• Statistics: Simple descriptive statistics of selected variable
• Bar Graph: Bar graph of variable values by interval
• Scatter Plot: Cartesian plot of two variables; includes tool to inspect points

Administrative buttons:

• Definitions: List of variable names and definitions
• Quit: Quit ARC/INFO
• Custom menu…: Invokes the menu of custom options and tools

In this inquiry, the clear patterns of teenage pregnancy encouraged students to explore further. The students began by drawing on their own experience, offering informal "causal stories" that would explain the higher teenage pregnancy in the south. They hypothesized that high teen pregnancy might be related to poor education, low income, ethnic background, or socio-economic status (SES). They then looked for database variables that might capture these factors, quickly finding P_HS_GRAD (percent of 25 year-olds in the county who graduated high-school). They then created a hotspot map of high school graduation rate (the left-hand map in Figure 3) and placed it beside the map of teen pregnancy. Looking at this map, the students noticed immediately that the general pattern "reversed" teen pregnancy—the northern states had high graduation rates while southern states were low. This both fit their expectations for that variable, and confirmed their idea that poor education might "cause" high teen pregnancy.

To examine this bivariate relationship in more detail, the students created a scatter plot of the two variables, displayed in the middle of Figure 3. The plot clearly shows the inverse relationship of teenage pregnancy rates and education level. The students then used a scatter plot tool to identify the counties that were "doing well", low teenage pregnancy and high graduation rates (these counties are displayed in the upper right hand area of Figure 3). At this point the students could have decided to look more closely at these counties and try to explain why the counties are "successful", or they could have used the scatter plot tool to identify counties that did not fit the general trend, or they could have looked more closely at the counties that were doing "poorly". These students decided to quantify how strong the correlation was by getting a Pearson's R coefficient (See Figure 3).

Although this sample exploration did not use them, our interface provided some additional tools to help gain insights into possible multivariate relationships. In particular, through the "Custom Menu" button (see Figures 2 and 3), more advanced students could choose "3-variable Graph" (scatter plots where values of a third variable are denoted by color) and "Scatter Plot Regression" (which allowed the user to exclude data on a scatter plot, then recompute correlations). In effect, these advanced tools provided students with an informal, visual stepwise multiple regression facility. Such tools were not included in the original ARC/INFO ESSCOTS system we designed, since we had not expected high-school and middle-school students to understand these analytic concepts at any level. But the students effectively demanded them—situated in the context of variables and relationships that are meaningful to the students, what we take to be highly abstract concepts became natural and concrete.

Procedure

Our initial pilot studies with the ARC/INFO ESSCOTS system have been very informal. We piloted the system with 2 pairs of students (one pair of 15 year-old boys, and one pair of 17 year-old girls) and one solo learner (a 12 year-old girl). Both pairs were friends. We chose students who were fairly sophisticated technology users (although by no means "wizards"), and had past familiarity with computers, windows and mice. In addition, all students were highly verbal, and thus were able to articulate their goals for inquiry, and to share their opinions on how useful they found the tools in supporting inquiries. Sessions with the ARC/INFO ESSCOTS system lasted at least two hours, and the number of sessions per group ranged from two (the solo girl) to six (the boys). Since we wanted to gather detailed data on the content of student inquiries and on specific student inquiry strategies, all sessions were videotaped using two cameras.

As with our initial pilot studies with previous microworlds, these sessions were only minimally structured for students. We wanted to provide an open environment for discovery where students could choose variables of personal interest, generate possible relationships between variables, and carry out all parts of their inquiry with minimal scaffolding or guidance from the human mentor at hand during all sessions. This approach contrasts sharply with many forms of classroom exploration where the goal is often established ahead of time and guided closely by the teacher or curriculum.

By organizing and guiding inquiries minimally we can see the strengths and weaknesses of students' inquiry skills when supported only by microworld software. This approach provides data with which to improve the design of software tools and to create appropriate curriculum materials and mentoring strategies. These new designs and supporting materials are then tested in more structured studies. In short, our research strategy is a design-and-test cycle, and this study reports results from only the first iteration in that cycle.

To introduce the ARC/INFO ESSCOTS system a mentor modeled a simple inquiry, carefully describing each action, and responding to student questions. The mentor selected a variable, used several tools to reveal patterns in the data, generated hypotheses regarding these patterns and discussed possible ways to explore the hypotheses. Thereafter, the mentor "faded", and provided little coaching during sessions. The general student task was thus articulated roughly as "find patterns of data that are interesting to you and see if you can find reasons for these patterns".

Based on our previous experiences with inquiries in a statistics microworld, we provided preformatted "lab book pages" for student use. On these pages, they recorded their issues, plans, and results, as well as ideas for future inquiries. As additional information sources we also provided reference materials such as hardcopy atlases and descriptions of various counties.

Results

In this section we briefly describe some results of students using our initial ARC/INFO ESSCOTS system. The report of results is by no means complete; the main intention is to describe some highlights that we then use to frame a broader discussion of the educational potentials of an ESSCOTS approach.

Highlights From Student Inquiries

Students spent high time on task. The students were so engaged in learning about the data that they easily spent two hours per session using the system. Although we had hoped sessions would last this long, we still find it significant that all students were happy to spend about three times longer with the ESSCOTS system than they spend in typical school classes. Certainly the novelty of the ARC/INFO tools and the experimental (i.e., videotaped) setting had some positive effect on student interest. But we think two other factors are also important. For one, we know from piloting previous microworlds and a student-centered course on statistics (McArthur, Robyn, Lewis & Bishay) that project-based curricula benefit from longer class periods. A 50-minute class, of which often 20 minutes is taken up with administrative chores, simply does not provide the time necessary for extended inquiries.

Students were engaged by the subject matter and this leveraged their subject-specific learning. The meaningfulness of the ARC/USA data, made accessible through visualization tools, also appeared to encourage extended student time on task. All our students had some common knowledge, perhaps even school knowledge, of American demographics. They knew enough to have expectations (e.g., the inverse relationship of teen pregnancy and education), to generate predictions, and to be gratified (or even worried or relieved!) when these predictions were confirmed. They also knew enough to be surprised when their expectations were violated.

One student made a surprising discovery while investigating the relationship between ethnicity and crime. She hypothesized that there would be a positive relationship between a county's crime rate and the percentage of its population that was part of a racial minority group. These variables were meaningful to her because as a member of a racial minority, she was hoping that her hypothesis would be false. When she made hotspot maps of the two variables she could see that they looked different. These maps gave her qualitative data that her hypothesis might be wrong. She was happy, but she knew that she needed more quantitative data before she rejected her hypothesis. She decided to compare the two variables in a scatter plot. The scatter plot showed no clear pattern between the variables. She then did a Pearson's R and found that the correlation coefficient between these variables was much smaller than the coefficients she had seen for other variables she had previously investigated (such as pregnancy and education). At this point, she confidently rejected her original hypotheses and was greatly relieved.

These surprises led to new knowledge about familiar variables. We believe such surprising results, and generally high affective involvement, contributed to students' willingness to continue inquiries for extended periods.

Students conducted some unexpectedly sophisticated inquiries and analysis. While our students were certainly not perfect scientists, we were generally impressed with their inquiries. No student needed more than about an hour of mentoring before they were ready to conduct virtually solo inquiries. Thereafter, mentors rarely offered help unless requested. Requests generally were treated by the mentor as opportunities for on-demand learning; for example, ways of introducing new tools from the Custom Menu that were apropos the current inquiry.

All students quickly transitioned from qualitative to quantitative explorations, and from explorations that were merely descriptive to more sophisticated explanatory inquiries. These, in turn required students to compare patterns of two or more variables. The students seemed to understand this strategy almost spontaneously, and had little trouble selecting and comprehending the appropriate tools for multivariate inquiries. In fact, as we noted above, it was their willingness to grapple with these complexities that forced us to develop additional analytic tools.

A pair of boys was interested in investigating teen pregnancy. They predicted that teen pregnancy rates would be high in minority groups. The made a hotspot map of teen pregnancy rate and found it was high in Georgia and low in New York. After seeing the map they felt they needed to explain the pattern so they looked for another variable which might be related to teen pregnancy rates. They decided to look at high school grad rates. The two maps looked almost opposite so they concluded that the more education the lower the teen pregnancy rate. They then made a scatter plot of the two variables. Next they searched for other variables which could be related to teen pregnancy. They made a hotspot map of income and found it looked similar to the map of high school grad rates but opposite of the map of teen pregnancy rates. They ended their exploration by discussing the three-way relationship between the variables they had investigated.

Students' inquiries focused both on characterizing general data patterns, and explaining data outliers. Unlike many statisticians who focus on general trends in the data, our students often were more interested in outliers—counties which seemed different from the others. Our tools often made outliers readily apparent. For example, hotspot maps revealed general patterns, but also dramatically highlighted outliers—one or two counties colored red amid a sea of surrounding blue ones. Similarly, scatter plots showed where the data points clustered but also showed the points far from the cluster.

Outlier counties showed the students that relationships between factors always have variability; there are many exceptions to the general trends and these exceptions require explanations. Sometimes the outliers were easy to explain and sometimes not.

While investigating the relationship between high school grad rates and county per capita spending on education, our pair of boys found a county which seemed very unusual. The county had a very high high-school grad rate but it spent almost no money on education. After looking closer at the county, they realized that the unusual county was Chatahoochee, the location of Fort Benning. Since Fort Benning has a military school, most of the education money for that county came from the federal government. The boys began to understand that looking only at the money spent by the county could be misleading.

Students understood types analysis tools they had never formally studied. None of our students were trained to interpret Cartesian graphs in school, but they were all able to use and understand our graphing tools. In the example above of the student investigation of ethnicity and crime, the student was able to correctly interpret the scatter plot because she understood the variables and had expectations of the relationship between the variables based on the maps of the variables she had previously seen. Generally we found that when students have built a strong understanding of the meaning of variables and their relationships, this understanding can guide their interpretation of the formal syntax of novel tools. In effect, they can leverage this knowledge to learn sophisticated tools on an as-needed basis.

Young students conducted effective inquiries. Although our sample is too small to make a strong argument, we did not notice many differences between the inquiries of our oldest (17) and youngest (12) students. Perhaps the older students were somewhat more skilled at managing inquiries—formulating and noting hypotheses, gathering appropriate data, for example. But differences here were slight at best, possibly because none of these students had received formal schooling in these kinds of inquiries.^[3] Moreover, any cognitive advantage enjoyed by the older students seemed offset by their timidness. The younger students seemed more willing to look for interesting patterns in a wider range of variables.

Students' inquiry skill exhibited several limitations. Although students conducted many interesting inquiries, we also noted several difficulties. These are of particular interest because they provide important data for redesigning the ARC/INFO ESSCOTS software and supporting materials, like curricula. Two consistent patterns are worth brief mention. First, especially in early sessions, there was substantial "tool hacking"—bringing up a tool, and putting data in it with no particular inquiry purpose in mind—except, possibly, to create a visually pleasing piece of graphical art.

On the other hand students also appeared to have inquiry problems when they were "on task". Significantly, most student problems were not classical scientific reasoning errors—for example, gathering too little data to support a hypothesis, gathering inappropriate data, or inferring relationships among variables on the basis of "positive" evidence alone. These low- to mid-level inquiry skills were relatively well understood even by our 12-year-old. Rather, most difficulties involved higher-level planning activities—more strategy than tactics. For example, in several sessions students needed the mentor to suggest particular variables as topics for inquiry. And in other sessions students would select topics, formulate plans to gather data and refine hypotheses, but then fail to adhere to the plan. For instance, they might begin by gathering descriptive information on one variable, see an interesting outlier, look at the values for that county on all variables, and then switch focus to one of these variables—without completing the initial plan. This behavior is related to difficulties we have referred to elsewhere as "thrashing" (McArthur & Lewis, 1991a, 1991b; Lewis, Bishay, & McArthur, 1993). In general, throughout the sessions much of the mentor's role was to curb thrashing and keep students organized by articulating or establishing relatively high level planning activities.

Discussion of the ARC/INFO ESSCOTS System

In this section we draw on some of our pilot results to motivate several reasons for developing educational software as ESSCOTS systems, instead of "from scratch". In the following section we discuss the relationship of ESSCOTS systems to a broader set of issues.

Cost-effectiveness. Our first motivation for developing new microworlds as ESSCOTS systems was simply to reduce the costs, in time and money, of creating each new world. We hoped this cost reduction would permit us to develop more microworlds, or would allow us to invest the savings in more careful evaluation and implementation studies. As we discuss below, we have subsequently broadened our reasons for developing ESSCOTS software, nevertheless cost-effectiveness is still central.

Our pilot results indicate that ESSCOTS software can indeed be cost-effective. The ARC/INFO ESSCOTS system seems to provide a high-quality learning environment for students, although it is premature to attempt any detailed comparison of current outcomes with those engendered by our other microworlds, or similar environments (e.g., Shute & Glaser, 1990). However, what makes this ESSCOTS system cost-effective is not that it can increase learning benefits, but that it dramatically reduced development time. We carefully tracked time required to program the ARC/INFO ESSCOTS software. Our programmer was originally hired as a research assistant but had some programming courses as part of her undergraduate degree in cognitive science. She had no prior experience with ARC/INFO, yet she created the first version of the interface, written with ARC/INFO's AML (ARC/INFO Macro Language), in about 35 hours. The total time required to develop the current version (including iterative refinements) was less than 70 hours. This is very rapid, compared to the development time for our other microworlds and compared to other educational software development projects, where the time usually is measured in person-months or person-years. In short, we estimate about an order of magnitude reduction in development time and cost.

Problem-oriented multi-disciplinary focus to learning. While our ARC/INFO system demonstrates that ESSCOTS systems can be built relatively inexpensively, it also suggests that microworlds developed as ESSCOTS systems may change educational outcomes and goals, not merely help us achieve existing ones at less cost. ESSCOTS systems organize learning and knowledge in new ways. Students using the ARC/INFO ESSCOTS software are not learning traditional subjects; rather, they have the opportunity to acquire knowledge orthogonal to the way traditional subjects divide up expertise. For example, by examining databases that are neither fictional nor simplified, the students were learning important inquiry skills (e.g., how to organize and plan inquiries with massive databases), and even began to address key social issues. To take one case, while investigating whether crime is related to ethnicity or education, our students naturally began to debate important social policies — should crime be attacked by spending more on law enforcement, or less directly by investing more in education?

In general, we believe that ESSCOTS systems can expose students to concepts and subjects that are important but rarely available in schools. Curricula built around ESSCOTS software will likely be multi-disciplinary, cutting across traditional academic subject boundaries—just as the kernel COTS themselves follow professional boundaries, not academic ones. COTS are problem-oriented tools, not discipline-oriented, and thus lend themselves to ESSCOTS systems and curricula built around projects and problem-solving. Instead of studying economics from a book, students could, using simulation and data analysis tools (e.g., Ithink or Stella), confront major problems facing policy makers and citizens today, such as how to reduce the national debt. Instead of learning the history of social policy and entitlement programs in theory, students may develop projects using risk analysis software (e.g., DEMOS; Morgan, 1993) or argument analysis systems (e.g., gIBIS; Conklin & Begeman, 1988) to project the ballooning cost of medical coverage and to suggest possible solutions.

Situated and authentic learning. As the above examples suggest, while ESSCOTS systems can transform the curriculum structure for learning, they can also enable students to participate in relatively authentic scientific, research, and business practice. We believe that students need not (perhaps should not) attempt to faithfully mimic the activities of professionals using the underlying COTS. For example, our students had relatively free reign, and focused mainly on exploring patterns of relationships that were personally meaningful, not necessarily ones that policy analysts might examine. However, their activities were authentic in the sense that they examined variables and issues meaningful both to themselves and to our culture as a whole, conducted extended projects, and used real datasets.

As we discussed, exploring issues with real data in ESSCOTS systems can have several salutary effects. From a broad educational perspective it means that students learn while tackling real and important social and scientific problems. In this context learning is not preparation for work—it is work. From a more personal perspective, real data and problems often become meaningful for students, actually challenging and changing their personal beliefs. Our observations (consistent with many others) confirm that this leads to high time-on-task. Perhaps more surprising, meaningful data can also leverage understanding of new tools and syntax. Our students, for example, did not have a general and abstract understanding of scatter plots. But they could make sense of these plots—learn about them in a situated context—when they knew the relationships embedded in their data and could imagine what the relationships should look like in a graphical representation.

While authentic tools, practice, and data can have significant educational value, they also pose challenges and difficulties. For example, in our ARC/INFO ESSCOTS system, when using the ARC/USA database students confront the same missing data and even inaccurate data that statisticians must face. Similarly, the relationships among variables they uncover are rarely perfect, simple and clear—many correlations are weak, and students are as likely to run into bogus and confounded relationships as straightforward positive or negative ones. And, in many cases, to make sense of these relationships, students must already know a lot about the situation at hand. For example, students observing high teen pregnancy rates in the south are unlikely to find covariates in the ARC/USA data sets unless they have knowledge (or preconceptions) about what life is like in southern states. Finally, as with many problems situated in real practice, there is rarely one "right answer". High teen pregnancy may be related to several other variables, and more than one may play a causal role.

Our pilot studies of the ARC/INFO ESSCOTS system underscore these problems, but it is obvious that virtually any ESSCOTS system built on real tools and real situations must face similar challenges. In particular, a main task in designing the educational component of an ESSCOTS system (the ESS), is to provide supports that overcome these complexities, inherent in authentic everyday use of the COTS. In the next section we discuss these aspects of the ESSCOTS approach.

General Discussion: The ESSCOTS Methodology for Developing Educational Technology

We have argued that our ARC/INFO ESSCOTS system demonstrates some interesting student learning outcomes, and we have hinted that these benefits can apply to other educational systems developed as ESSCOTS software. In this section we look more seriously at the possibility of generalizing this success into a broad ESSCOTS approach to educational software development.

Many COTS Can Serve as the Basis for ESSCOTS Systems

Initial pilot results indicate that our ARC/INFO ESSCOTS system was cost-effective because it appeared to lead to high-quality student learning yet required relatively little time to develop. Will other ESSCOTS systems be as easy to build? Not necessarily; but we have begun to compile a list of COTS that might be good ESSCOTS kernels, and our initial impression is that many of them provide a basis on which an effective learning environment could be developed very quickly. Perhaps more to the point, the number of COTS that could qualify is growing rapidly all the time.

As we have compiled a list of candidate COTS we also have begun to abstract some general features that COTS should possess if they are to provide a basis for a useful ESSCOTS systems. These features include:

• The COTS should run on a range platforms, ideally ones that are relatively inexpensive (or on a powerful server that can manage multiple clients). It should not require an esoteric hardware or software base.
• The COTS should be tailorable to different users' needs, and it should also provide "hooks" that let programmers build new facilities as well as modify existing ones. Ideally, the COTS should be designed as an "open" system—one that permits communication between the COTS and other applications (e.g., a non-resident ESS system).
• The COTS should have an easy-to-learn and easy-to-use interface that also includes a wide range of powerful yet accessible tools. Today, this is often achieved through well-designed GUIs (graphical user interface).
• The COTS should be applicable to a wide range of problems that could be interesting to students, not just a narrow set of puzzles of interest to one profession.

For several reasons, the number of COTS that satisfy these conditions has grown dramatically. First, powerful machines have become cheap. Ten years ago neither powerful COTS nor AI-based educational systems could run on machines costing less than $50,000. But today platforms like Macintosh comfortably run AI-based systems, educational environments, and graphically-oriented COTS—all the software ingredients we need to build ESSCOTS software.

Second, COTS are increasingly geared for a wide audience, not for a few technical experts. A decade ago, powerful computer systems were the purview of a relatively few programmers, computer scientists, engineers, and researchers in the basic sciences. Today the market for sophisticated COTS is booming. The consumer market has grown in part because of a relatively new perception that computer technology can be applied to many everyday problem situations, not just esoteric scientific issues. Consumers are demanding COTS for the home (e.g., financial management and budgeting systems), business (e.g., spreadsheets, and systems analysis software) and recreation (e.g., chess-playing programs and adventure games). At the same time, this broader consumer market will not tolerate a slow learning curve. COTS that succeed not only help solve important everyday problems, but can be learned quickly, usually through carefully designed GUIss. In short, these forces broaden the range of problems to which COTS can be applied, and simplify COTS interfaces, in just the way ESSCOTS systems need.

Finally, as the number of COTS has grown, there is increased pressure from the marketplace for standardization and interoperability. Consumers recognize, for example, that a document processor which can accept input directly from a spreadsheet or data-management system is much more useful than one which cannot. Responding to this problem, more and more commercial products are designed as open or interoperable systems that permit or encourage communication of different software components. Again, then, ESSCOTS systems which require mechanisms for tailoring and building on top of existing COTS should benefit from these broad forces that are already pushing COTS in exactly this direction. The features that ESSCOTS systems would demand of COTS are just those that the commercial marketplace is already demanding.

What Is Left for Educational Software Developers to Do?

Commercial software development is a huge industry. Although it is difficult to estimate, corporations in this country invest at least $20 billion per year in developing new COTS. By contrast, even adding together development costs of for-profit educational software firms to dollars spent on educational software research, we still spend less than $200 million per year developing educational software (Perelman, 1992; OTA, 1988). If we can use COTS as the basis for educational software as the ESSCOTS approach suggests, the fact that roughly two orders of magnitude more are spent on COTS than educational software leads to several obvious questions: How can we compete with the giants of the commercial software industry? What, if any, is the role of educational software development? Where is our "value added"? By providing some formative answers to these questions, we can elaborate what it means to pursue a ESSCOTS approach to educational software development.

First, it is important to acknowledge immediately that high-quality educational software can, and will, continue to be developed "from scratch". The fact that COTS are increasingly appealing kernels for ESSCOTS systems does not mean that they will be the only way to develop new educational systems. This is simply the approach we are now taking, and we are as interested in understanding the limits of the ESSCOTS approach as its strengths.

Second, assuming we do adopt an ESSCOTS approach and let commercial giants develop much of the kernel educational software, we believe there are still many roles for researchers in educational software development to play. To modify Newton's metaphor: We can do more by standing on these growing giant shoulders than by trying to compete with them. Education researchers' "value added" will probably fall into several classes of activities. Below, we briefly discuss several activities we are pursuing.

Developing more effective ESSes: powerful passive tools and smart agents. Our experience piloting the ARC/INFO ESSCOTS system already indicates the cost-savings of the ESSCOTS approach is substantial. We plan to exploit this savings in several ways. Some of our savings will be invested in developing new powerful educational tools and "smart agents" to more effectively support inquiry learning (McArthur, Lewis & Bishay, 1993). As we discussed, our analysis of students' inquiries in the ARC/INFO ESSCOTS system has uncovered general cognitive problems with this style of knowledge acquisition—problems that persist across all our microworlds. In addition, we also noted that while the authentic nature of inquiry using ESSCOTS systems has advantages, it places additional new cognitive demands on students. Manufacturers of COTS are not directly interested in the special cognitive needs of students; their designs are shaped by the business and professional requirements of their customers. Accordingly, an important role for the educational research community will be to develop and test general tools and software agents to meet these cognitive difficulties.

Scaling up microworlds and educational software: Evaluation and implementation. Another important way to invest savings is to tackle the issue of scaling up more aggressively. We believe that virtually all past efforts to develop computer-based systems for learning through inquiry have two fundamental shortcomings that have nothing to do with the quality of software. First, there has been no concerted evaluation effort to really measure what students learn using these tools. Second, and related, there has been little implementation work to move these environments out of a lab and into classrooms at large. If we can delegate some of the software development to COTS, then tackling both the evaluation and implementation problems are critical next steps for educational technology research.

Developing principles of ESSCOTS software. Delegating much software development to industry will afford education researchers more time to develop effective ESSes; but perhaps a more important goal will be to develop general principles for choosing COTS and developing ESSes that turn COTS into ESSCOTS systems. These principles can represent an enduring research contribution, probably outliving any specific ESSCOTS system we might develop. Previously we discussed some principles for selecting COTS we have begun to formulate. Here we equally briefly mention some principles for designing ESSes that have arisen as we developed our ARC/INFO ESSCOTS software.

• Provide incremental increases in issue complexity. In ESSCOTS rich with authentic data like ARC/INFO there is a real danger that students will be overwhelmed with possible issues and patterns to pursue—real search spaces are often large ones. In general, ESSCOTS systems will need mechanisms that manage such complexity, ideally adding complexity just as the student is able to handle it. Our ARC/INFO ESSCOTS system achieved incremental increases in complexity two ways: (i) the mentor began by steering the students toward relatively simple issues, later relinquishing this control; and (ii) the set of variables available for the students to explore was initially small and later increased. These are just two elementary ways ESSCOTS software can slowly increase the size of students' search spaces. Future research should consider additional mechanisms, possibly including intelligent agents that monitor students' progress.
• Provide incremental increases in tool complexity. Students using ESSCOTS systems effectively have two spaces to search: the space of issues or variables to investigate, and the space of tools with which to uncover patterns in variables. Just as students often need to begin with simple issues, the complexity of available tools should also increase incrementally. In the ARC/INFO ESSCOTS system this was accomplished through the Custom Menu, which, as we noted above, gave more advanced students access to tools like "3 variable graphs". In this case, the availability of new tools was primarily under students' control. Future research can consider mechanisms that are also under mentor control, or under software control.
• Provide on-demand information about issues and data. Early in our piloting we were aware that some of our students knew little about the variables represented in the ARC/INFO databases. Rather than tutor them about the variables beforehand, we have adopted an "on-demand" philosophy towards learning this "prerequisite" knowledge (Fischer, 1991). That is, we provide students with tools so they can learn about variables when they need to—when it is necessary to choose an issue or make sense of results. In the ARC/INFO ESSCOTS, system we permitted on-demand learning about variables simply by supplying a "Definitions" menu option (see Figure 1). Of course, many more extensive mechanisms are possible. For example, it is easy to imagine built-in libraries of issues that are organized to permit students to inspect them.
• Provide on-demand information about tools. We noted above that our students had little prior knowledge of some analytic tools provided by the ARC/INFO ESSCOTS system. They have already shown us that such novel tools can be learned without formal teaching, by exploiting students' understanding of the variables and relationships represented by the tools. However, additional on-demand learning mechanisms might deepen their comprehension. Tools could be made inspectable, just as library issues. For example, at the student's request, the ESSCOTS system could bring up examples of scatter plots, with hypercards annotating various parts of the plot and interpretations of the data.

In general these principles recognize that COTS impose challenging decision-making tasks on learners and they suggest ways to take advantage of the positive aspects of this richness while minimizing the negative features. The principles may be incomplete or even incorrect in detail, and clearly more principles deserve consideration. We will continue to elaborate these and other principles—and test their generality—as we implement additional ESSCOTS systems. The point we wish to make here is that the development of such principles is a feasible and important research activity in an ESSCOTS approach to developing educational software. If educational researchers decide to stop developing their systems from scratch there is still a lot for them to do.

ESSCOTS as a New Model for Educational Technology Research

Viewed broadly, the ESSCOTS approach suggests a new model for educational technology research. Currently, much educational software is built by researchers from scratch. ESSCOTS systems suggest that some educational technology research might be better accomplished if research is built upon an increasingly rich foundation of off-the-shelf tools that have been developed for the workplace, by the workplace. We have argued that delegating much development to industry still leaves much work for education technology researchers. But the ESSCOTS approach demands a new division of labor.

In a traditional approach to educational software development, technology researchers have worked relatively independently of both commercial software vendors and classroom practitioners. At best, the interaction could be viewed as a "trickle down": Education researchers first develop cutting-edge ideas for applications, implement them in small-scale proof-of-concept demonstrations, and then publish primarily through conference proceedings and academic journals. For the most part, this is as far as the ideas go. Only on rare occasions would commercial vendors take these ideas, re-engineer the software into an "industrial strength" product, and thus make the ideas available to schools on a large scale.

In the ESSCOTS approach to educational software development, commercial developers, researchers, and educators would work together much more closely. On the one hand, researchers would borrow ideas and software from industry as much as they would contribute new software demonstrations. On the other hand, as we have suggested, the savings obtained by developing educational applications through COTS could be invested in more careful attention to evaluation and implementation—which means working more closely with teachers and schools as well as commercial vendors.

Clearly the details of these new relationships—what they are, what their benefits might be, and what problems might arise—remain unresolved at this time. In general, the division of labor implied by the ESSCOTS approach to educational technology research and development needs to be examined as a policy issue, addressing questions that explore the pros and cons of the approach: What kinds of technologies can we expect business to produce? How can we ensure they are affordable by a relatively poor educational market? What sorts of important ideas for educational software might not be developed through industry? What will the role of research be if most of the software is developed outside education? Should educational technology researchers help guide industry software development? Or provide the cutting-edge demonstrations that industry develops? How can we ensure that the software industry listens to these demonstrations? As the available industry software grows in volume and quality, we believe the ESSCOTS approach will be more and more cost-effective, and that these questions will become more and more pressing.

How the ESSCOTS Approach Impacts Other Important Issues

When we first began to build the ARC/INFO system we thought that the ESSCOTS approach would be a good way to develop high-quality educational software more cost-effectively. The ARC/INFO ESSCOTS system shows that we can indeed develop educational software much more rapidly when we build it on a commercial base. However, in discussing the ARC/INFO ESSCOTS system with other researchers several new reasons for the ESSCOTS approach have become apparent. In this final section we briefly list a few of the additional possibilities that ESSCOTS systems afford and important software development issues with which they intersect.

ESSCOTS systems can suggest project-based curricula and curriculum reform ideas. ESSCOTS represent a potentially powerful vehicle for designing new classroom structures and curricula. In particular, if ESSCOTS systems are the hub of many classes, curricula will naturally be project-based. Professionals use kernel COTS in projects as part of their day-to-day practice; it makes sense that authentic educational use of such COTS in an ESSCOTS would also involve projects.

ESSCOTS systems can stimulate connections between the classroom and work. Classrooms that use ESSCOTS systems naturally will become linked (either through face-to-face discussions or electronically, through networks like NREN) with professionals that are using the COTS in their daily practice. Useful interactions may be two-way. Students will certainly have questions about tools and data that their teachers will not be able to answer but which experts can address. Less obviously, students may come up with interesting ideas and conjectures that experts could refine and pursue. In short, ESSCOTS systems will naturally extend education and learning out of the schools physically and conceptually. Students will be both more likely to work on authentic projects and interact with real practitioners.

ESSCOTS systems can profitably couple science, business, and education. There is increasing demand for business to become involved in education. If we can show that many products developed in business, science, and entertainment can be used in effective ESSCOTS software, we may open up a major new market for software (and hardware) development firms. Until now, education has not been an important target for these companies. In turn, education has missed out on a vast collection of "power tools". It will continue to miss out until linkages like ESSCOTS systems change the balance of supply and demand. We already have anecdotal evidence that our ARC/INFO ESSCOTS system can shift this balance. Through this ESSCOTS system, ESRI—the developer of ARC/INFO— are now interested in marketing a version of their software for education. They also have generously provided us with copies of ARC/INFO and even hardware on which to run our ESSCOTS software. We see this hopefully as the first of many examples of increased corporate involvement in education stimulated by ESSCOTS systems.

ESSCOTS systems can help solve upward compatibility problems. There are many technical problems involved in developing software—educational or otherwise—from scratch. One is that the language in which you develop the system, or the interface tools you use, can change unexpectedly. In theory, this is often desirable, because it means the platform has been upgraded. But in practice, it can be disastrous because your system may not work under the upgraded software. Building educational systems as ESSCOTS can isolate you from this problem to a considerable extent. Developers of COTS are responsible to a large and powerful user community, and therefore usually provide upgrades that are carefully tested and upward compatible. In effect, by developing ESSCOTS systems educational software developers can enjoy the high-quality of service usually available only to high-paying customers.

ESSCOTS systems can help solve the problem of heterogeneous platforms and portability. Schools, more than most workplaces, will always be stuck with a mixed collection of computer systems—often different batches of "hand-me-downs" supplied by businesses that are upgrading. This makes it very difficult to provide the same software environment to every student in a class, because many educational programs—especially ones coming directly out of research—only run on one type of computer system. ESSCOTS systems may be able to overcome this portability problem because commercial vendors often try to enlarge their market share by providing versions of their COTS for each major hardware platform. For example, variants of ARC/INFO run on Sun workstations, Macintoshes, Silicon Graphics workstations, among others. Thus, with no effort on our part, the ARC/INFO ESSCOTS system is already highly portable.

ESSCOTS systems can exploit new trends towards interoperability. Interoperability is now a major concern in developing commercial and military software. Interoperable systems adhere to standards (e.g., a standard protocol for defining software objects and data) that would permit independent applications to work together in a large software environment. In the future more and more COTS will be interoperable, and this will allow us to easily develop interoperable ESSCOTS software. Currently, almost all educational software systems are "stand alone". However, we envision many exciting educational possibilities for interoperable systems. For example, our current ARC/INFO ESSCOTS system could be greatly enhanced if we could combine it with an existing package of visualization and statistical analysis tools. In this way we could have cut our development time to even less than 60 hours, and at the same time increased functionality. In general, if ESSCOTS systems can be built from interoperable COTS we envision that high-quality educational environments may be constructed simply by "pasting" together judiciously selected commercial systems.

ESSCOTS systems can help learners of all ages who need to master complex software systems. Our current ESSCOTS system helps high-school students learn new skills using complex commercial software. Part of the role of the ESS is to simplify the use of the COTS itself, so students can focus on learning and problem-solving with the COTS, not learning about it. Novices of all ages who confront complex software systems often need similar help reducing the slope of the learning curve. In business, for example, financial planning and document processing tools change every few months. In the military, training problems arise for similar reasons— sophisticated modeling and simulation systems are becoming integral parts of many jobs, and new personnel move into those jobs as fast as the software tools change. In all these cases powerful software supports like ESSes can play an important role in improving and accelerating the training process. We recognize that the learning needs of high-school students and military recruits differ, as do the software tools they must master. However, we believe that many of the principles for designing ESSes we are now developing should generalize across different learners, and across different learning or training goals.

ESSCOTS system can provide important dual use applications of technologies. Recent reductions in military spending and the Technology Reinvestment Project have placed an increasing emphasis on dual use technologies— technologies that were originally designed for military or commercial uses but which can be applied to other areas. ESSCOTS systems seem like an ideal way to develop dual use technologies by extending existing software to new educational applications. In this connection we have considered the potential educational applications of military simulations as well as commercial database systems like ARC/INFO. Certainly many such simulations could be the basis for ESSCOTS systems used in military training. But we also believe that some simulations, especially those not wedded to warfare applications, could be used in K12 ESSCOTS systems that help students learn valuable planning skills.

ESSCOTS systems represent a way to exploit increasingly powerful commercial software systems for purposes of learning and education. We have discussed ESSCOTS software in the classroom, but of course it is easy to imagine how they could be used wherever learning happens—at home, in libraries, or on the job. Learning is no longer something we do from 9 to 5, or from K to 12; in the information age it is becoming a life-long necessity. Much of the learning we do will involve software tools that help us do our job, learn for fun, or entertain ourselves. In the future, mastery of such systems will be essential for success in virtually every phase of our lives, and in each case novices will need help in learning the system. On the one hand they will need support that reduces the complexity inherent in useful software systems; on the other hand they will need support that adds documentation, examples, and other "training wheels". ESSCOTS systems represent one approach to developing such supporting environments and easing the acquisition of important tools for working, learning, and problem solving.

References

• Anderson, J., and Pelletier, R. (1991). A development system for model-tracing tutors. In Proceedings of the International Conference of the Learning Sciences, 1–8.
• Brown, J.S., and VanLehn, K., Repair Theory: A Generative Theory of Bugs in Procedural Skills. Cognitive Science, Vol. 4, pp. 379–426, 1980.
• Cohen, D.K. (1988). Educational technology and school organization. In R.S. Nickerson & P.P. Zodhaites (Eds.), Technology in Education: Looking Toward 2020 (pp. 231–264). Hillsdale, NJ: Erlbaum.
• Conklin, J., and Begeman, M.L. (1988). gIBIS: A hypertext tool for exploratory policy discussions. In Proceedings of the ACM CSCWí88 Conference on Computer-Supported Cooperative Work, p. 140–152.
• Cuban, L. (1986). Teachers and machines. New York: Teachers College Press.
• Fischer, G. (1991). Supporting Learning on Demand with Design Environments. Proceedings of the International Conference on the Learning Sciences, Evanston, pp. 165–171.
• Forbus, K. (1991). Towards tutor compilers: Self-explanatory simulations as an enabling technology. In Proceedings of the International Conference on the Learning Sciences, Evanston, IL.
• Lewis, M., Bishay, M., and McArthur, D. (1993, August). The macrostructure and microstructure of inquiry activities: evidence from students using a microworld for mathematical discovery. Proceedings of the World Conference on AI and Education, Edinburgh.
• Lewis, M., Bishay, M., and McArthur, D. (under review). Supporting discovery learning in mathematics: Design and analysis of an exploration environment and inquiry activities. Instructional Science.
• Lewis, M., McArthur, D., Bishay, M., and Chou, J. (1992). Object-oriented microworlds for learning mathematics through inquiry: Preliminary results and directions. Proceedings of the East-West Conference on Emerging Computer Technologies in Education, Moscow, April.
• McArthur, D. and Lewis, M. (1991a). Object-oriented microworlds for learning mathematics through inquiry. N-3242-NSF, RAND Corporation, Santa Monica, CA.
• McArthur, D. and Lewis, M. (1991b). Overview of object-oriented microworlds for learning mathematics through inquiry. Proceedings of the International Conference on the Learning Sciences, Chicago, August.
• McArthur, D., Lewis, M, and Bishay, M. (1993). The roles of artificial intelligence in education: Current progress and future prospects. RAND DRU-472-NSF.
• McArthur, D., Robyn, A., Lewis, M. and Bishay, M. (1992). Designing new curricula for mathematics: A case-study of computer-based statistics in high school. RAND WD-5930-ED.
• Morgan, G. (1993). Risk analysis and management. Scientific American, July, pp. 32–41.
• OTA (1988). Power On! New Tools for Teaching and Learning. Washington DC: Office of Technology Assessment.
• Perelman, L. (1992). School's Out. New York: William Morrow.
• Richmond, B. (1993). Systems thinking: Critical thinking skills for the 1990s and beyond. Systems Dynamics Review, 9(2), 113–133.
• Shute, V., and Glaser, R. (1990). Large-scale evaluation of an intelligent discovery world: SMITHTOWN. Interactive Learning Environments, 1, 51–77.
• Sleeman, D.H. (1982). Assessing aspects of competence in algebra. In D. H. Sleeman & J. S. Brown (Eds.), Intelligent Tutoring Systems, 185–200. New York: Academic Press.
• Sleeman, D.H. (1987). PIXIE: A shell for developing intelligent tutoring systems. In Lawler, R, & Yazdani, M (Eds.) (1987). Artificial Intelligence and Education: Learning Environments and Intelligent Tutoring Systems. Norwood NJ: Ablex.
• van Berkum, J. J., and de Jong, T. (1991). Instructional environments for simulations. Education & Computing, 305–358.

Notes

• [1] We used the eastern states even though our students were more familiar with west-coast geography because the average county size is much smaller in the east, providing a finer grain-size of patterns. Data for the entire country was available to students through the custom menu.
• [2] Much of the visual impact and information of the color map is lost in a black-and-white paper. To facilitate discussion we have reformatted the maps so that light shades represent high variable values and dark shades represent low variable values. We will also interpret hotspot maps for the reader where necessary.
• [3] Of course, few if any schools provide these kind of opportunities. Our students are hardly unique in this respect.

Acknowledgments

This research was funded by the National Science Foundation (Applications of Advanced Technologies Program) under grant MDR-9055573. The views or conclusions expressed in this paper do not necessarily represent the policies or opinions of the sponsor.