There are several convergent challenges in facing the wastewater treatment workforce.  First, as the baby boom generation begins to retire, the sector anticipates the need to replace as much as 80% of its workforce over five years.  However, relevant training programs in technical colleges have little capacity to handle the necessary surge in enrollment.  Second, most hands-on training takes place in active wastewater treatment plants, so the opportunity to make errors and learn from them as part of the training process is necessarily limited.  When errors are made, they have the potential to introduce substantial environmental harms to water resources.  Third, most assessments used in wastewater treatment training programs and operator licensure examinations are multiple choice tests that measure conceptual knowledge rather than the successful acquisition of operator skills.

This project addressed all three challenges by developing a web-based training simulation that assesses operator trainees? decision making and performance as they operate a virtual biological wastewater treatment plant.  Using a cognitive task analysis of multiple expert operators, the project distilled a set of decision rules that were incorporated into an intelligent tutoring system capable of comparing a trainee?s step-by-step actions and decision making within the simulated plant to expert performance.  Based on trainees? alignment with expert performance, the system provided feedback and guidance in real time.  Additionally, the simulated environment was designed to be fully open-ended, responding with real-world accuracy to the actions of users.  This design feature allowed trainees to make mistakes that yielded realistic consequences and then scored their abilities to appropriately resolve the problems they created.

To accomplish this, several technical challenges were overcome:  First, the cognitive task analyses successfully elicited both tacit and explicit knowledge from experts.  Tacit knowledge is that information that experts utilize without conscious awareness of doing so?their habits that have become so routine that they are able to execute them appropriately without deliberate effort or intention. Given that experts typically omit up to 70% of necessary information when they explain how they perform tasks in their domain of expertise, this was a critical step in ensuring that the assessment functions were able to evaluate learner performance against expert processes. 

Second, the graphic user interface and underlying simulation of the biological wastewater mechanisms were developed so that each set of controls in the simulated environment would both respond realistically to the actions of the user and provide the correct visual and data feedback reflecting realistic physical changes to the plant and its systems.

Third, we developed an automated assessment system that could evaluate users? step-by-step responses to situations that were not predetermined by the system in response to expert decision rules.  This system evaluates the virtual location of the user within the simulated plant and all relevant physical information tracked by the simulator, then utilizes the expert decision rules to determine the next optimal step.  The system then evaluates the user action in relation to this expert-derived step, and feedback can be given in real time.  This system stands in contrast to previously developed systems that can only evaluate user actions in relation to a predetermined sequence or scenario with hard-coded optimal steps that cannot adapt to the unpredicted consequences of users? previous errors.

Evaluation data indicate that as students become accustomed to using the system and learn more about how to conduct essential wastewater treatment operation tasks, the percentage of expert-aligned steps in their simulation performance increases as the overall number of steps taking increases.  This trend indicates that although these trainees still make mistakes that they need to correct in the simulation, their strategies for correcting mistakes are increasingly aligned with expert strategies.

The system developed under this project is designed specifically for the biological wastewater management domain.  However, the model and strategies developed represent a generalizable mechanism by which such dynamic adaptive simulation-based training and assessment systems can be developed and deployed.

Last Modified: 12/18/2018
Modified by: David F Feldon