Intellectual Merit

NSF solicitations include an expectation that each proposed project or centre will incorporate a mechanism to assess success. In the Advanced Technological Education (ATE) program, typically that expectation is fulfilled by annual summative evaluation reporting. The core principle of the Formative Assessment Systems for Advanced Technological Education (FAS4ATE) worskshop project, based on the stated needs of Principal Investigators (PIs) and evaluators involved in the ATE program, was that the assessment of success annually, or at the completion of a grant cycle, is not sufficient. Ongoing access to valid, reliable, useful, and accessible, real-time evaluative data for decision making is needed to ensure that ATE funded efforts achieve the best possible results with their funds. However, those same ATE PIs and evaluators reported that limits on time, expertise, and resources have meant that this need often remains unmet. FAS4ATE workshop project, led by The Evaluation Center at Western Michigan University and The Centre for Program Evaluation at University of Melbourne, Australia, addressed these needs by creating an international, multi-disciplinary space where experts on formative assessment and evaluation interacted with ATE PIs and evaluators to bring best practices in evaluation into the ATE program. 

Project Summary

FAS4ATE was designed as the first phase (Figure 1) of a process to address this need for real time data, by learning about the typical program designs and formative data needs in projects. In the long term (phase 2), this knowledge will enable the FAS4ATE project leaders to design a system and set of tools for formative assessment and evaluation that are informed by real needs in the ATE program and built to specifications that will make them usable for ongoing decision making and summative reporting.

The main goals of FAS4ATE were to:

1. 1.       Engage ATE PIs and evaluators with experts to create evaluation logic models that facilitate the identification and prioritization of data that projects and centers must have access to in order learn, and thereby improve, their practice and reporting.
2. 2.       Engage ATE PIs and evaluators with experts in assessment, evaluation, and IT to create specifications for the systems and tools that would be necessary to support the gathering, analysis and dissemination of that information.
3. 3.       Provide an international, multi-disciplinary space where experts will interact with ATE PIs and evaluators to bring best practices in IT and evaluation into the ATE program.

To achieve those goals, our overall strategy for the project promoted engagement by ensuring that each step had benefits for participants and the project (Figure 2). We conducted two 1.5 hour pre-workshop preparatory webinars (with 85 and 52 participants, respectively) and a one day workshop (with 28 participants from more than 20 ATE projects and centers). Based on feedback from the workshop participants, we followed up with a series of five blogs discussing ideas raised in the workshops posted on the EvaluATE website.

Specific Contributions to Intellectual Merit and Broader Impact

Through the main project activities, participants delineated the structure of their ATE funded activities in various streams (professional development, materials development, etc.) via logic models, and then mapped evaluation questions to those logic models to determine data needs and timing. They then created high level archetype logic models for their activity streams, identified and prioritized data needs, and provided a set of specifications for data tools, systems and templates needed to streamline this evaluation work in their projects and centers.

Based on survey feedback from 100% of the workshop participants (N=28), nearly two-thirds were able to apply what they learned (Figure3) through FAS4ATE to improve their

• delineation of evaluation questions
• use of evaluation resources
• evaluation design
• scheduling of data collection, and
• evaluation reporting

In addition, participants reported no negative effects from applying the learnings in their workplaces, and many participants noted a positive effect (Figure 4).

These results were indicative of the project’s success not only in increasing knowledge but in motivating the participants to use their new knowledge base in practical real-life application, and in reported benefits to their projects and centres.

Through collaboration and engagement with ATE PIs and evaluators, FAS4ATE was able to document evaluation ideas, designs, and acquire specifications for systems that could enable all ATE projects and centers to make the most effective use of grant monies for the broadest impact.

Products

The project produced many resources for the ATE stakeholders which were made available to the public online as highlighted in Table 1.

Last Modified: 11/23/2016
Modified by: Arlen R Gullickson