Project Background

Emergency Preparedness Exercise Evaluation Tool

Use of the Tool

We hope that public health and healthcare agencies find both the evaluation instrument and this exercise evaluation database helpful in developing appropriate tools for optimal evaluation of their exercises.

Project Background

Exercises are essential tools that can help emergency planners improve their readiness for emergencies when used as part of the improvement cycle of planning, training, evaluating, and acting on priority issues. Well–designed and well-executed exercises allow agencies to examine capabilities, validate plans, and challenge assumptions in a controlled setting. In order to be maximally useful as part of the improvement cycle, evaluation of the participants’ performance during an exercise must also be well-structured and well-executed.

Because of a lack of access to validated measures of performance and many other logistical constraints, public health and health agencies often find it daunting to identify and/or create appropriately specific, valid, and reliable measures for their exercises. In order to help address this challenge, a group of researchers at the Harvard School of Public Health’s CDC-sponsored Preparedness and Emergency Response Center, called LAMPS (Linking Assessment and Measurement to Performance in PHEP Systems), have created and tested an evaluation instrument that allows evaluators to best capture data during an exercise. In a related project, we have also tested the validity and reliability of numerous measures of performance during emergency preparedness exercises, and we have placed these measures into a searchable database that is available to exercise planners, evaluators, and others involved in health emergency preparedness.

Searchable Database of Evaluation Measures

The web database accessible here is comprised of numerous evaluation measures that have been selected, written, tested, and validated by the HSPH-LAMPS program for use in health tabletop and functional emergency preparedness exercises. Users can identify relevant evaluation measures by searching this database, for Centers for Disease Control and Prevention (CDC) public health preparedness capabilities and the Assistant Secretary for Preparedness and Response (ASPR) healthcare preparedness capabilities, and by naming the Incident Response Stage (i.e. notification, mobilization, etc.) and/or by Keyword. The results from the search can be edited and tailored by the user to combine or delete individual measures to fit the needs of the exercise. After the user has identified appropriate measures that he or she wishes to use for an evaluation, we have also designed the database to help the user create a comprehensive evaluation tool for evaluators, based on the objectives of the exercise being conducted. Thus, the end result will be an exercise evaluation tool designed for each specific exercise that includes both quantitative and qualitative measures of performance.

We have assessed the reliability and validity of the evaluation measures in this database using various methodologies. Reliability was assessed by statistical measure of inter-rater agreement and correlation, external evaluators’, and content expert interviews. Discriminative ability of the measures and ability to detect expected changes in performance of entities over time have been used to assess the validity of the measures.

ACKNOWLEDGEMENTS

The Emergency Preparedness Exercise Evaluation Tool was developed by a group of researchers at the Harvard School of Public Health’s CDC<sponsored Preparedness and Emergency Response Research Center HSPH-PERRC), called LAMPS Linking Assessment and Measurement to Performance in PHEP Systems). We gratefully acknowledge funding support for this project from the US Centers for Disease Control and Prevention CDC) grant number 5PO1TP000307-04. The content of this database are solely those of the authors and do not necessarily represent the views of any partner organizations, the CDC or the US Department of Health and Human Services nor does mention of trade names, commercial practices, or organizations imply endorsement by the US Government.