By: Mary Anne Rizzolo
We learned a great deal about how to design simulations that can be used for assessment and we developed guidelines for ensuring that they are implemented as consistently as possible, but the evaluation phase was the most challenging aspect of the project. While I had expected that the evaluation of students was going to be difficult, I did not anticipate HOW hard it would be! Compared to other types of assessment and evaluation, clinical evaluation is, by it very nature, subjective. Achieving some measure of objectivity, reliability, and validity in the process of clinical evaluation presents a real challenge.
Nurse educators are placed in a tough situation. They need to nurture students along their learning path, then evaluate them a few days/weeks later. As changing hats is very hard to do, faculty subscribe to a wide variety of philosophies about assessment and evaluation. And, unlike most other teachers in higher education, nurse educators have an additional burden to bear; they have a duty and an obligation to protect the public from a practitioner who is unsafe.
Some find it easier to take on that role than others. Yet, we expect every nursing instructor to do it, even adjuncts who have had little, if any, training on how to approach this very difficult task. Keep an eye on the outcomes of the extension of the original study conducted by our colleagues in Minnesota. In an effort to improve intra- and inter- rated reliability, they have explored the implementation of an additional training intervention, having faculty complete the StrengthsFinder tool to see if certain personality characteristics have an influence. I can’t wait to see their findings!
So, should we start implementing summative assessments using simulation?
While there are many more things we need to learn about how to use simulation for summative clinical assessments, is our current practice – relying on one person’s opinion of student competency performed in the chaotic health care setting with “real” patients – a valid and reliable method? At the very minimum, simulation provides a controlled environment, so I believe it is both feasible and important to start now – AS LONG AS YOU FOLLOW THE BEST PRACTICES THAT HAVE BEEN IDENTIFIED WITH RECENT STUDIES!
None of our assessment methods are perfect, so why not add simulation as another assessment tool to provide a more complete picture of your students and how they are progressing? The NLN’s Fair Testing Guidelines for Nursing Education urges faculty to use multi-modal assessments. But don’t jump in without a well-thought-out plan. Make sure everyone is on board and committed to move forward and that you have the time and resources to invest in exploring tools, norming them to reflect the behaviors you are evaluating, then training those who will function as raters. Plan to spend considerable time talking with your faculty colleagues to illuminate everyone’s vision of what the student “looks like/behaves like” at the end of a course or program. Get a commitment from faculty to support the project even if they disagree with some aspects. It will need to be revisited and revised each year, incorporating new findings from the literature.
Make it an adventure! Perhaps you could invite another school to go on this journey with you and share the work involved in planning your exploration. Exchange videos of student performances in simulation to remove some of the biases inherent in evaluation. Become a leader and make it a research study and advance our knowledge of how to do this well. Get funding to pay statisticians and others for some of the tedious work that needs to be done. Below are resources to help you begin the journey. Enjoy the ride!
RESOURCES
Adamson, K., Kardong-Edgren, S., & Willhaus, J. (2013). An updated review of published simulation evaluation instruments. Clinical Simulation in Nursing, 9, e393-e400.
Kardong-Edgren, S., Adamson, K.A., & Fitzgerald, C. (2010). A review of currently published evaluation instruments for human patient simulation. Clinical Simulation in Nursing, 6, e25-e35.
Kardong-Edgren, S., Oermann, M.H., Rizzolo, M.A., & Odom-Maryon, T. (2017). Establishing inter- and intrarater reliability for high-stakes testing using simulation, Nursing Education Perspectives, 38(2), 63-68. doi:10.1097/01.NEP.0000000000000114
McIvor, W.R., Banerjee, A., Boulet, J.R., Bekhuis, T., Tseytlin, E., Torsher, L. . . Weinger, M.B. (2017). A taxonomy of delivery and documentation deviations during delivery of high fidelity simulations. Simulation in Healthcare, 12(1), 1-8.
Oermann, M., & Gaberson, K. (2017). Evaluation and testing in nursing education (5th ed.). New York, NY: Springer Publishing.
Oermann, M.H., Kardong-Edgren, S., & Rizzolo, M.A. (2016). Summative simulated-based assessment in nursing programs, Journal of Nursing Education, 55(6), 323-328. doi: 10.3928/01484834-20160516-04.
Oermann, M.H., Kardong-Edgren, S., & Rizzolo, M.A. (2016). Towards an evidence-based methodology for high-stakes evaluation. Teaching and Learning in Nursing, 11(4), 133-137. doi:10.1016/j.teln.2016.04.001
Rizzolo, M.A., Kardong-Edgren, S., Oermann, M.H., & Jeffries, P.R. (2015). The National League for Nursing Project to explore the use of simulation for high-stakes assessment: Process, outcomes, and recommendation. Nursing Education Perspectives, 36(5), 299-303. doi:10.5480/15-1639
Rutherford-Hemming, T., Kardong-Edgren, S., Gore, T., Ravert, P., & Rizzolo, M.A. (2014). High-stakes evaluation: Five years later. Clinical Simulation in Nursing, 10, e605-610.
Willhaus, P., Burleson, G., Palaganas, J., & Jeffries, P. (2014). Authoring simulations for high-stakes student evaluation. Clinical Simulation in Nursing, 10, e177-e182.
Wolf, L., Dion, K., Lamoureaux, E., Kenny, C., Curnin, M., Hogan, M.A….Cunningham, H. (2011). Using simulated clinical scenarios to evaluate student performance. Nurse Educator, 36(3), 128-134.
Mary Anne Rizzolo, EdD, RN, ANEF, FAAN
Dr. Rizzolo’s career has focused on exploring new technologies, determining how they can inform and educate nurses, operationalizing their cost-effective delivery, and disseminating their value for education and practice. She pioneered the development of screen-based simulations that won national and international awards, and created one of the first websites in the world offering continuing education, journal articles, and networking opportunities for nurses. She has delivered over 250 national & international presentations, has served on many national committees and advisory boards. Dr. Rizzolo just completed her term on the Society for Simulation in Healthcare’s Board of Directors and recently chaired the committee that developed the SSH Academy.
During her tenure at the National League for Nursing, Dr. Rizzolo was the staff liaison for the original simulation research project with Pamela Jeffries, the SIRC development project, the ACES unfolding cases, vSim projects, and the project to explore the use of simulation for high stakes assessment and more. She is the founder of the Leadership Development Program for Simulation Educators and continues to consult on several simulation projects.