NSG/516ED Week 6: Evaluation Plan, sample paper

Evaluation Plan: Measuring a Teach-Back and Plain-Language Insulin Pen Program at Four Levels, From Staff Reaction to Patient Outcomes

[Student Name]

University of Phoenix

NSG/516ED: Practicum I (education track)

Week 6 Assignment

[Instructor Name]

[Date]

The practicum site, staff and plan are a composite written for a model paper. Practicum hours and logs are recorded separately and are not part of this paper.

The practicum project changes how nurse educators at the diabetes education center teach insulin pen use, with the goal of fewer dosing errors and less early hypoglycemia among older adults and patients who need language support. Earlier weeks assessed the situation, the population and the gap, planned the intervention and planned its implementation. This paper plans the evaluation. A staff program that everyone enjoyed but that did not change what happens at home after the first insulin dose would not be a success, and the evaluation has to be able to tell the difference.

The Evaluation Model

Kirkpatrick and Kirkpatrick (2016) describe four levels for evaluating training: reaction, whether participants find it favorable, engaging and relevant; learning, the degree to which they acquire the intended knowledge, skills and confidence; behavior, the degree to which they apply what they learned on the job; and results, the degree to which targeted outcomes occur. The model fits this project because the intervention works through a chain: staff learn a method, apply it in visits and patients benefit. Measuring each link shows where the chain holds or breaks. Nurse educator texts recommend evaluation that addresses both the process and the outcomes of teaching (Bastable, 2019).

Level 1: Reaction

Measure: a five-item anonymous survey after each staff session, rating relevance, usefulness of role-play, confidence in using teach-back, clarity of the materials and willingness to use the approach, on a five-point scale, with one open comment.

Target: a mean of 4 or higher on relevance and willingness.

Use: comments will guide adjustments to the second session and to coaching.

Level 2: Learning

Measures: a role-play skills checklist scored during the second staff session, covering teach-back on three safety concepts and coaching an eight-step return demonstration with a planted error; and a brief case-based quiz on selecting the correct guide, language version and aids for three scenarios.

Target: every educator passes the checklist, with coaching and repeat practice for anyone who does not, and scores all three quiz scenarios correctly.

Level 3: Behavior

Measures: fidelity observations using the same checklist as the Week 3 gap analysis, at two visits per educator during the two coaching weeks and one per educator per month for three months; and the documentation rate from the new template fields for return demonstration and teach-back, pulled monthly from the electronic record for all insulin start visits.

Baseline: return demonstration of all eight steps in 20% of observed visits and teach-back of safety concepts in 13%.

Target: at least 80% of observed visits meet all fidelity elements, and at least 90% of insulin start visits document both fields by month three.

Level 4: Results

Primary outcome: the percentage of patients starting insulin pens who need an unscheduled call or visit for dosing or technique problems within 30 days. Baseline: 38% (24 of 64) over the prior six months. Target: 15% or lower.

Secondary outcomes: independent return demonstration of all eight steps at the first visit, recorded by the educator; correct teach-back of hypoglycemia treatment at the 72-hour follow-up call; and emergency department visits for hypoglycemia within 60 days, reviewed with the hospital's quality department.

Subgroups: all results will be reported separately for patients 65 or older and for patients whose preferred language is not English, since these groups were the reason for the project.

Balancing Measure

Visit length will be tracked from scheduling data to see whether the new approach lengthens visits beyond the scheduled 45 minutes, which could reduce access. Staff will also be asked at monthly huddles whether the method is sustainable.

Data Collection and Timing

Reaction and learning data are collected at the staff sessions. Behavior data are collected during coaching and monthly for three months. Results data are collected for all patients starting insulin during the six months after implementation and compared with the six months before. De-identified data will be stored on the center's secure drive under the manager's supervision.

Analysis

The center starts about ten patients on insulin pens each month, so the six-month comparison will include roughly 60 patients on each side. Proportions will be reported with denominators, and the primary outcome's before and after proportions will be tested with chi-square or, if expected counts are small in the subgroups, Fisher's exact test, with significance set at .05. Because other events in the same months could also move the numbers, monthly results will also be plotted on a run chart with the implementation date marked, and the team will look for a sustained shift rather than a single good month (Provost & Murray, 2011). Emergency visits for hypoglycemia are expected to be few, so each one will get an individual case review instead of a statistical test.

Ethical Considerations in the Evaluation

The evaluation uses information collected in routine care, and it must protect patients and staff. Only aggregate, de-identified data leave the center's records, and small subgroup counts will be combined or suppressed in reports when a number is small enough to identify an individual. Fidelity observations are framed as coaching, and individual educators' results are shared only with that educator and the champion, not with the manager, so that the evaluation supports learning rather than performance discipline. Patients are told that the center is checking its teaching methods, and no patient is asked to do anything beyond usual care. Because the project is quality improvement, it went through the hospital's quality project screening, which confirmed that research ethics review was not required.

Reporting and Sustaining

Findings will be reported to the educators and manager at a staff meeting three months after implementation and in a written summary at six months, with recommendations on whether to continue, modify or spread the program to the hospital's inpatient insulin teaching. The champion educator will continue quarterly fidelity checks after the practicum ends, and the documentation fields will allow the manager to monitor the program without additional data collection.

Conclusion

The evaluation plan measures the project at every link in its chain: whether educators valued the training, whether they learned the skills, whether they used them with patients and whether patients made fewer errors and needed less unscheduled help. Specifying measures, baselines, targets and analysis in advance, reporting results for the groups the project was designed to serve and tracking a balancing measure will allow the center to judge honestly whether the program worked and whether it should continue.

References

Bastable, S. B. (2019). Nurse as educator: Principles of teaching and learning for nursing practice (5th ed.). Jones & Bartlett Learning.

Kirkpatrick, J. D., & Kirkpatrick, W. K. (2016). Kirkpatrick's four levels of training evaluation. ATD Press.

Provost, L. P., & Murray, S. K. (2011). The health care data guide: Learning from data for improvement. Jossey-Bass.