DNP 751 Week 6 Evaluation, Measures and Analysis Plan Example

Three Kinds of Evidence From One Pilot: Clinical, Process and Implementation Outcomes, and How Each Will Be Measured and Analyzed in the Diabetes Distress Project

[Student Name]

University of Phoenix

DNP/751: DNP Applied Project II

Week 6 Assignment: Proposal Section 6 Draft

[Instructor Name]

[Date]

The health center, staff and figures are composites written for a model paper.

Why Three Kinds of Evidence

A screening pathway can fail for different reasons: because distress does not respond to the follow-up offered, because the pathway is not delivered as planned, or because staff find it unworkable. If the evaluation measured only distress scores, a disappointing result could not be explained, and a good result could not be credited to the pathway. This plan therefore gathers clinical outcomes for patients, process outcomes for the pathway, and implementation outcomes for the staff who deliver it.

Clinical Outcomes

Primary clinical outcome: how far distress shifts for each patient with a positive screen, defined as the difference between the first full-scale mean item score and the score repeated 12 weeks later, with the proportion moving out of the high range (3.0 or above) reported alongside. Source: the structured form in the electronic health record. The subscales will be analyzed the same way to see whether the subscale targeted by each patient's response improved most, which tests the logic of matching responses to appraisals set out in the framework section.

Exploratory clinical outcome: A1C at the last result before the first positive screen and the first result at least 12 weeks later, reported descriptively. Twelve weeks is too short for a reliable change, and the design cannot attribute change to the pathway, so no test will be run.

Process Outcomes

Screening rate: eligible visits with both screener items answered, over all eligible visits, reported weekly. Declines will be counted separately, and a second rate excluding declines will be shown. Timely follow-up: positive screens followed by a care manager contact documented within 14 days, over positive screens, reported weekly with a two-week lag. Balancing measure: minutes added to rooming, timed by the champion across 20 visits per cycle.

Implementation Outcomes

Proctor et al. (2011) proposed eight implementation outcomes that describe how well a change is put into practice, distinct from its clinical effect: how acceptable staff find it, whether they take it up, whether it fits the setting, whether it can be done, whether it is delivered as designed, what it costs, how far it reaches and whether it lasts. The project will measure four of them. Acceptability, appropriateness and feasibility will be measured among staff at the end of the second and third improvement cycles with the Acceptability of Intervention Measure, Intervention Appropriateness Measure and Feasibility of Intervention Measure. Weiner et al. (2017) developed and tested these three scales and found that four-item versions had high internal consistency, with alphas from .85 to .91, good structural validity and acceptable test-retest reliability. Fidelity will be measured as the proportion of the pathway's steps completed for each positive screen and by the champion's observations of five screening encounters per cycle. Cost will be tracked as staff hours and direct expenses.

Analysis of Process Data

Process outcomes will be read on run charts rather than tested with a single before and after statistic, because the question the team needs answered each week is whether the latest change moved the process, and a run chart answers it as the data arrive. Each chart's middle line will sit at the median of the opening weeks, as Perla et al. (2011) recommend, and the team will watch for four signals that change is real: at least six consecutive weeks landing on the same side of the median line; at least five consecutive weeks each moving past the week before in one direction; a count of runs outside the range expected by chance; and one week far out of line with the rest. Each signal will be checked against the cycle log to see which change preceded it.

Analysis of Distress Scores

Paired scores will be summarized with means, standard deviations and the average within-patient change bounded by a 95% confidence interval. A paired t test will be used if the differences are roughly normal; if they are clearly skewed, the Wilcoxon signed-rank test will be used instead. The proportion leaving the high range will be reported as a count and percentage. With an expected 50 to 60 paired scores, the analysis can estimate a moderate change with reasonable precision but not detect small differences between subgroups.

Subgroups and Equity

All process and clinical outcomes will be shown by preferred language, by age group and by insulin use, since the literature suggests distress is higher among people treated with insulin. Subgroups will be described, not tested, because numbers will be small. A gap of more than 10 percentage points in the screening rate between English and Spanish speakers will trigger a review of the Spanish workflow.

Linking Results to the Adoption Decision

The evaluation exists to support one decision: whether the health center should keep the pathway at this site and extend it to the other two. The team agreed on decision rules before any data were collected, so results cannot be read to fit a preference afterward. The pathway will be recommended for adoption if the screening rate holds at or above 80% for the last four weeks of the pilot, timely follow-up reaches 85%, rooming time rises by no more than two minutes, and mean staff scores on the three implementation measures reach at least 4 on their 5-point scales. If the process targets are met but distress does not improve, the pathway will be kept and the response guide revised. If the process targets are not met, the team will decide whether a further cycle could fix the problem or whether the pathway should stop.

Limits of the Evaluation

The design cannot separate the pathway's effect from other influences on distress, such as seasonal stress or changes in insurance coverage during the pilot. Regression toward the mean will tend to lower repeat scores among patients who first scored highest, which could make the pathway look more effective than it is. Staff ratings may be kinder at a site where the project lead is a colleague, which is why the surveys are anonymous. Every report will name these limits plainly.

Missing Data

A screen counts as complete only when both items are answered. Patients without a repeat full-scale score will be compared with those who have one on first scores, age and language, so any pattern in who was lost can be seen. No values will be imputed.

Data Management

Weekly reports will use study numbers, the linking key will stay with the quality coordinator, and all files will stay on the health center's secure drive. Staff surveys will be anonymous and collected on paper in a sealed box.

Reporting Plan

The medical director and quality committee will receive the run charts, the paired results and the staff measures in a two-page summary to support the adoption decision. The final manuscript in DNP 752 will follow the SQUIRE 2.0 guidelines, and the poster will present the three kinds of evidence side by side.

Summary

The evaluation measures what happens to patients, whether the pathway is delivered and how staff experience it, with an analysis matched to each type of data. The next section addresses ethics and the quality improvement determination.

References

Perla, R. J., Provost, L. P., & Murray, S. K. (2011). The run chart: A simple analytical tool for learning from variation in healthcare processes. BMJ Quality & Safety, 20(1), 46-51. https://doi.org/10.1136/bmjqs.2009.037895

Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65-76. https://doi.org/10.1007/s10488-010-0319-7

Weiner, B. J., Lewis, C. C., Stanick, C., Powell, B. J., Dorsey, C. N., Clary, A. S., Boynton, M. H., & Halko, H. (2017). Psychometric assessment of three newly developed implementation outcome measures. Implementation Science, 12, Article 108. https://doi.org/10.1186/s13012-017-0635-3