MHA 515 Week 2 Validating Information About Health Sector Trends Example

Half the Documentation Time, the Brochure Said: Validating a Vendor's AI Scribe Claims Against a Randomized Trial and a Before-and-After Study

[Student Name]

University of Phoenix

MHA/515: Scanning the Health Sector as an Industry Expert

Week 2 Assignment

[Instructor Name]

[Date]

The health system, the vendor, its claims and the decision are composites written for a model paper; study findings come from the published sources cited.

Two months after the environmental scan flagged artificial intelligence for closer review, the chief medical officer of a composite three-hospital Tucson system forwarded a vendor's brochure to the director of strategic planning. The brochure promised that its ambient artificial intelligence scribe, which listens to visits and drafts clinical notes, would cut physicians' documentation time in half and end burnout. The vendor proposed a contract covering all 900 employed and affiliated physicians. The chief medical officer asked whether the claims were true. This paper describes how the director validated the information.

Why Validation Matters Now

Trends generate enthusiasm and sales pitches faster than evidence. A system-wide contract would cost several million dollars over three years, and physician time and trust are hard to recover after a failed rollout. Validating information before acting protects both money and credibility.

The Criteria

The director worked through a widely taught five-part test covering timeliness, fit, expertise, evidence and intent: how recent the information is, whether it fits the question, who produced it and with what expertise, whether it is supported by evidence and whether it is meant to inform or to sell (Blakeslee, 2004).

Testing the Brochure

The brochure was current and relevant. Its author was the vendor, whose purpose was commercial. The claim of halving documentation time cited "customer results" without naming a study, sample or comparison group. On accuracy and purpose, the brochure failed: it offered a claim, not evidence.

Tracing the Claim

The director asked the vendor for the source of the 50% figure. The vendor supplied a slide from one customer showing that physicians who used the tool most heavily spent less time in notes than before. There was no comparison group, and light users were excluded. The number was real but could not support a conclusion about all physicians.

Evidence One: A Multicenter Study

The director searched for published research. A multicenter quality improvement study enrolled 451 clinicians, of whom 263 in ambulatory practice completed surveys before and after 30 days of using an ambient scribe. The proportion reporting burnout fell from 51.9% to 38.8%, and clinicians reported lower cognitive load for notes, more undivided attention to patients and less documentation after hours (Olson et al., 2025).

Judging the First Study

The study is peer reviewed and large, and its authors have clinical and research expertise. But it compares the same clinicians before and after, with no control group, and 60.3% of enrolled clinicians completed both surveys. Burnout might have fallen for other reasons, and clinicians who liked the tool may have been more likely to respond. The study shows association and change, not proof of cause.

Evidence Two: A Randomized Trial

A pragmatic randomized trial assigned 238 outpatient physicians in 14 specialties to one of two ambient scribe products or usual care for two months. One product reduced time in notes by 9.5% compared with the control group; the other showed no significant change. Physicians used the tools in about a third of visits. Burnout, task load and work exhaustion measures suggested improvement for both products, but the authors called for confirmation in larger trials and reported that clinically significant inaccuracies occurred occasionally (Lukac et al., 2025). The best evidence found a benefit about a fifth the size of the brochure's promise, for one product.

Judging the Second Study

Randomization makes the trial the strongest evidence available on whether the tools cause change. Its limits are a single health system, a short period and use in only a third of visits. Its finding that two products performed differently is important: evidence about one tool does not transfer automatically to another.

Reconciling the Evidence

The sources agree on direction: physicians tend to feel better with ambient scribes, and some time is saved. They differ on size. The brochure claimed 50%; the randomized trial found 9.5% for one product and no significant effect for the other. The burnout findings are encouraging but come mainly from a study without a control group. Occasional inaccuracies mean notes still need physician review.

Questions the Evidence Could Not Answer

The published studies did not show effects on patient outcomes, errors reaching patients, billing or visit volume. They also did not test the specific product the vendor was selling. These gaps became questions for a local pilot.

Other Voices

The director spoke with two peer systems that had adopted scribes. One reported strong physician satisfaction; the other had seen uneven use, with many physicians returning to their old workflows. Anecdotes are weaker than studies, but they confirmed that results depend on implementation.

Matching the Decision to the Evidence

Promising but mixed evidence does not justify a 900-physician contract. It does justify a careful pilot. The director recommended a six-month pilot with 60 physicians across primary care and three specialties, randomized by clinic to start immediately or after three months, so the system could compare users with non-users.

Measures for the Pilot

The pilot will measure time in notes and after-hours documentation from the electronic record's audit logs, burnout on a validated survey, note accuracy through physician review of a sample, patient experience scores and cost. The vendor agreed to a contract that allows the system to stop after the pilot without penalty.

Privacy and Consent

Ambient scribes record conversations with patients. The pilot requires patient notice and the option to decline, a business associate agreement with the vendor and review of how recordings are stored and deleted.

The Cost Question

The finance team asked what the pilot's evidence would need to show to justify the full contract. At the vendor's price, the system would need each participating physician to add roughly one visit a week, or to stay rather than leave because of burnout, for the tool to pay for itself. The published studies did not measure either outcome, so the pilot added visit volume and physician retention intentions to its measures.

Watching for Harm

Because the randomized trial reported occasional clinically significant inaccuracies, the pilot includes a safety check. Each participating physician reviews every draft note before signing, and a clinical reviewer audits a random sample of 20 notes per physician per month, comparing the note with the recording. Any error that could affect care is reported through the safety system, and the pilot stops if serious errors appear.

Presenting the Findings

The director presented a one-page summary: the claim, what the evidence showed, how strong it was, what remained unknown and the recommendation. The chief medical officer and chief financial officer supported the pilot.

Limits of This Review

The review covered two peer-reviewed studies, the vendor's material and peer conversations. More studies are being published quickly, and the review will be updated before any wider decision.

Conclusion

The brochure's claim of halving documentation time did not survive validation. Applying source criteria exposed its commercial purpose and missing evidence, and two published studies of different designs showed real but smaller benefits, with differences between products and a need for accuracy checks. By matching the decision to the strength of the evidence, the system chose a measured pilot that will produce its own answer before it commits to a system-wide purchase.

References

Blakeslee, S. (2004). The CRAAP test [Article in LOEX Quarterly, vol. 31, no. 3]. LOEX Clearinghouse for Library Instruction, Eastern Michigan University. https://commons.emich.edu/loexquarterly/vol31/iss3/4

Lukac, P. J., Turner, W., Vangala, S., Chin, A. T., Khalili, J., Shih, Y. T., Sarkisian, C., Cheng, E. M., & Mafi, J. N. (2025). Ambient AI scribes in clinical practice: A randomized trial. NEJM AI, 2(12). https://doi.org/10.1056/AIoa2501000

Olson, K. D., Meeker, D., Troup, M., Barker, T. D., Nguyen, V. H., Manders, J. B., Stults, C. D., Jones, V. G., Shah, S. D., Shah, T., & Schwamm, L. H. (2025). Use of ambient AI scribes to reduce administrative burden and professional burnout. JAMA Network Open, 8(10), e2534976. https://doi.org/10.1001/jamanetworkopen.2025.34976