HINF 500 Week 6 Management Evaluation Report Example

Management Evaluation Report on Informatics at a 310-Bed Community Hospital: Five Areas Rated, a Sepsis Algorithm Questioned and a Two-Year Improvement Plan

[Student Name]

University of Phoenix

HINF/500: Informatics for Health Administration

Week 6 Assignment

[Instructor Name]

[Date]

The hospital, its ratings, costs and plan are composites written for a model paper; research findings and federal rules come from the sources listed.

Before the next capital and operating budget, the hospital's chief executive wanted a candid verdict on its informatics, and she gave the operations vice president the job of an honest evaluation of how well the hospital uses data and systems. This report rates five areas, examines one emerging risk in depth and recommends a prioritized plan.

Executive Summary

The hospital has a mature electronic record and capable technical staff, and recent projects on emergency department boarding, sepsis documentation and falls show that data can drive good decisions. But security controls lag current threats, clinical informatics leadership is thin, data ownership is unclear and a vendor sepsis prediction model is running without local evaluation. Eight recommendations costing $1.3 million over two years address these gaps.

Method

Each area was rated strong, adequate or needs improvement, based on interviews with 22 leaders and staff, review of reports and incident data, the analyses completed this year and comparison with research and federal requirements.

Area One: Data for Strategy (Adequate)

The boarding analysis turned record timestamps into a decision that avoided a $19 million bed addition, and the falls analysis used control charts to support a phased expansion. But these analyses depended on one vice president's initiative. Most departments still receive static monthly reports without trends or context.

Area Two: Security and Privacy (Needs Improvement)

Monthly audits of record access work, and privacy is independently overseen. However, multifactor authentication did not cover administrator accounts, backups had never been tested for full restoration and the phishing click rate was 14%. The ransomware attack at a neighboring system showed that these gaps carry real risk.

Area Three: Data Quality and Reporting (Adequate)

The sepsis review showed that the hospital's low score reflected documentation and abstraction problems more than slow care, and fixes are under way. Barcode timestamps are reliable; free-text documentation remains a weak point. No one owns data definitions across departments.

Area Four: Structure and Governance (Needs Improvement)

The department map showed a part-time physician informatics leader, no nursing counterpart, an irregular clinical council and 140 vendor relationships without regular review.

Area Five: Decision Support and Predictive Tools (Needs Improvement)

The hospital uses alerts, order sets and a vendor sepsis prediction model that scores every inpatient and alerts nurses when the score passes a threshold. It fires on about 16% of admissions. No one has checked how well it performs for this hospital's patients.

A Closer Look at the Sepsis Model

The concern is not theoretical. When one academic health system checked a widely sold sepsis model on roughly 38,000 of its own hospitalizations, the tool separated patients who would and would not develop sepsis only modestly better than chance, with a discrimination score of 0.63 where 1.0 is perfect. Two in three of the patients who became septic never triggered it, yet it alerted on nearly one admission in five, flooding nurses with false alarms (Wong et al., 2021). A model that performs well in its developer's data may perform poorly elsewhere. An algorithm the hospital has never tested is an opinion with a confidence score attached.

The Risk of Bias

Predictive tools can also be unfair. A study of a population health algorithm used to select patients for extra care found that, because it used past health costs as a proxy for need, Black patients assigned the same risk score as white patients were considerably sicker; correcting the design would have greatly increased the share of Black patients selected for help (Obermeyer et al., 2019). The hospital has never examined whether its sepsis model or its readmission risk score performs differently by race, sex, age or language.

What Federal Rules Require

A 2024 federal rule requires developers of certified health IT to support transparency for decision support interventions, including information on how predictive tools were developed, what data they used, how they were validated and their intended use and limits (Office of the National Coordinator for Health Information Technology, 2024). The hospital can request this information from its vendor and should use it in local evaluation.

Testing the Sepsis Model Locally

The analytics team proposed a straightforward local test. Using two years of admissions, analysts will identify patients who met clinical criteria for sepsis, then check how often the model's alert fired before those criteria were met, how many alerts fired for patients who never developed sepsis and whether performance differed by age, sex, race and preferred language. They will also measure what nurses did after alerts, since a model that fires constantly teaches staff to ignore it. If the model performs poorly, options include raising the threshold, limiting it to certain units, pairing it with a nurse screening tool or turning it off. The test will take about three months and requires no new software.

Why This Matters to the Budget

Predictive tools are often added at no visible cost because they come bundled with the record, which makes them easy to overlook in budgeting. Their costs appear elsewhere: nurse time spent responding to false alerts, missed cases the model fails to flag and the risk that unfair tools widen gaps in care. The evaluation recommends treating every predictive tool as a clinical intervention that requires evidence, monitoring and an owner.

Strengths to Protect

The evaluation also found strengths. Analysts know the data well, the privacy program is independent and active, the record is stable and well supported and leaders have begun to use data for decisions. The plan builds on these.

Recommendations in Priority Order

1. Require a second login factor on every privileged and email account and prove twice yearly that full backups can be restored; $140,000; information security officer; within six months.

2. Create an algorithm review committee to evaluate every predictive tool locally for accuracy and fairness before and after use, starting with the sepsis model; $90,000 for analyst time; chief medical information officer; first review within four months.

3. Expand clinical informatics leadership, with half-time physician and nurse informatics officers; $310,000 a year; chief medical and nursing officers; within six months.

4. Make the clinical informatics council a monthly standing body with nursing and pharmacy members; no added cost; chief medical information officer; next quarter.

5. Appoint a data governance lead and publish a data dictionary for the 50 most used measures; $120,000; analytics manager; within a year.

6. Replace static monthly reports with trend dashboards and control charts for key measures; $180,000; analytics manager; within 18 months.

7. Add two security analysts, funded partly by eliminating duplicate software licenses; $220,000 net; chief information officer; within a year.

8. Review all vendor contracts for use, cost and security terms; $40,000; chief information officer; within a year.

Measures

Progress will be tracked through phishing click rates, backup restoration test results, sepsis model accuracy and alert rates by patient group, clinician satisfaction with the record, measure definitions published and on-time completion of each recommendation, reported quarterly to the board.

Conclusion

The hospital's informatics foundation is sound, and this year's analyses show what data can do. The main risks lie in security, thin clinical informatics leadership, unowned data and untested algorithms. The eight recommendations, ranked by urgency and costed at $1.3 million over two years, turn this evaluation into a plan.

References

Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453. https://doi.org/10.1126/science.aax2342

Office of the National Coordinator for Health Information Technology. (2024). Health data, technology, and interoperability: Certification program updates, algorithm transparency, and information sharing. Federal Register, 89, 1192. https://www.federalregister.gov/d/2023-28857

Wong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., Ghous, M., & Singh, K. (2021). External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Internal Medicine, 181(8), 1065-1070. https://doi.org/10.1001/jamainternmed.2021.2626