HCS 483 Week 4 Information System Evaluation Example

Is the New Safety Event Reporting System Working? An Evaluation Plan Built on DeLone and McLean's Six Dimensions of Success

[Student Name]

University of Phoenix

HCS/483: Health Care Information Systems

Week 4 Assignment

[Instructor Name]

[Date]

The hospital, its reporting system and its figures are composites written for a model paper; frameworks come from the sources listed.

Twelve months ago, a 280-bed community hospital replaced its paper incident forms and after-hours phone line with an electronic safety event reporting system. Staff can now report a fall, medication error, near miss or equipment problem from any workstation or phone in about three minutes, and managers receive alerts and track investigations online. The system cost about $180,000 in licenses and implementation. Like other electronic systems, it promised legible, shareable information and better support for quality work, along with new costs and a period of disrupted routines (Menachemi & Collum, 2011). The patient safety committee now wants to know whether it is working. This paper sets out an evaluation plan built on a recognized model of information systems success.

Why the Hospital Bought the System

The purpose determines what counts as success. The old process produced about 90 reports a month, mostly from nurses and mostly about falls, and forms often arrived days late and incomplete. Leaders wanted four things: more reporting, especially of near misses; faster reporting; more complete and consistent data; and faster follow-up by managers. The evaluation must answer whether each happened.

A Six-Part Model of Success

DeLone and McLean (2003) updated their widely used success model to six dimensions that affect one another: the quality of the system, of the information it produces and of the support behind it, then the level of use, the satisfaction users feel and, finally, what the system is worth to people and the organization. The dimensions influence one another: a system that is reliable and easy to use, that produces good information and that is well supported is more likely to be used and liked, and use and satisfaction lead to benefits for individuals and the organization. The model suits this evaluation because it covers the technology, the people and the results.

System Quality

Question: Is the system available, fast and easy to use? Measures: uptime from vendor logs; median time to complete a report, measured from the system's timestamps; and staff ratings of ease of use on a short survey. Target: availability above 99.5% and median submission time under five minutes.

Information Quality

Question: Are reports complete, accurate and consistent? The Agency for Healthcare Research and Quality (n.d.) publishes Common Formats, standardized definitions and forms for patient safety event reporting, which the system uses. Measures: the share of reports with all required fields completed; agreement between the reporter's event category and the safety officer's review on a random sample of 100 reports; and the share of reports that include the time and location of the event. Target: 90% completeness and 85% category agreement.

Service Quality

Question: Do users get help when they need it? Measures: help desk tickets about the system per month, time to resolve them and the number of staff trained. The survey asks whether staff know whom to contact with a question.

Use

Question: Who is reporting, and how much? Measures: monthly reports by unit and by role, the share of near misses among all reports and the share of units that submitted at least one report a month. A rise in reports after implementation is usually a sign that reporting became easier and that staff feel safer speaking up, not that care became more dangerous, so the evaluation reads report volume alongside the severity mix.

User Satisfaction

Question: Do reporters and managers find the system worthwhile? Measures: an annual ten-item survey of a random sample of 300 staff, with separate questions for managers about the investigation tools, and a few open-ended comments.

Net Benefits

Question: Has the system improved safety work? Measures: median days from event to report and from report to completed manager review; the number of improvement actions documented from event reviews; and repeat events of the same type on the same unit. Where possible, the committee will compare the rate of injurious falls, adjusted for patient days, before and after, while recognizing that many other factors affect falls.

Voices Behind the Numbers

Numbers will show how much the system is used but not why some staff never use it. The evaluation adds three short focus groups: one with nurses from high-reporting units, one with staff from units that rarely report and one with managers who review events. Questions cover what makes reporting feel worthwhile or pointless, whether staff hear back after a report and whether fear of blame still keeps some events unreported. Feedback to reporters is a known driver of continued reporting, and focus groups are the quickest way to learn whether it is happening.

Design and Limits

The evaluation compares the 12 months before implementation, using paper form counts and dates, with the first 12 months after. The design cannot separate the system's effect from other changes, such as a hospital-wide safety culture campaign launched at the same time, and paper records from before are incomplete. The report will state these limits and focus on changes large enough to be meaningful.

Early Results as a Check on the Plan

A preliminary look at data from the first six months shows why several dimensions are needed. Reports rose to about 150 a month, and near misses rose from 8% to 22% of reports, a sign of stronger use. But completeness of required fields was only 74%, and several units still submitted no reports in some months. Those findings point to training and form design, not to failure of the system as a whole.

Reporting and Decisions

The evaluation will go to the patient safety committee and the chief nursing and medical officers at the one-year mark. Its findings will guide three decisions: whether to renew the license on current terms, what training and form changes to make and whether to extend the system to the hospital's outpatient clinics.

Conclusion

An electronic safety event reporting system succeeds only if it is reliable, produces good information, is supported, is used, is valued and helps the hospital act on what it learns. The DeLone and McLean model turns those ideas into questions and measures, and the evaluation ties each one to the reasons the hospital bought the system.

References

Agency for Healthcare Research and Quality. (n.d.). Common formats for event reporting. Patient Safety Organization Program. https://pso.ahrq.gov/common-formats

DeLone, W. H., & McLean, E. R. (2003). The DeLone and McLean model of information systems success: A ten-year update. Journal of Management Information Systems, 19(4), 9-30. https://doi.org/10.1080/07421222.2003.11045748

Menachemi, N., & Collum, T. H. (2011). Benefits and drawbacks of electronic health record systems. Risk Management and Healthcare Policy, 4, 47-55. https://doi.org/10.2147/RMHP.S12985