HINF 520 Week 4 Systems Operations and Maintenance Example

The Night the A1c Numbers Dropped by a Third: Running, Monitoring and Maintaining a Health System's Diabetes Data Pipeline So Silent Failures Get Caught

[Student Name]

University of Phoenix

HINF/520: Data Management and Design in Health Administration

Week 4 Assignment

[Instructor Name]

[Date]

The health system, its data pipeline, incident and procedures are composites written for a model paper; research findings come from the sources listed.

On a Monday morning, the director of the composite health system's diabetes care program opened the dashboard built on the new data mart and saw that just 41% of patients with diabetes showed an A1c result within the prior half year, down from 63% the week before. Clinics had not stopped testing. The data had stopped arriving. This paper uses that incident to describe how the data mart is operated and maintained.

What Went Wrong

Over the weekend, one hospital's laboratory upgraded its information system. The upgrade replaced the local test code that had been mapped to the standard A1c code with a new internal code. Results continued to flow into the electronic record and appeared normally for clinicians, but the data mart's nightly load, which selected results by the mapped code, found none from that hospital's laboratory and its outreach clinics. No error occurred. The load completed successfully, with about a third fewer A1c results than usual.

Silent Failures Are Common

Failures like this are not rare. A study at one academic medical center described four malfunctions in clinical decision support, including an alert that stopped working when an internal identifier for a drug was changed in another system, and a survey found that 93% of responding chief medical information officers had experienced at least one decision support malfunction, two-thirds at least annually, often persisting for long periods (Wright et al., 2016). The data mart's failure followed the same pattern: a change in one system silently broke another.

Routine Operations

Every night at 1 a.m., an automated process extracts new and changed records from the data warehouse, transforms them using the value sets and mappings, applies the quality rules described in the design paper and loads them into the data mart. The process finishes by 4 a.m. Backups run after each load, and the data mart can be restored to any of the past 30 nights. An analyst on the operations rotation reviews the load summary each morning.

Monitoring Beyond Success or Failure

Before the incident, monitoring checked only whether the load finished. Afterward, the team added checks on content. A data quality framework distinguishes checks on conformance to expected formats and values, completeness of expected data and plausibility of values and patterns (Kahn et al., 2016). The new checks compare each night's volume of results by source, clinic and test with the average of the same weekday over the past eight weeks and flag drops or spikes of more than 20%. They also compare distributions, such as the share of A1c results above 9%, and flag sudden shifts. A load that finishes on time with a third of its data missing is not a success; it is an undetected failure.

Service Levels

The team defined service levels for the data mart: data current by 6 a.m. on business days, content checks reviewed by 8 a.m., users notified of any known problem by 9 a.m. and critical problems fixed within one business day. Meeting these levels is reported monthly.

Incident Management

When the Monday problem was reported, the on-call analyst opened an incident, posted a notice on the dashboard warning users not to rely on the A1c measures and traced the missing results to the laboratory code within three hours. A temporary mapping restored the data, and the missing results were reloaded by Tuesday morning. The incident log records each step and time.

Problem Management

An incident fix restores service; problem management asks why it happened and how to prevent it. The review found three causes: the laboratory had no list of downstream systems that relied on its codes, the upgrade's change notice did not mention code changes and the data mart had no content monitoring. Each became an action.

Change Management Across Systems

The system's change advisory board, which reviews changes to clinical systems, now requires every change to a laboratory, pharmacy or registration system to identify downstream data consumers, and the analytics team is on the notification list. Changes to the data mart itself follow the same process: documented, tested in a separate environment, approved and released on a set schedule with notice to users.

Maintenance

Maintenance keeps the data mart accurate as the world around it changes. Each quarter, analysts apply updated code sets, including new diagnosis codes each October and new drug ingredients, review mappings for new local laboratory codes, retire unused reports and tune slow queries. Annually, they review access rights and remove users who no longer need access.

Users as Monitors

Users caught the failure before the system did, which is common. The team now treats users as part of monitoring: the dashboard carries a feedback button that sends questions straight to the on-call analyst, and the diabetes program director and clinic managers receive a short weekly summary of data volumes so unusual patterns are visible to people who know what normal looks like. Every user report is logged and answered within one business day, even when the data turn out to be correct, because users who are ignored stop reporting.

Documentation

Operations depend on documentation that outlasts any one analyst. The team maintains a runbook describing each nightly job, its sources, expected volumes, common failure modes and recovery steps, along with a data lineage map showing which source fields feed each measure. The lineage map made it possible to trace the missing A1c results to one laboratory code in under an hour; without it, the analyst would have searched every source.

Roles

The data mart has named owners: an analytics manager accountable for operations, an on-call rotation of three analysts, a data steward in the quality department for definitions and a contact in each source system team. Before the incident, no one in the laboratory knew the data mart existed.

A Sociotechnical View

The incident was not only a technical failure. A sociotechnical model of health information technology treats technology problems as arising from eight dimensions that interact, from the software and its clinical content to the people who use it, the way work and messages flow, the organization's own policies, outside rules and the monitoring that should catch trouble (Sittig & Singh, 2010). The Monday failure involved several: software change, missing communication between teams, no organizational process linking them and no monitoring. Fixing only the mapping would have left the other causes in place.

Measuring Operations

The team tracks on-time loads, content check alerts and how many were real problems, time to detect and resolve incidents, changes that caused incidents and user-reported problems. In the six months after the new checks began, they caught four upstream changes before users noticed.

Conclusion

A laboratory upgrade silently removed a third of the diabetes data mart's A1c results, a failure of the kind that research shows is common in health IT. Operating a data system reliably requires more than nightly loads and backups: content monitoring, service levels, incident and problem management, change control across connected systems and clear ownership. Viewed through a sociotechnical lens, the fix was as much about people and processes as about code.

References

Kahn, M. G., Callahan, T. J., Barnard, J., Bauck, A. E., Brown, J., Davidson, B. N., Estiri, H., Goerg, C., Holve, E., Johnson, S. G., Liaw, S.-T., Hamilton-Lopez, M., Meeker, D., Ong, T. C., Ryan, P., Shang, N., Weiskopf, N. G., Weng, C., Zozus, M. N., & Schilling, L. (2016). A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. eGEMs, 4(1), 1244. https://doi.org/10.13063/2327-9214.1244

Sittig, D. F., & Singh, H. (2010). A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Quality and Safety in Health Care, 19(Suppl 3), i68-i74. https://doi.org/10.1136/qshc.2010.042085

Wright, A., Hickman, T.-T. T., McEvoy, D., Aaron, S., Ai, A., Andersen, J. M., Hussain, S., Ramoni, R., Fiskio, J., Sittig, D. F., & Bates, D. W. (2016). Analysis of clinical decision support system malfunctions: A case series and survey. Journal of the American Medical Informatics Association, 23(6), 1068-1076. https://doi.org/10.1093/jamia/ocw005