ORG 726 Week 5 Data, Analytics and Decisions Example

Models, Judgment and Who Decides: Data and Analytics in an Insurer's Underwriting and Claims Decisions

[Student Name]

University of Phoenix

ORG/726: The Impact of Technology on Organizations

Week 5 Assignment

[Instructor Name]

[Date]

Prairie Shield Mutual, its models, people and figures are composites written for a model paper.

Prairie Shield, the invented Iowa mutual examined since Week 1, now makes thousands of decisions a week with models: fraud scores on every claim, routing of claims to adjusters and recommended premiums for farm policies. Leaders want to know which decisions should belong to models and which to people. This paper reviews the research and applies it to three decisions.

Data-Driven Decision Making

Brynjolfsson and McElheran (2016) used Census data on U.S. manufacturing plants and reported that the share of plants basing decisions on data rose sharply in the late 2000s and that such plants were more productive, with the relationship strongest where complementary assets such as information technology and educated workers were present. Their findings suggest that data help most when organizations have the skills and systems to use them, a condition relevant to Prairie Shield, which has only two data scientists.

Models Versus Experts

Grove et al. (2000) conducted a meta-analysis of 136 studies comparing mechanical prediction, using statistical rules, with clinical prediction, using expert judgment, across medicine, psychology and other fields. Mechanical methods were substantially more accurate in about half the studies and about equally accurate in most of the rest; clinical judgment was more accurate in only a small number. The advantage held regardless of the judges' experience or the type of prediction. The finding supports using models for repetitive, structured predictions with good data.

How People Respond to Algorithms

Dietvorst et al. (2015) ran several forecasting studies in which participants who saw an algorithm make errors became less likely to choose it over a human forecaster, even when the algorithm outperformed the human, a pattern they called algorithm aversion. People seemed to forgive human errors more readily than algorithmic ones. Logg et al. (2019) found in other experiments that laypeople often gave more weight to advice when told it came from an algorithm than from a person, which they called algorithm appreciation, but that experts discounted algorithmic advice and lost accuracy as a result. Together these studies suggest that responses to algorithms depend on who is deciding and what they have seen.

Experienced underwriters behave like the experts in the experiments: they trust their own judgment most, sometimes at a cost.

Decision 1: Fraud Scoring

The fraud model scores every claim from 1 to 100; claims above 80 go to the special investigations unit. Before the model, adjusters referred claims based on experience. In its first year, the model's referrals produced confirmed fraud in 31 percent of cases, against 18 percent for adjusters' referrals, and it flagged about 40 percent more fraud overall. These results fit the meta-analytic evidence. But in the spring, a new scheme involving staged hail damage through a group of contractors went undetected by the model, which had no examples of it, until an adjuster noticed the same contractor on several claims. Models trained on past patterns miss new ones, while experienced adjusters can notice anomalies the data do not yet contain.

Decision 2: Claims Routing

As Week 2 showed, locked routing in Omaha produced delays and turnover, while Des Moines's override rights allowed adjusters to correct misrouted claims and fed improvements back to the model. Overrides in Des Moines were correct about 70 percent of the time when reviewed, suggesting adjusters held information the model lacked, such as what they observed at the property.

Decision 3: Farm Pricing

Week 3 found that underwriters resisted the pricing tool partly because it missed hazards visible only on inspection. After an inspection input was added, the model's first-year loss ratio predictions improved, and underwriter overrides fell from 44 to 21 percent of quotes. Here, the best results came from adding human observation to the model, not from choosing between them.

Why the Underwriters Discount the Model

The Logg et al. finding about experts fits Prairie Shield's underwriters closely. Senior underwriters, the most experienced, discounted the pricing model most, as Week 3 showed, and some of their overrides cost accuracy: a review of 200 overrides found that those reducing the model's premium were followed by higher losses than those raising it. But Dietvorst's finding about errors also applies. Underwriters saw the model miss equipment hazards in its first months, and their trust fell. Fixing the hazard input and showing outcome data addresses both: it removes a visible error source and gives experts evidence about when their judgment adds value.

When Models Should Not Decide

Some decisions should stay with people even if a model could predict well. Decisions to deny a claim or to cancel a member's policy carry consequences that members, regulators and the company's mutual mission require a person to own. A model can inform such decisions, but a named person must be able to explain them. Speed and consistency are not the only values at stake; accountability and the member's right to an explanation matter too, a theme Week 6 takes up.

Organizational Conditions

Analytics require more than models. Prairie Shield's data are uneven: claims data are rich, but farm property data depend on agents' entries, which vary in quality. Model governance is informal; no one is responsible for monitoring models for drift or for reviewing whether they treat members in different regions fairly. Feedback loops are weak: adjusters and underwriters rarely see whether their overrides were right.

A Framework for Dividing Decisions

Structured, repetitive decisions with good data and modest error costs, such as auto-paying simple claims: models decide, with sampling audits.

Structured decisions with high error costs, such as fraud referral: models screen, people decide, with people encouraged to flag patterns the model misses.

Decisions where people hold information the model lacks, such as routing after inspection and farm pricing: models recommend, people decide with recorded reasons, and overrides are reviewed.

Novel or ambiguous decisions, such as large complex losses: people decide, with models as one input.

Costs of the Framework

The framework is not free. Recording reasons for overrides adds a minute or two to each decision; reviewing them takes senior staff time. Fairness and drift monitoring needs at least one more data scientist. These costs are small next to the losses from an unmonitored model or from ignoring adjusters' knowledge, but they should be budgeted rather than assumed.

Governance and Learning

A model risk committee will monitor each model's accuracy and drift quarterly, review fairness across regions and approve changes. Outcome tracking will compare model and human decisions, and results will be shared with adjusters and underwriters, addressing algorithm aversion with evidence rather than mandates.

Conclusion

Research shows that statistical models outperform human judgment in many structured predictions and that organizations using data well tend to perform better. It also shows that people's trust in models depends on experience and on seeing errors. At Prairie Shield, fraud scoring, routing and pricing illustrate both the strength of models and the value of human information. A framework that assigns decisions by structure, error cost and information, with governance and feedback, lets the insurer use both.

References

Brynjolfsson, E., & McElheran, K. (2016). The rapid adoption of data-driven decision-making. American Economic Review, 106(5), 133-139. https://doi.org/10.1257/aer.p20161016

Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114-126. https://doi.org/10.1037/xge0000033

Grove, W. M., Zald, D. H., Lebow, B. S., Snitz, B. E., & Nelson, C. (2000). Clinical versus mechanical prediction: A meta-analysis. Psychological Assessment, 12(1), 19-30. https://doi.org/10.1037/1040-3590.12.1.19

Logg, J. M., Minson, J. A., & Moore, D. A. (2019). Algorithm appreciation: People prefer algorithmic to human judgment. Organizational Behavior and Human Decision Processes, 151, 90-103. https://doi.org/10.1016/j.obhdp.2018.12.005