RES 710 Week 7 Appraising Quantitative Research Example

Reading the Numbers Critically: Appraising a Quantitative Study on Queue Technology

[Student Name]

University of Phoenix

RES/710: Statistical Research Methods and Design I

Week 7 Assignment

[Instructor Name]

[Date]

The learner, the credit union, the vendor study and all data are composites written for a model paper.

Two weeks after the Week 6 correlation results reached the credit union's leadership, a vendor of queue display systems sent the operations director a report titled as a study of member experience. It claims that branches using its screens, which show each member's place in line and expected wait, saw satisfaction rise 25 percent. The director's findings on perceived wait make the claim interesting, and the system costs about $18,000 per branch. Before recommending a purchase, she appraises the study.

Summary of the Study

The report describes six branches at a Midwest credit union that installed the displays and six that did not. Satisfaction was measured with one survey item before installation and three months after. The report states that the share of members rating satisfaction 9 or 10 on a 10-point scale rose from 32 percent to 40 percent in display branches, a 25 percent increase, and was unchanged in comparison branches. No statistical tests are reported. A footnote mentions that wait times, transaction counts and staff ratings were also tracked.

Choosing an Appraisal Standard

The STROBE guidance, explained in detail by Vandenbroucke et al. (2007), sets out what authors of observational research ought to describe, such as the setting and dates, eligibility and selection of participants, definitions of all outcomes, efforts to address bias, statistical methods and confounding and full reporting of results. Although written for epidemiology, it suits a nonrandomized comparison like this one, and the learner uses it as a checklist.

Design and Internal Validity

Branches were not randomly assigned; the report says displays went to branches "selected for the pilot." A later page shows that four of the six display branches were renovated in the same quarter, with new lobbies and seating. Renovation alone could raise satisfaction, so the design cannot separate the display's effect from the renovation's. The comparison branches provide some control for time trends, a strength, but only if they are otherwise similar, and the report gives no data on their size, membership or prior satisfaction trend.

Sampling and External Validity

The report gives no number of survey respondents per branch or period, no response rate and no description of who was surveyed. Without these, readers cannot judge precision or whether respondents changed between periods. The study involves one credit union, so generalizing to Grand Rapids requires judgment about similarity.

Measurement

Satisfaction was a single 10-point item, analyzed as the share choosing 9 or 10. Single items are less reliable than scales, and collapsing a 10-point scale into a top-box share discards information. Wait time was measured but not reported, though it is the mechanism the display is supposed to affect.

Analysis and Reporting

No test, interval or effect size is reported, and branches are the unit of assignment while members are the unit of measurement, so any test would need to account for clustering within 12 branches. With so few clusters, uncertainty would be large.

Simmons et al. (2011) showed through simulations and an experiment that flexibility in data collection and analysis, such as choosing among several outcomes after seeing results, can push false-positive rates far above the nominal 5 percent. Here, the footnote reveals nine tracked measures, but only one is reported. If the reported outcome was chosen because it looked best, the result may be a false positive.

The footnote is the most important line in the report: nine outcomes were tracked, and only one appears.

Recalculating the Headline

The 25 percent figure is a relative change: from 32 to 40 percent is 8 percentage points, and 8 divided by 32 is 25 percent. Assuming a typical distribution on the 10-point item, an 8-point rise in the top-box share corresponds to a mean increase of roughly 0.3 points on a 7-point scale, similar to the size of the gap between this credit union's satisfaction and its target. That could be worth having, but it is far smaller than "25 percent" suggests, and without an interval it could easily fall within normal quarter-to-quarter variation across six branches.

Credibility in Context

Ioannidis (2005) argued that the odds of a claimed finding holding up fall as samples shrink, effects get smaller, more relationships are tested, designs and analysis choices loosen and financial stakes rise. The vendor study meets almost every condition: 12 branches, a modest effect, nine outcomes, an unregistered analysis and a sponsor selling the product. None of this proves the claim false, but together these features mean the claim should carry little weight on its own.

What a Stronger Report Would Include

Following the reporting guidance, a credible version of this report would state how branches were chosen, give respondent counts and response rates for each branch and period, report all nine outcomes with means, intervals and tests that account for clustering and describe the renovation schedule and any other changes in the same period. It would also show each branch's satisfaction trend for a year before installation, so readers could see whether display branches were already improving. Several of these items could be supplied from data the vendor already holds, which is why asking for them is a reasonable first step.

Questions to Put to the Vendor

The learner drafted five questions: How were pilot branches chosen? How many members responded in each branch and period? What happened to the other eight outcomes, especially wait times? Were any display branches renovated, and can results be shown without them? And will the vendor share branch-level data for an independent analysis? The answers, or a refusal to answer, will themselves inform the decision.

Strengths

The study used a comparison group and measured before and after, which is better than a simple before-and-after design. It also targets a plausible mechanism, perceived wait, which the learner's own data support.

Ranking the Flaws

The most serious problems are the renovation confound and the selective reporting, since either could produce the result entirely. Next is the absence of any test or interval that accounts for clustering. Lesser problems are the single-item measure and missing sample details, which limit precision but would not by themselves create a false effect.

Recommendation

The learner recommends against buying displays for all 24 branches on this evidence. Instead, the credit union could run its own pilot: randomly assign displays to six of its 12 busiest branches, measure satisfaction with the existing four-item scale and recorded and perceived waits for three months, and analyze the results with methods that account for branches. Asking the vendor for all nine outcomes and the branch-level data would also help.

Conclusion

Appraised against STROBE items and research on false-positive and unreliable findings, the vendor study offers weak support for its headline claim. Confounding by renovation, selective reporting and a relative statistic that overstates a modest change outweigh its strengths. A small randomized pilot would answer the question for this credit union directly. Week 8 will apply the course's statistics to a full analysis plan.

References

Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. https://doi.org/10.1371/journal.pmed.0020124

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366. https://doi.org/10.1177/0956797611417632

Vandenbroucke, J. P., von Elm, E., Altman, D. G., Gøtzsche, P. C., Mulrow, C. D., Pocock, S. J., Poole, C., Schlesselman, J. J., & Egger, M. (2007). Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): Explanation and elaboration. PLoS Medicine, 4(10), e297. https://doi.org/10.1371/journal.pmed.0040297