QNT 375 Week 3 Validity and Reliability Example

Can We Trust These Numbers? Evaluating Validity and Reliability in a Car Wash Retention Study

[Student Name]

University of Phoenix

QNT/375: Business Data Analytics

Week 3 Assignment

[Instructor Name]

[Date]

Summit Shine Car Wash, its locations, members, data and figures are composites written for a model paper.

Week 2 designed data collection for Summit Shine Car Wash, a composite Denver-area chain studying why members cancel. The data are now in hand: 12 months of membership records, 2.1 million wash log entries, queue camera wait estimates, weather and competitor data and 640 completed surveys. Before any analysis, this paper asks whether these data can be trusted.

Validity and Reliability

Cooper and Schindler (2014) explain that reliability concerns consistency, whether a measure produces stable results, and validity concerns whether it measures what it claims to measure. Validity takes several forms: content validity, whether a measure covers the concept; construct validity, whether it behaves as the concept should; internal validity, whether a study can support causal conclusions; and external validity, whether findings generalize. A measure can be reliable but not valid, consistently measuring the wrong thing.

Data Quality Beyond Accuracy

Wang and Strong (1996) surveyed data users and found that they judged data quality on many dimensions beyond accuracy, which they grouped into four families: qualities built into the data itself, qualities that depend on the task at hand, qualities of how data are presented and qualities of access. Each of Summit Shine's sources has strengths and weaknesses on these dimensions.

Wash Logs

License plate readers record each member wash. To test accuracy, staff at four locations manually recorded 1,500 member washes over two weeks. The readers missed 7 percent, mostly cars with dirty, damaged or obscured plates and mostly in winter. Missed washes lower measured visit frequency for affected members, which could make them look disengaged when they are not, a threat to the validity of the washing-frequency variable. Correction: estimate a missed-wash rate by location and month and adjust visit counts, and flag members with repeated read failures.

Wait Times

Queue cameras estimate wait times from the length of the line. Manual timing of 400 cars at three locations showed close agreement on clear days, with errors averaging under two minutes, but large errors on snowy days, when the cameras misjudged lines. The measure is reliable in good weather and unreliable in snow. Correction: use wait times from non-snow days for each location's average and treat snow-day estimates as missing.

Cancellation Reasons

The reason field is incomplete, missing for about two-thirds of cancellations, because it is optional online. Its options changed in May, merging "price" and "value" into one choice, so reasons before and after cannot be compared. The field fails on completeness and consistency. Correction: drop it from the main analysis and rely on the survey for reasons.

The field meant to explain why members leave turned out to be the least trustworthy data the company had.

The Survey Scale

The survey included a five-item satisfaction scale covering wash quality, speed, staff, value and convenience. Cronbach (1951) introduced coefficient alpha as a measure of internal consistency for multi-item scales. The scale's alpha was 0.86, indicating that the items measure a common concept consistently. Content validity was reviewed by three site managers, who agreed the items covered the main parts of the member experience. As a check on construct validity, satisfaction scores were lower among canceled members than current ones, as expected.

Nonresponse remains a concern. Canceled respondents had slightly longer average tenure than canceled nonrespondents, suggesting that the most recent and possibly least attached members are underrepresented. Survey results will be weighted by tenure group.

Threats to the Natural Comparison

The comparison of locations near and far from competitor openings faces threats to internal validity. One nearby location was remodeled during the study, closing for three weeks, which could raise cancellations for reasons unrelated to the competitor. Two distant locations had new road construction that slowed access. Correction: exclude remodel weeks and add indicators for construction periods. External validity is also limited: results from Denver's market may not apply to other cities.

Membership Records

The membership system is the most accurate source, since it drives billing, but it has its own quirks. Members who switch plans appear as a cancellation and a new sign-up on the same day, which inflates cancellation counts by about 4 percent. Members who pause their membership for travel, a feature added last year, were coded as canceled in the first two months after the feature launched. Correction: link plan switches into continuous memberships and recode early pauses. These fixes lower the measured cancellation rate slightly in every month but do not change the trend.

Competitor Distances

Distances from each Summit Shine site to the nearest competitor were calculated by straight line. Driving routes can differ sharply where highways and rivers intervene. A check of 20 locations found that straight-line and driving distances ranked sites the same way in 18 cases. The measure is acceptable for comparing near and far sites, though not for precise distance effects.

Summary of Corrections

Adjust wash counts for missed reads.

Use non-snow wait time estimates.

Drop the cancellation reason field; use survey reasons.

Weight survey responses by tenure.

Exclude remodel weeks and flag construction periods.

Documenting the Changes

Every correction is recorded in a data log: what was changed, why, how many records were affected and who approved it. The log lets anyone repeat the analysis, explains differences from figures managers have seen before and protects the analysis from the charge that results were adjusted to fit a conclusion. For example, the log notes that linking plan switches removed 3,180 false cancellations from the twelve-month total.

What Remains Uncertain

Even after corrections, some uncertainty remains. Adjusted wash counts are estimates. Survey reasons reflect what members remember and are willing to say. The natural comparison cannot rule out every other difference between locations. Conclusions should be stated with these limits in mind.

Conclusion

Summit Shine's data are rich but imperfect. Testing wash logs and wait times against manual counts, examining the reason field's completeness and consistency, checking the survey scale's reliability and identifying threats to the natural comparison show which measures can be trusted and how to correct the rest. The analysis in Week 4 can now proceed on firmer ground.

References

Cooper, D. R., & Schindler, P. S. (2014). Business research methods (12th ed.). McGraw-Hill Education.

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555

Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems, 12(4), 5-33. https://doi.org/10.1080/07421222.1996.11518099