IOP 480 Week 1 Reliability, Validity and Fairness in Assessment Example

Is the Hospitality Aptitude Test Any Good? Reliability, Validity and Fairness for a Resort Company

[Student Name]

University of Phoenix

IOP/480: Assessment Tools for Organizations

Week 1 Assignment

[Instructor Name]

[Date]

The resort company, staff, vendor and data are composites written for a model paper; measurement standards and research come from the sources listed.

Organizations use assessments to make decisions about hiring, promotion and development. Before trusting any assessment, they need evidence that it measures consistently, that its scores support the intended interpretation and that it treats people fairly. This paper applies those standards to a test a resort company is considering.

The Proposal

Saguaro Springs Resorts operates four resorts in the Phoenix area with about 2,400 employees and hires roughly two thousand people a year, many for seasonal front desk, food service and housekeeping roles. A vendor offers a forty-item online hospitality aptitude test that, it says, identifies people with "the service gene." The vendor's two-page summary reports a coefficient alpha of .91, a correlation of .45 between test scores and customer service ratings at a national retail chain and testimonials from three hotels. The chief people officer asks Jonah Whitaker, the company's HR analytics lead, whether to approve it.

Reliability

Reliability is the consistency of measurement. Test-retest reliability asks whether people get similar scores on different occasions. Internal consistency asks whether items hang together. Interrater reliability asks whether different raters agree. Without adequate reliability, scores contain so much random error that they cannot support confident decisions.

The vendor reports only internal consistency. There is no test-retest evidence, which matters because applicants may take the test more than once.

What Alpha Does and Does Not Show

Cortina (1993) explained that coefficient alpha reflects both the number of items and how strongly they correlate with one another. A long test can produce a high alpha even when items measure several different things. Alpha is therefore not evidence that a test measures one trait, and very high values can reflect redundant items rather than broad coverage. Cortina recommended reporting the number of items and examining the structure of the test rather than relying on alpha alone.

With forty items, an alpha of .91 is unsurprising. It does not show that the test measures a coherent "service" trait, and the vendor provides no analysis of the test's internal structure.

Validity as an Argument

Messick (1995) treated validity as one overarching question: how strongly do evidence and theory back the particular meaning and use given to the scores? Messick identified several aspects of validity evidence, including content, the substantive processes respondents use, the internal structure of scores, generalizability across groups and settings, relations with external criteria and the social consequences of using the test. Validity belongs not to the test itself but to particular interpretations and uses.

Applied to the vendor's evidence, the argument is thin. The single correlation of .45 comes from a retail chain, not resorts, and the criterion, "customer service ratings," is undefined: were they supervisor ratings, customer surveys or mystery shoppers? There is no content evidence linking items to resort job tasks, no evidence that results generalize to Saguaro Springs' applicant pool and no consideration of consequences, such as screening out applicants with limited English who could perform housekeeping work well.

A forty-item test that agrees with itself at .91 can still be measuring the wrong thing very consistently.

Fairness

The Standards for Educational and Psychological Testing (American Educational Research Association et al., 2014) treat fairness as a fundamental validity issue. They call for test developers and users to minimize construct-irrelevant barriers, to examine whether scores have the same meaning across groups, to provide accommodations when appropriate and to evaluate the consequences of testing for different groups. Users are responsible for gathering evidence that supports their own use of a test.

The vendor offers no data on score differences by race, ethnicity, sex, age or language background and no evidence on measurement bias. The resort company draws applicants from across the Valley, and a large share grew up speaking Spanish at home. If the test is offered only in English and requires reading beyond the job's demands, it may measure English reading rather than service aptitude.

What Jonah Recommends

Jonah recommends not approving the test for hiring decisions yet. He proposes a three-step path. First, request the vendor's full technical manual, including test-retest reliability, structure analyses, validity studies with defined criteria and subgroup data. Second, conduct a job analysis of the main entry-level roles so items can be compared with actual tasks. Third, run a local pilot: give the test to new hires without using scores, then compare scores with supervisor ratings using a structured form, guest comment scores and ninety-day retention, and examine differences across groups. Only if local evidence supports valid, fair use would scores inform decisions.

Content Evidence From the Job

Content evidence asks whether a test samples the knowledge, skills and behaviors the job requires. Jonah reads the forty items and finds that most ask about preferences, such as "I enjoy meeting new people," and none involves handling a guest complaint, juggling check-ins during a rush or noticing a safety hazard in a room. A housekeeper who prefers quiet work could score low yet excel at the job. Pairing the job analysis with an item review by experienced supervisors would show whether the content fits resort work at all.

Consequences of Use

Messick's inclusion of consequences means asking what happens when a test is used. If the test screens out older applicants, people with limited English or applicants with disabilities at higher rates, and those differences are unrelated to job performance, the test causes harm and legal exposure regardless of its reliability. Jonah adds a review of consequences to the pilot: after one season, the company will compare pass rates by group and examine whether any differences track real differences in later performance.

Costs and Benefits

The chief people officer worries about delay. Jonah notes that hiring with an unvalidated test risks rejecting good applicants, accepting poor ones and inviting legal challenge. A pilot over one hiring season costs little and produces evidence the company owns.

Conclusion

The vendor's test reports high internal consistency and one correlation from a different industry, which falls far short of the evidence needed. Research on what alpha shows, a unified view of validity and the professional testing standards all point to the same conclusion: Saguaro Springs needs local evidence of validity and fairness before using the test to make hiring decisions.

References

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.

Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78(1), 98-104. https://doi.org/10.1037/0021-9010.78.1.98

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as inquiries into score meaning. American Psychologist, 50(9), 741-749. https://doi.org/10.1037/0003-066X.50.9.741