PSYCH 655 Week 3 Validity Evidence Example

Does the Work-Sample Battery Predict Who Finishes Training? Building a Validity Argument for a Vocational Assessment

[Student Name]

University of Phoenix

PSYCH/655: Psychometrics

Week 3 Assignment

[Instructor Name]

[Date]

The vocational rehabilitation office, its battery and outcome data are composites written for a model paper; measurement theory and research findings come from the sources listed.

Asking whether a test is valid, full stop, is the wrong question. Validity concerns whether evidence supports a particular interpretation and use of scores. Below, the battery from last week gets a validity argument tied to one job it does in our office: steering clients toward short training programs.

The Intended Use

Our office uses the six-station work-sample battery, whose reliability I examined last week, to help counselors recommend clients for funded training programs of four to twelve weeks in three tracks: light assembly, warehouse and logistics and office support. The intended interpretation is that higher station scores relevant to a track indicate greater readiness to complete that track's training and obtain related work. The decision is consequential: recommendations influence funding and clients' plans.

An Argument-Based Approach

Kane (2013) proposed an argument-based approach to validation in which test developers and users first state an interpretation and use argument, laying out the chain of inferences from observed performance to scores, from scores to a broader domain, from the domain to the target construct or criterion and from there to decisions. Validation then evaluates the plausibility of each inference with evidence, focusing especially on the weakest links. Kane emphasized that the more ambitious the interpretation, the more evidence is needed.

For our battery, the chain includes several inferences: station tasks are scored accurately; scores generalize beyond the particular day and evaluator; station tasks represent the skills required in training and target jobs; scores relate to training completion; and using scores to make recommendations benefits clients without unfair consequences.

Evidence From Content

The scoring and generalization inferences were addressed by last week's reliability work. On content, I asked six employers and three training instructors to rate how well each station's tasks represented skills needed in each track. Assembly and sorting stations were rated highly relevant for the assembly and warehouse tracks. The cash register and written instructions stations were rated relevant for office support, but instructors noted that office training now emphasizes computer data entry and phone etiquette, which no station measures. This gap is a content deficiency for the office track.

Evidence From Relations With Other Variables

Campbell and Fiske (1959) proposed the multitrait-multimethod approach to evaluating validity: measuring several traits by several methods and examining the pattern of correlations. Convergent evidence appears when different methods measuring the same trait correlate highly; discriminant evidence appears when measures of different traits correlate less, even when they share a method. High correlations among different traits measured by the same method suggest that method variance is inflating scores.

We had data on two traits, manual dexterity and following written procedures, each measured by a battery station and by a standardized test. The assembly station correlated .64 with a standardized dexterity test, convergent evidence, but only .21 with a reading comprehension test, discriminant evidence. The written instructions station correlated .58 with reading comprehension and .19 with dexterity. The pattern suggests the stations measure distinct skills rather than general test-taking ability.

The assembly station tracked dexterity, not reading; that pattern is part of what makes its scores mean something.

Evidence From Response Processes

Another kind of evidence asks whether test takers engage the processes the test intends to measure. I observed twelve clients completing the written instructions station and asked them afterward to describe what they did. Most read each step and followed it, as intended, but three clients with limited English said they skipped written steps and copied what the evaluator had demonstrated during the practice item. Their scores therefore reflected imitation of a demonstration as much as following written procedures. This response-process evidence anticipates the consequences issue discussed below and suggests that the practice demonstration should be shortened or removed.

Evidence From Internal Structure

Within the battery, a simple factor analysis of the six stations with data from 240 clients suggested two clusters: a manual skills cluster, assembly, sorting, measuring and circuit board, and a procedural and clerical cluster, cash register and written instructions. This structure matches the battery's design and supports reporting the two composites separately rather than a single total.

Evidence From Prediction

We followed 112 clients who completed the battery and entered training over two years. Completion was recorded from training provider records. For the assembly track, the assembly and sorting station composite correlated .41 with completion; for warehouse, .36; for office support, the relevant station composite correlated only .18, not statistically reliable with that track's small sample. Counselors had access to scores when making recommendations, which may have introduced some criterion contamination if they gave extra support to high scorers.

Consequences of Score Use

Messick (1995) argued for a unified view of validity in which the consequences of score interpretation and use are part of the validity argument, alongside content, substantive, structural, generalizability and external aspects. If a test produces adverse consequences traceable to construct-irrelevant variance or construct underrepresentation, those consequences bear on validity.

We examined whether recommendations based on the battery steered some groups away from opportunities. Clients who were not native English speakers scored lower on the written instructions station, which affects office track recommendations, although some succeeded in office training when they enrolled anyway. Part of that station's difficulty may reflect English reading rather than procedural skill, construct-irrelevant variance for clients whose jobs would not require extensive English reading.

Alternative Explanations

A validity argument must also consider rival explanations for the predictive correlations. Clients with higher scores may complete training more often because of motivation, transportation or stable housing, factors related to both testing and training success. Our records show that clients with stable housing scored somewhat higher and completed training more often, so part of the battery's prediction may reflect life circumstances rather than skills. Controlling for housing status reduced the assembly track correlation from .41 to .35, still meaningful, which supports the skills interpretation while showing that it is not the whole story.

Weighing the Argument

The argument is reasonably strong for the assembly and warehouse tracks: content is relevant, convergent and discriminant evidence is supportive and prediction is moderate. It is weak for the office track: content is incomplete, prediction is low and consequences raise concerns.

Recommendations

Continue using assembly and warehouse composites as one input to recommendations, alongside interviews and work history. For the office track, add a computer data entry station, offer the written instructions station in Spanish where appropriate and stop using office scores as a gate until new evidence is gathered. Repeat the prediction study with a larger sample and record whether counselors' knowledge of scores affected support.

Conclusion

Validity is an argument about a use, built from several kinds of evidence. For our battery, evidence supports use in two tracks and reveals weak links in the third. An argument-based approach, the multitrait-multimethod logic and attention to consequences together show where the battery serves clients well and where it needs revision.

References

Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81-105. https://doi.org/10.1037/h0046016

Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1-73. https://doi.org/10.1111/jedm.12000

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749. https://doi.org/10.1037/0003-066X.50.9.741