PSYCH 655 Week 2 Reliability Example

Would Two Evaluators Give the Same Score? Reliability Evidence for a Vocational Work-Sample Battery

[Student Name]

University of Phoenix

PSYCH/655: Psychometrics

Week 2 Assignment

[Instructor Name]

[Date]

The vocational rehabilitation office, its work samples and the reliability data are composites written for a model paper; research findings come from the sources listed.

A score that changes depending on the day of testing or the person scoring it cannot support important decisions. Reliability is the consistency of scores, and different kinds of reliability evidence answer different questions. This paper examines the reliability of a work-sample battery used at the vocational rehabilitation office where I work.

The Battery

Our office in Albuquerque uses a six-station work-sample battery to help clients and counselors choose training paths. Stations include small-parts assembly, sorting mail by zip code, measuring and cutting materials to specification, operating a cash register simulation, assembling a simple circuit board and following a set of written multistep instructions to complete a task. Each station yields a score combining speed and accuracy, and scores are combined into a profile. Recommendations based on the battery affect whether clients are funded for particular training programs, so the stakes are meaningful.

True Scores and Error

Classical test theory proposes that any observed score equals a true score plus error. The true score is the average a person would obtain over infinite independent testings; error includes anything that makes an observed score differ from it, such as fatigue, distraction, luck in guessing or differences between scorers. Reliability is the proportion of observed score variance due to true score variance, ranging from zero to one. No single reliability coefficient captures all sources of error, so test users must choose evidence that fits their purpose.

Internal Consistency and Its Limits

Cronbach (1951) introduced coefficient alpha as a general formula for the internal consistency of a test, showing that it equals the mean of all possible split-half coefficients and that it reflects how strongly items or parts covary. Alpha became the most widely reported reliability statistic in psychology.

McNeish (2018) argued that alpha is widely misused because it rests on assumptions rarely met in practice, especially that every item relates equally strongly to the underlying construct, called tau equivalence, and that errors are uncorrelated. When these assumptions fail, alpha can underestimate or overestimate reliability. McNeish recommended alternatives such as omega coefficients computed from factor models and urged researchers to choose reliability estimates that fit their data.

Our battery's six stations measure different skills by design, so high internal consistency across stations is neither expected nor desirable; alpha across stations was .58, which reflects the battery's breadth rather than poor measurement. Internal consistency within each station matters more, but the most important evidence for our purposes is stability over time and agreement between raters.

Test-Retest Reliability

Thirty clients completed the full battery twice, two weeks apart, with no training between sessions. The correlation between the two sets of total scores was .84, and station correlations ranged from .71 for circuit board assembly to .88 for sorting. Scores rose slightly on retest, by about four percent on average, a practice effect, particularly on the timed stations. A two-week interval balances memory and practice effects against real change in skills.

Interrater Reliability

Several stations require evaluator judgment. Koo and Li (2016) provided a guideline for selecting and reporting intraclass correlation coefficients, explaining that there are several forms depending on whether raters are a random sample or fixed, whether a single rater or the mean of raters is used and whether absolute agreement or consistency is of interest. They suggested that values below .5 indicate poor reliability, .5 to .75 moderate, .75 to .9 good and above .9 excellent, and urged researchers to report which form they used.

Two evaluators independently scored twenty clients. Using a two-way random effects model for absolute agreement and single raters, interrater reliability was .91 for timed stations, where scoring depended mainly on a stopwatch and error counts, but only .62 for the written instructions station, where evaluators judged whether each step was completed correctly.

On the stopwatch stations, two evaluators agreed almost perfectly; on the station that needed judgment, they agreed only moderately.

Uncertainty Around One Score

The standard error of measurement estimates how much an individual's observed score would vary around their true score. It equals the standard deviation times the square root of one minus the reliability. For the battery total, with a standard deviation of 12 points and test-retest reliability of .84, the standard error is about 4.8 points. A client scoring 70 has a 95 percent confidence band of roughly 61 to 79. For the written instructions station, lower reliability produces a much wider band, which counselors should understand before treating small score differences as meaningful.

How Reliable Is Reliable Enough?

The level of reliability needed depends on the stakes. For research comparing group averages, coefficients around .70 may be acceptable. For decisions about individuals, such as whether a client is funded for a specific training program, much higher reliability is expected, often .85 to .90 or above for key scores, because error in an individual score directly affects that person. Our battery total, at .84, approaches that level; the written instructions station, at .62 for interrater agreement, falls well short and should not drive decisions on its own.

Reliability Is Not Validity

High reliability shows that scores are consistent, not that they measure what the office intends or predict success in training. A station could be scored with perfect agreement and still have little to do with the jobs clients pursue. Validity evidence, the subject of next week, must be examined separately. Reliability sets an upper limit on validity: a test cannot correlate with an outcome more strongly than its scores correlate with themselves.

Sources of Error

The main sources of error were practice effects on timed stations and evaluator judgment on the written instructions station, where the rubric left room for interpretation about partially completed steps. Fatigue may also affect clients with certain disabilities in later stations.

Clients With Disabilities

Our clients have a wide range of disabilities, which raises reliability questions of its own. A client with chronic pain may perform differently on a morning and an afternoon, and a client with a seizure disorder may have good and bad weeks. For such clients, a single administration may not represent typical performance, and evaluators may need to schedule testing across two sessions or note conditions that could have depressed scores. These considerations do not lower the reliability coefficient for the group, but they matter for the meaning of an individual's score.

Improvements

I recommended rewriting the written instructions station's scoring rubric with specific examples of complete, partial and incorrect steps, training evaluators with practice videos until agreement reaches at least .80, rotating the order of stations to spread fatigue effects and reporting confidence bands with every score.

Conclusion

Reliability evidence must match how scores are used. For our work-sample battery, test-retest and interrater reliability matter more than alpha. The battery is stable over time and reliably scored on timed stations, while the judgment-based station needs a clearer rubric. Confidence bands make the remaining uncertainty visible to the people who rely on the scores.

References

Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555

Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155-163. https://doi.org/10.1016/j.jcm.2016.02.012

McNeish, D. (2018). Thanks coefficient alpha, we'll take it from here. Psychological Methods, 23(3), 412-433. https://doi.org/10.1037/met0000144