Corrections for criterion reliability in validity generalizationThe consistency of hermes, the utility of midas
- Jesús F. Salgado
- Silvia Moscoso
- Neil Anderson
ISSN: 1576-5962
Year of publication: 2016
Volume: 32
Issue: 1
Pages: 17-23
Type: Article
More publications in: Revista de psicología del trabajo y de las organizaciones = Journal of work and organizational psychology
Abstract
There is criticism in the literature about the use of interrater coefficients to correct for criterion reliability in validity generalization (VG) studies and disputing whether .52 is an accurate and non-dubious estimate of interrater reliability of overall job performance (OJP) ratings. We present a second-order meta-analysis of three independent meta-analytic studies of the interrater reliability of job performance ratings and make a number of comments and reflections on LeBreton et al.�s paper. The results of our meta-analysis indicate that the interrater reliability for a single rater is .52 (k = 66, N = 18,582, SD = .105). Our main conclusions are: (a) the value of .52 is an accurate estimate of the interrater reliability of overall job performance for a single rater; (b) itis not reasonable to conclude that past VG studies that used .52 as the criterion reliability value have a less than secure statistical foundation; (c) based on interrater reliability, test-retest reliability, and coefficient alpha, supervisor ratings are a useful and appropriate measure of job performance and can be confidently used as a criterion; (d) validity correction for criterion unreliability has been unanimously recommended by �classical� psychometricians and I/O psychologists as the proper way to estimate predictor validity, and is still recommended at present; (e) the substantive contribution of VG procedures to inform HRM practices in organizations should not be lost in these technical points of debate.
Funding information
The research reported was partially supported by Grant PSI2014-56615-P from the Ministry of Economy and Competitiveness to Jesús F. Salgado and Silvia Moscoso and by a Leverhulme Trust grant number IN-2012-095 to Neil Anderson. Appendix Formulas used 1) Formula to attenuate the reliability coefficient due to range restriction (Formula of Otis-Kelley): = = unrestricted reliability coefficient. r y y = 1 − U 2 1 − R y y ; where U SD / sd and R yy 2) Formula to correct validity for range restriction (Thorndike's Case I; Pearson, 1902; Thondike,1949): = = observed validity; R = corrected validity. R 12 = 1 − u 2 2 1 − r 12 2 ; where u sd / SD ; r 12 12 3) Formula to correct validity for double range restriction ( Pearson, 1908 ): = = sd/SD in variable 2; = observed validity, and = validity corrected for double range restriction. R 12 = r 12 u 1 u 2 1 − r 12 2 1 − u 1 2 1 − r 12 2 1 − u 2 2 ; where u 1 sd / SD in variable 1, and u 2 r 12 R 12Funders
- Ministerio de Economía y Competitividad Spain
-
Leverhulme Trust
United Kingdom
- IN-2012-095
Bibliographic References
- N.R. Anderson Relationships between practice and research in personnel selection: Does the left hand know what the right is doing? A. Evers, N. Anderson, O. Smit-Voskuyl (Eds.), Handbook of personnel selection 2005 Blackwell Oxford, UK 1 24
- L.J. Cronbach Essentials of psychological testing Fifth edition 1990 Harper & Row New York, NY
- E. Ghiselli, and C. Brown Personnel and industrial psychology 1948 McGraw-Hill New York, NY
- E. Ghiselli, J.P. Campbell, and S. Zedeck Measurement theory for the behavioral sciences 1981 Freeman San Francisco, CA
- J.P. Guilford Psychometric methods 1954 McGraw-Hill New York, NY
- R.M. Guion Personnel testing 1965 McGraw-Hill New York, NY
- R.M. Guion Assessment, measurement, and prediction for personnel decisions 1998 Lawrence Erlbaum Associates Mahwah, NJ
- R.M. Guion, and S. Highouse Essentials of personnel assessment and selection 2006 Erlbaum Mahwah, NJ
- H. Gulliksen Theory of mental tests 1950 Wiley New York, NY
- J.E. Hunter, and R.F. Hunter Validity and utility of alternative predictors of job performance Psychological Bulletin 96 1984 72 98
- T.L. Kelley The reliability of test scores Journal of Educational Research 3 1921 370 379
- J.M. LeBreton, K.T. Scherer, and L.R. James Correction for criterion reliability in validity generalization: A false prophet in a land of suspended judgment Industrial and Organizational Psychology 7 2014 478 500
- Q. McNemar Psychological statistics 3rd edition 1962 Wiley New York, NY
- K.R. Murphy, and R. De Shon Interrater correlations do not estimate the reliability of job performance ratings Personnel Psychology 53 2000 873 900
- J. Nunnally Psychometric methods 1978 McGraw-Hill New York, NY
- A.S. Otis A method for inferring the change in a coefficient of correlation resulting from a change in the heterogeneity of the group Journal of Educational Psychology 13 1922 293 294
- K. Pearson On the influence of double selection on the variation and correlation of two characters Biometrika 6 1908 111 112
- Ronan, W.W., & Prien, E. (Eds.) (1971). Perspectives of the measurement of human performance. New York, NY: Appleton-Century Crofts.
- H.R. Rothstein Interrater reliability of job performance ratings: Growth to asymptote level with increasing opportunity to observe Journal of Applied Psychology 75 1990 322 327
- P.R. Sackett The status of validity generalization research: Key issues in drawing inferences from cumulative research studies K.R. Murphy (Ed.), Validity generalization: A critical review 2003 Lawrence Erlbaum Associates, Inc Mahwah, NJ 91 114
- P.R. Sackett, D.J. Putka, and R.A. McCloy The concept of validity and the process of validation Neal Schmitt (Ed.) The Oxford handbook of personnel assessment and selection. 2012 Oxford University Press Oxford, UK
- J.F. Salgado Estimating coefficients of equivalence and stability for job performance ratings: the importance of controlling for transient error on criterion measurement International Journal of Selection and Assessment 23 2015 37 44
- J.F. Salgado, N. Anderson, S. Moscoso, C. Bertua, F. De Fruyt, and J.P. Rolland A meta-analytic study of general mental ability validity for different occupations in the European Community Journal of Applied Psychology 88 2003 1068 1081
- J.F. Salgado, N. Anderson, and G. Tauriz The validity of ipsative and quasi-ipsative forced-choice personality inventories for different occupational groups: A comprehensive meta-analysis Journal of Occupational and Organizational Psychology 88 2015 797 834
- Salgado, J. F., Bastida, M., Vázquez, S., & Moscoso, S. (manuscript under review). Happiness, positive emotions and job performance: a four-year longitudinal study
- J.F. Salgado, and S. Moscoso Meta-analysis of interrater reliability of job performance ratings in validity studies of personnel selection Perceptual and Motor Skills 83 1996 1195 1201
- J.F. Salgado, and G. Tauriz The five-factor model, forced-choice personality inventories and performance: A comprehensive meta-analysis of academic and occupational validity studies European Journal of Work and Organizational Psychology 23 2014 3 30
- F.L. Schmidt, and J.E. Hunter Measurement error in psychological research: Lessons from 26 research scenarios Psychological Methods 1 1996 199 223
- F.L. Schmidt, and J.E. Hunter The validity and utility of selection methods in personnel psychology: Practical and theoretical implications of 85 years of research findings Psychological Bulletin 124 1998 262 274
- F.L. Schmidt, H. Le, and I. Remus Beyond Alpha: an empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs Psychological Methods 8 2003 206 224
- N. Schmitt, and R. Klimoski Research methods in human resources management 1991 South-Western Publishing Co Cincinnati, OH
- B. Schneider, and N. Schmitt Staffing organizations 1986 Scott, Foresman Glenview, IL
- R.L. Thorndike Personnel selection. Test and measurement techniques 1949 Wiley New York, NY
- C. Viswesvaran, D. Ones, and F.L. Schmidt Comparative analysis of the reliability of job performance ratings Journal of Applied Psychology 81 1996 557 574
- C. Viswesvaran, F.L. Schmidt, and D.S. Ones The moderating influence of job performance dimensions on convergence of supervisor and peer ratings of job performance: Unconfounding construct-level convergence and rating difficulty Journal of Applied Psychology 87 2002 345 354