Quasi-ipsative Forced-Choice Personality Inventories and the Control of FakingThe Biasing Effects of Transient Error

  1. Alexandra Martínez 1
  2. Jesús F. Salgado 1
  3. Mario Lado 1
  1. 1 University of Santiago de Compostela, Spain
Journal:
Revista de psicología del trabajo y de las organizaciones = Journal of work and organizational psychology

ISSN: 1576-5962

Year of publication: 2022

Volume: 38

Issue: 3

Pages: 241-248

Type: Article

DOI: 10.5093/JWOP2022A16 DIALNET GOOGLE SCHOLAR lock_openOpen access editor

More publications in: Revista de psicología del trabajo y de las organizaciones = Journal of work and organizational psychology

Abstract

To date, experimental research on the effect of faking on personality measures has used two types of designs: within-subject designs and between-subjects designs. None of these designs permit us to control for the effects of transient error on faking. Using a Latin-square design (LSD), the current study examines the effects of faking on the Big Five as assessed by a quasi-ipsative forced-choice (FC) personality inventory. LSD is a type of experimental design that simultaneously permits us to control for between-subject differences, within-subject variability, and transient error. The sample consisted of 246 participants (four experimental groups, assessed twice, 2-3 week interval). The results showed that (1) faking effect size can be largely attributed to transient error and (2) the quasi-ipsative FC format shows great resistance to faking behavior. The average effect size (Cohen’s d) for the Big Five was 0.21, 0.12, and 0.09 for observed faking, transient error, and true faking, respectively. On average, 62% of observed faking effect size can be attributed to transient error. To conclude, we discuss the implications of these findings for the research and practice of personnel selection.

Bibliographic References

  • Abad, F. J., Kreitchmann, R. S., Sorrel, M. A., Nájera, P., García-Garzón, E., Garrido, L. E., & Jiménez, M. (2022). Building adaptive forced choice tests “On the Fly” for personality measurement. Psychologist Papers, 43(1), 29-35. https://doi.org/10.23923/pap.psicol.2982
  • Adair, C. (2014). Interventions for addressing faking on personality assessments for employee selection: A meta-analysis (Unpublished doctoral dissertation). DePaul University.
  • Baron, H. (1996). Strengths and limitation of ipsative measurement. Journal of Occupational and Organizational Psychology, 69(1), 49-56. https://doi.org/10.1111/j.2044-8325.1996.tb00599.x
  • Bartram, D. (1996). The relationship between ipsatized and normative measures of personality. Journal of Occupational and Organizational Psychology, 69(1), 25-39. https://doi.org/10.1111/j.2044-8325.1996.tb00597.x
  • Becker, G. (2000). How important is transient error in estimating reliability? Going beyond simulation studies. Psychological Methods, 5(3), 370-379. https://doi.org/10.1037/1082-989X.5.3.370
  • Birkeland, S. A., Manson, T. M., Kisamore, J. L., Brannick, M. T., & Smith, M. A. (2006). A meta-analytic investigation of job applicant faking on personality measures. International Journal of Selection and Assessment, 14(4), 317-335. https://doi.org/10.1111/j.1468-2389.2006.00354.x
  • Brown, A., & Maydeu-Olivares, A. (2013). How IRT can solve problems of ipsative data in forced-choice questionnaires. Psychological Methods, 18(1), 36-52. https://doi.org/10.1037/a0030641
  • Cao, M., & Drasgow, F. (2019). Does forcing reduce faking? A meta-analytic review of forced-choice personality measures in high-stakes situations. Journal of Applied Psychology, 104(11), 1347-1368. https://doi.org/10.1037/apl0000414
  • Christiansen, N. D., Burns, G. N., & Montgomery, G. E. (2005). Reconsidering forced-choice item formats for applicant personality assessment. Human Performance, 18(3), 267-307. https://doi.org/10.1207/s15327043hup1803_4
  • Clemans, W. V. (1966). An analytical and empirical examination of some properties of ipsative measures. Psychometric Monographs, 14, 1-56.
  • Cochran, W. G., & Cox, G. M. (1978). Experimental designs. John Wiley & Sons, Inc.
  • Converse, P. D., Oswald, F. L., Imus, A., Hedricks, C., Roy, R., & Butera, H. (2006). Forcing choices in personality measurement. In R. L. Griffith & M. H. Peterson (Eds.), A closer examination of applicant faking behavior (pp. 263-282). Information Age.
  • Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Houghton Mifflin.
  • Dilchert, S., & Ones, D. S. (2012). Measuring and improving environmental sustainability. In S. E. Jackson, D. S. Ones, & S. Dilchert (Eds.), Managing HR for environmental sustainability (pp. 187-221). Jossey-Bass/Wiley.
  • Donovan, J. J., Dwight, S. A., & Schneider, D. (2014). The impact of applicant faking on selection measures, hiring decisions, and employee performance. Journal of Business and Psychology, 29, 479-493. https://doi.org/10.1007/s10869-013-9318-5
  • Douglas, E. F., MacDaniel, M. A., & Snell, A. F. (1996). The validity of non-cognitive measures decays when applicants fake. Academy of Management Proceedings, 1996(1), 127-131. https://doi.org/10.5465/ambpp.1996.4979062
  • Dunlap, W, P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods, 1(2), 170-177. https://doi.org/10.1037/1082-989X.1.2.170
  • Ellingson, J. E., Sackett, P. R., & Connelly, B. S. (2007). Personality assessment across selection and development contexts: Insights into response distortion. Journal of Applied Psychology, 92(2), 386-395. https://doi.org/10.1037/0021-9010.92.2.386
  • Fisher, P. A., Robie, C., Christiansen, N. D., Speer, A. B., & Schneider, L. (2019). Criterion-related validity of forced-choice personality measures: A cautionary note regarding Thurstonian IRT versus classical test theory scoring. Personnel Assessment and Decisions, 5(1), 49-61. https://doi.org/10.25035/pad.2019.01.003
  • Gao, L. (2005). Latin squares in experimental design. Unpublished manuscript. Michigan State University.
  • García-Izquierdo, A. L., Ramos-Villagrasa, P. J., & Lubiana, M. A. (2020). Developing biadata for public selection purposes: A comparison between fuzzy logic and traditional methods. Journal of Work and Organizational Psychology, 36(3) 231-242. https://doi.org/10.5093/jwop2020a22
  • Grant, D. A. (1948). The latin square principle in the design and analysis of psychological experiments. Psychological Bulletin, 45(5), 427-442. https://doi.org/10.1037/h0053912
  • Griffith, R. L., Chmielowski, T., & Yoshita, Y. (2007). Do applicants fake? An examination of the frequency of applicant faking behavior. Personnel Review, 36(3), 341-355. https://doi.org/10.1108/00483480710731310
  • Griffith, R. L., & Converse, P. D. (2012). The rules of evidence and the prevalence of applicant faking. In M. Ziegler, C. MacCann, & R. D. Roberts (Eds.), New perspectives on faking in personality assessment (pp. 34-52). Oxford University Press.
  • Heggestad, E. D., George, E., & Reeve, C. L. (2006). Transient error in personality scores: Considering honest and faked responses. Personality and Individual Differences, 40(6), 1201-1211. https://doi.org/10.1016/j.paid.2005.10.014
  • Hicks, L. E. (1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74(3), 167-184. https://doi.org/10.1037/h0029780
  • Hooper, A. C. (2007). Self-presentation on personality measures in lab and field settings: A meta-analysis (Unpublished doctoral dissertation). University of Minnesota.
  • Horn, J. L. (1971). Motivation and dynamic calculus concepts from multivariate experiment. In R. B. Cattell (Ed.), Handbook of multivariate experimental psychology (2nd printing, pp. 611-641). Tand McNally.
  • Hough, L. M., Eaton, N. K., Dunnette, M. D., Kamp, J. D., & McCloy, R. A. (1990). Criterion-related validities of personality constructs and the effect of response distortion on those validities. Journal of Applied Psychology, 75(5), 581-595. https://doi.org/10.1037/0021-9010.75.5.581
  • Jackson, D. N., Wroblewski, V. R., & Ashton, M. C. (2000). The impact of faking on employment tests: Does forced choice offer a solution? Human Performance, 13(4), 371-388. https://doi.org/10.1207/S15327043HUP1304_3
  • Kirk, R. E. (2013). Experimental design: Procedures for the behavioral sciences (4th Edition). Sage.
  • Komar, S., Brown, D. J., Komar, J. A., & Robie, C. (2008). Faking and the validity of conscientiousness: A Monte Carlo investigation. Journal of Applied Psychology, 93(1), 140-154. https://doi.org/10.1037/0021-9010.93.1.140
  • MacCann, C., Ziegler, M., & Roberts, R. D. (2012). Faking in personality assessment: Reflections and recommendations. In M. Ziegler, C. MacCann, & R. D. Roberts (Eds.), New perspectives on faking in personality assessment (pp. 309-329). Oxford University Press.
  • Martínez, A. (2019). Evaluación empírica de un modelo teórico de los efectos del faking sobre las medidas de personalidad ocupacional [Empirical assessment of a theoretical model of the effects of faking on the scores of occupational personality] (Unpublished doctoral dissertation). University of Santiago de Compostela.
  • Martínez, A., Moscoso, S., & Lado, M. (2021). Effects of faking on the predictive validity of a quasi-ipsative forced-choice personality inventory: Implications for sustainable personnel selection. Sustainability, 13(8), 4398. https://doi.org/10.3390/su13084398
  • Martínez, A., Moscoso, S., & Lado, M. (2021). Faking effects on the factor structure of a quasi-ipsative forced-choice personality inventory. Journal of Work and Organizational Psychology, 37(1), 1-10. https://doi.org/jwop2021a7
  • Martínez, A., & Salgado, J. F. (2021). A meta-analysis of the faking resistance of forced-choice personality inventories. Frontiers in Psychology, 12, Article 732241. https://doi.org/10.3389/fpsyg.2021.732241
  • Meade, A. W. (2004). Psychometric problems and issues involved with creating and using ipsative measures for selection. Journal of Occupational and Organizational Psychology, 77(4), 531-552. https://doi.org/10.1348/0963179042596504
  • Mesmer-Magnus, J., & Viswesvaran, C. (2006). Assessing response distortion in personality tests: A review pf research designs and analytic strategies. In R. L. Griffith & M. H. Peterson (Eds.), A closer examination of applicant faking behavior (pp. 85-114). Information Age.
  • Morgeson, F. P., Campion, M. A., Dipboye, R. L., Hollenbeck, J. R., Murphy, K., & Schmitt, N. (2007). Reconsidering the use of personality tests in personnel selection contexts. Personnel Psychology, 60(3), 683-729. https://doi.org/10.1111/j.1744-6570.2007.00089.x
  • Morillo, D., Abad, F. J., Kreitchmann, R. S., Leenen, I., Hontangas, P., & Ponsoda, V. (2019). The journey from Likert to forced-choice questionnaires: Evidence of the invariance of item parameters. Journal of Work and Organizational Psychology, 35(2), 75-83. https://doi.org/10.5093/jwop2019a11
  • Murphy, K. R. (2005). Why don’t measures of broad dimensions of personality perform better as predictors of job performance? Human Performance, 18(4), 343-357. https://doi.org/10.1207/s15327043hup1804_2
  • Ones, D. S., Viswesvaran, C., & Reiss, A. D. (1996). Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81(6), 660-679. https://doi.org/10.1037/0021-9010.81.6.660
  • Otero, I., Cuadrado, D., & Martínez, A. (2020). Convergent and predictive validity of the Big Five Factors assessed with single stimulus and quasi-ipsative questionnaires. Journal of Work and Organizational Psychology, 36(3), 215-222. https://doi.org/10.5093/jwop2020a17
  • Rothstein, M. G., & Goffin, R. D. (2006). The use of personality measures in personnel selection: What does current research support? Human Resource Management Review, 16(2), 155-180. https://doi.org/10.1016/j.hrmr.2006.03.004
  • Salgado, J. F. (2014). Reliability, construct, and criterion validity of the Quasi-Ipsative Personality Inventory (QI5F/Tri). Unpublished manuscript. University of Santiago de Compostela.
  • Salgado, J. F. (2016). A theoretical model of psychometric effects of faking on assessment procedures: Empirical findings and implications for personality at work. International Journal of Selection and Assessment, 24(3), 209-228. https://doi.org/10.1111/ijsa.12142
  • Salgado, J. F. (2017). Moderator effects of job complexity on the validity of forced-choice personality inventories for predicting job performance. Journal of Work and Organizational psychology, 33(3), 229-238. https://doi.org/10.1016/j.rpto.2017.07.001
  • Salgado, J. F., Anderson, N., & Táuriz, G. (2015). The validity of ipsative and quasi-ipsative forced-choice personality inventories for different occupational groups: A comprehensive meta-analysis. Journal of Occupational and Organizational Psychology, 88(4), 797-834. https://doi.org/10.1111/joop.12098
  • Salgado, J. F., & Lado, M. (2018). Faking resistance of a quasi-ipsative forced-choice personality inventory without algebraic dependence. Journal of Work and Organizational Psychology, 34(3), 213-216. https://doi.org/10.5093/jwop2018a23
  • Salgado, J. F., & Táuriz, G. (2014). The Five-Factor Model, forced-choice personality inventories and performance: A comprehensive meta-analysis of academic and occupational validity studies. European Journal of Work and Organizational Psychology, 23(1), 3-30. https://doi.org/10.1080/1359432X.2012.716198
  • Schmidt, F. L., Le, H., & Ilies, R. (2003). Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. Psychological Methods, 8(2), 206-224. https://doi.org/10.1037/1082-989X.8.2.206
  • Smith, D. B., & McDaniel, M. (2012). Questioning old assumptions: Faking and the personality-performance relationship. In M. Ziegler, C. MacCann, & R. D. Roberts (Eds.), New perspectives on faking in personality assessment (pp. 53-70). Oxford University Press.
  • Vautier, S., & Jmel, S. (2003), Transient error or specificity? An alternative to the staggered equivalent split-half procedure. Psychological Methods, 8(2), 225-238. https://doi.org/10.1037/1082-989X.8.2.225
  • Viswesvaran, C., & Ones, D. S. (1999). Meta-analyses of fakability estimates: Implications for personality measurement. Educational and Psychological Measurement, 59(2), 197-210. https://doi.org/10.1177/00131649921969802
  • Zickar, M. J., & Gibby, R. E. (2006). A history of faking and socially desirable responding on personality tests. In R. L. Griffith & M. H. Peterson (Eds.), A closer examination of applicant faking behavior (pp. 21-42). Information Age.
  • Ziegler, M., MacCann, C., & Roberts, R. D. (2012). Faking: knowns, unknowns, and points of contention. In M. Ziegler, C. MacCann, & R. D. Roberts (Eds.), New perspectives on faking in personality assessment (pp. 3-16). Oxford University Press.
Data source: Dialnet