Handling Observations with Low Interrater Agreement Values

Gensheng Liu, Mohammad M. Amini, Emin Babakus, Marla B. Royne Stafford

.
Department of Marketing & Supply Chain Management, University of Memphis, Memphis, USA.

DOI: 10.4236/jasmi.2011.12002                

Abstract

Considerable research has been conducted on how interrater agreement (IRA) should be established before data can be aggregated from the individual rater level to the organization level in survey research. However, little is known about how researchers should treat the observations with low IRA values fail to meet the suggested standard. We seek to answer this question by investigating the impact of two factors (the relationship strength and the overall IRA level of a sample) on the IRA decision. Using both real data from a service industry and simulated data, we find that both factors affect whether a researcher should include or exclude observations with low IRA values. Based on the results, practical guidelines on when to use the entire sample are offered.

Keywords

Interrater Agreement, Aggregation, Survey Research

Conflicts of Interest

The authors declare no conflicts of interest.

References

[1]	T. D. Jick, “Mixing Qualitative and Quantitative Methods: Triangulation in Action,” Administrative Science Quarterly, Vol. 24, No. 4, 1979, pp. 602-611. doi:10.2307/2392366
[2]	C. C. Snow and D. C. Hambrick, “Measuring Organizational Strategies: Some Theoretical and Methodological Problems,” Academy of Management Review, Vol. 5, No. 4, 1980, pp. 527-538.
[3]	J. M. Le Breton and J. L. Senter, “Answers to 20 Questions about Interrater Reliability and Interrater Agreement,” Organizational Research Methods, Vol. 11, No. 4, 2008, pp. 815-852. doi:10.1177/1094428106296642
[4]	L. R. James, R. G. Demaree and G. Wolf, “rwg: An Assessment of Within-Group Interrater Agreement,” Journal of Applied Psychology, Vol. 78, No. 2, 1993, pp. 306-309. doi:10.1037/0021-9010.78.2.306
[5]	L. R. James, R. G. Demaree and G. Wolf, “Estimating Within-Group Interrater Reliability with and without Responses Bias,” Journal of Applied Psychology, Vol. 69, No. 1, 1984, pp. 85-98. doi:10.1037/0021-9010.69.1.85
[6]	J. M. Le Breton, J. R. D. Burgess, R. B. Kaiser, E. K. Atchley and L. R. James, “The Restriction of Variance Hypothesis and Interrater Reliability and Agreement: Are Ratings from Multiple Sources Really Dissimilar?” Organizational Research Methods, Vol. 6, No. 1, 2003, pp. 80-128.
[7]	J. M. George, “Personality, Affect, and Behavior in Groups,” Journal of Applied Psychology, Vol. 75, No. 2, 1990, pp. 107-116. doi:10.1037/0021-9010.75.2.107
[8]	J. M. Le Breton, L. R. James and M. K. Lindell, “Recent Issues Regarding rWG, r*WG, rWG(J), and r*WG(J),” Organizational Research Methods, Vol. 8, No. 1, 2005, pp. 128-138. doi:10.1177/1094428104272181
[9]	S. W. Kozlowski and K. Hattrup, “A Disagreement about Within-Group Agreement: Disentangling Issues of Consistency versus Consensus,” Journal of Applied Psychology, Vol. 77, No. 2, 1992, pp. 161-167. doi:10.1037/0021-9010.77.2.161
[10]	M. K. Lindell and C. J. Brandt, “Assessing Interrater Agreement on the Job Relevance of a Test: A Comparison of the CVI, T, rWG(J), and r*WG(J) Indexes,” Journal of Applied Psychology, Vol. 84, No. 4, 1999, pp. 640-647. doi:10.1037/0021-9010.84.4.640
[11]	J. M. Charnes and C. Schriesheim, “Estimation of Quantiles for the Sampling Distribution of the rWG Within-Group Agreement Index,” Educational and Psychological Measurement, Vol. 55, No. 1995, 1995, pp. 435-437.
[12]	A. Cohen, E. Doveh and U. Eick, “Statistical Properties of the rWG(J) Index of Agreement,” Psychological Methods, Vol. 6, No. 3, 2001, pp. 297-310. doi:10.1037/1082-989X.6.3.297
[13]	S. Boerner, V. Moser and J. Jobst, “Evaluating Cultural Industries: Investigating Visitors’ Satisfaction in Theatres,” The Service Industries Journal, Vol. 31, No. 6, 2011, pp. 877-895. doi:10.1080/02642060902960792
[14]	J. Shaw, D., N. Gupta and J. E. Delery, “Congruence between Technology and Compensation Systems: Implications for Strategy Implementation,” Strategic Management Journal, Vol. 22, No. 4, 2001, pp. 379-386. doi:10.1002/smj.165
[15]	K. Tasa and G. Whyte, “Collective Efficacy and Vigilant Problem Solving in Group Decision Making: A Non-Linear Model,” Organizational Behavior and Human Decision Processes, Vol. 96, No. 2, 2005, pp. 119-129. doi:10.1016/j.obhdp.2005.01.002
[16]	G. J. Liu, R. Shah and R. G. Schroeder, “Linking Work Design to Mass Customization: A Sociotechnical Systems Perspective,” Decision Sciences, Vol. 37, No. 4, 2006, pp. 519-545. doi:10.1111/j.1540-5414.2006.00137.x
[17]	J. W. Dean and S. A. Snell, “Integrated Manufacturing and Job Design: Moderating Effects of Organizational Inertia,” Academy of Management Journal, Vol. 34, No. 4, 1991, pp. 776-804. doi:10.2307/256389
[18]	S. A. Snell and J. W. Dean, “Integrated Manufacturing and Human Resource Management: A Human Capital Perspective,” Academy of Management Journal, Vol. 35, No. 3, 1992, pp. 467-504. doi:10.2307/256484
[19]	D. R. Deeter-Schmelz and R. P. Ramsey, “An Investigation of Team Information Processing in Service Teams: Exploring the Link Between Teams and Customers,” Journal of the Academy of Marketing Science, Vol. 31, No. 4, 2003, pp. 409-424. doi:10.1177/0092070303255382
[20]	A. M. Susskind, K. M. Kacmar and C. P. Borchgrevink, “Customer Service Providers’ Attitudes Relating to Customer Service and Customer Satisfaction in the Customer-Server Exchange,” Journal of Applied Psychology, Vol. 88, No. 1, 2003, pp. 179-187. doi:10.1037/0021-9010.88.1.179
[21]	B. Schneider, M. G. Ehrhart, D. M. Mayer, J. L. Saltz and K. Niles-Jolly, “Understanding Organization-Customer Links in Service Settings,” Academy of Management Journal, Vol. 48, No. 6, 2005, pp. 1017-1032. doi:10.5465/AMJ.2005.19573107
[22]	J. J. Cronin Jr., M. K. Brady and G. T. M. Hult, “Assessing the Effects of Quality, Value, and Customer Satisfaction on Consumer Behavioral Intentions in Service Environments,” Journal of Retailing, Vol. 76, No. 2, 2000, pp. 193-216. doi:10.1016/S0022-4359(00)00028-2
[23]	M. D. Hartline and K. C. Jones, “Employee Performance Cues in a Hotel Service Environment: Influence on Perceived Service Quality, Value, and Word-of-Mouth Intentions,” Journal of Business Research, Vol. 35, No. 3, 1996, pp. 207-215. doi:10.1016/0148-2963(95)00126-3
[24]	S. Anderson, L. K. Pearo and S. K. Widener, “Drivers of Service Satisfaction: Linking Customer Satisfaction to the Service Concept and Customer Characteristics,” Journal of Service Research, Vol. 10, No. 4, 2008, pp. 365-381. doi:10.1177/1094670508314575
[25]	T. Hennig-Thurau, K. P. Gwinner and D. D. Gremler, “Understanding Relationship Marketing Outcomes: An Integration of Relational Benefits and Relationship Quality,” Journal of Service Research, Vol. 4, No. 3, 2002, pp. 230-247. doi:10.1177/1094670502004003006
[26]	J. F. Hair, W. C. Black, B. J. Babin and R. E. Anderson, “Multivariate Data Analysis,” 7th Edition, Prentice-Hall, Inc., Upper Saddle River, 2010.
[27]	L. S. Aiken and S. G. West, “Multiple Regression: Testing and Interpreting Interactions,” Sage, Newbury Park, 1991.
[28]	J. Jaccard and R. Turrisi, “Interaction Effects in Multiple Regression,” Sage University Papers Series on Quantitative Applications in the Social Sciences, 2nd Edition, Sage, Thousand Oaks, 2003.
[29]	C. Fornell, M. D. Johnson, E. W. Anderson, C. Jaesung and B. E. Bryant, “The American Customer Satisfaction Index: Nature, Purpose, and Findings,” Journal of Marketing, Vol. 60, No. 4, 1996, pp. 7-18. doi:10.2307/1251898
[30]	M. K. Brady and J. J. Cronin Jr., “Customer Orientation: Effects on Customer Service Perceptions and Outcome Behaviors,” Journal of Service Research, Vol. 3, No. 3, 2001, pp. 241-251. doi:10.1177/109467050133005