Skip to main content
SearchLoginLogin or Signup

The Measurement of Organizational Justice Matters: A Research Note

This experiment addressed whether, when administering a survey to police managers, it is best to measure organizational justice using attitudinal questions tapping into perceived importance or behavioral self-reports.

Published onJul 16, 2020
The Measurement of Organizational Justice Matters: A Research Note


Objectives: This experiment addressed whether, when administering a survey to police managers, it is best to measure organizational justice using attitudinal questions tapping into perceived importance or behavioral self-reports. Methods: We administered a survey to a national probability sample of police executives using a split-ballot experimental design, where respondents randomly received items measuring either (a) the perceived importance of organizational justice or (b) the self-reported usage of organizational justice. Results: Perceived importance of organizational justice was not significantly associated with the perceived quality of relationships with subordinates. However, actual usage of organizational justice was, increasing the perceived quality of relationships (b = .554, p < .001). Conclusions: Our results suggest that: 1) the measurement of organizational justice matters, and 2) mangers who believe organizational justice is important still sometimes fail to use it. Future research should measure the concept using behavioral reports and should seek to explain this importance-usage gap.

Keywords: organizational justice; measurement; police; management; survey; questionnaire


In any work setting, supervisors strive to get the best outcomes from their employees through a range of management strategies. The business management and social psychology literatures have identified organizational justice as a key management strategy for achieving beneficial work-related outcomes among employees (Colquitt, 2001). Subordinates who perceive greater organizational justice (i.e., transparency, fair distribution of rewards and disciplinary actions, allowing employees to voice their opinions and concerns, and respectful treatment) exhibit greater productivity, a stronger commitment to organizational goals, and lower turnover intentions (Ambrose & Schminke, 2009; Cohen-Charash & Spector, 2001; McFarlin & Sweeney, 1992), and are less likely to engage in counterproductive work behaviors (e.g., cyber-loafing during work hours; Bechtoldt, Welk, Zapf, & Hartig, 2007; Fox, Spector, & Miles, 2001; Lim, 2002). More broadly, employees who have experienced organizational justice express greater job satisfaction than their counterparts (Ambrose & Schminke, 2009; Sweeney & McFarlin, 1993).

Researchers have directed attention to the role of organizational justice in the policing context in recent years. Police officers work in an organizational environment that is similar in many ways to business contexts. Line-level officers, as subordinates, are given directives and goals from their immediate supervisors and command staff (i.e., managers). Research reveals that police employees expect to be treated with organizational justice in the same was as employees in other work settings. Officers who believe they have been treated fairly by their supervisors are more likely to support community-oriented policing and procedurally-fair treatment of citizens (Myhill & Bradford, 2013; Tankebe, 2014; Trinkner, Tyler, & Goff, 2016; Wolfe & Nix, 2016), have more favorable attitudes toward the public (Myhill & Bradford, 2013), and demonstrate less cynicism toward the job (Bradford & Quinton, 2014; Bradford, Quinton, Myhill, & Porter, 2014). The experience of organizational justice also appears to cultivate greater job satisfaction (Donner, Maskaly, Fridell, & Jennings, 2015; Rosenbaum & McCarty, 2017), organizational identification, internalization of organizational goals, supervisor trust, and self-legitimacy among line-level officers (Bradford et al., 2014; Bradford & Quinton, 2014; Carr & Maxwell, 2017; Haas, Van Craen, Skogan, & Fleitas, 2015; Nix & Wolfe, 2016; Van Craen & Skogan, 2017a, 2017b; Wolfe & Nix, 2017). Further, such officers are significantly less likely to engage in misconduct (Myhill & Bradford, 2013; Tankebe, 2014; Trinkner et al., 2016; Wolfe & Piquero, 2011). A study of federal law enforcement agents showed that one of the reasons organizational justice is important is that it reduces employee uncertainty about future opportunities within their agency (e.g., promotions; Wolfe et al., 2018).

Nearly all this research has explored police employees’ evaluations of their management. Much less is known about police managers’ own orientations toward and use of organizational justice. The importance of this issue has not been lost in the broader organizational behavior literature. In recent years, several studies have examined the factors that predict managers’ “justice rule adherence”—the extent to which they treat subordinates in an organizationally fair manner (Brockner et al., 2015; Matta, Scott, Colquitt, & Koopman, 2017; Scott, Colquitt, & Zapata-Phelan, 2007). For example, research demonstrates that supervisors (either real supervisors or those in role-playing experiments) are more willing to treat their employees fairly when they believe their subordinates have a stronger need for a feeling of belongingness (Cornelis, Hiel, Cremer, & Mayer, 2013) and when their employees are more assertive (Korsgaard, Roberson, & Rymph, 1998). Supervisors’ level of self-control also appears to influence the level of fairness they are willing to exercise with their subordinates (Matta et al., 2017; Whiteside & Barclay, 2018). Managers (again, real or role-playing managers) who treat subordinates with organizational justice are more likely to be evaluated as fair by those subordinates (Koopman, Matta, Scott, & Conlon, 2015; Zapata, Olsen, & Martins, 2013).

While it appears that assessing supervisors’ orientations toward the use of organizational justice is worthwhile, the research literature provides mixed results depending in part on the operationalization of justice. Whiteside and Barclay (2018) showed that undergraduate students subjected to a self-regulation depletion task were less likely to believe that they treated subordinates in a role-playing exercise with justice. However, independent coding of the participants’ use of fairness revealed that those subjected to the self-regulation depletion task overestimated the amount of fairness they actually used with subordinates. Korsgaard and colleagues (1998) similarly demonstrated that undergraduate management students’ self-reported use of interactional justice (one component of organizational justice) during a role-playing experiment was higher than independent coding of the level of fairness displayed in their actual behavior. Cornelis et al. (2013) used samples of Flemish supervisors and undergraduate students (in role-playing experiments) to show that those with higher levels of empathy were more likely than their less empathic counterparts to use procedural justice (one component of organizational justice) with subordinates. Interestingly, however, the relationship between empathy and procedural justice was less pronounced among the supervisor sample than the student samples. This result could be due to differences in the samples themselves. Or, it is possible that variation in the measurement of justice impacted the results. Within the sample of supervisors, Cornelis and colleagues asked respondents to self-report their use of procedural justice with specific subordinates (e.g., when dealing with this subordinate “I allowed them to voice their opinion”). Within the student samples, however, they asked participants the extent with which they wanted to treat their subordinates with fairness (e.g., “I want to know this group member’s opinion”). Taken together, these findings suggest that different results may be obtained if a survey measures supervisors’ orientations toward the use of justice compared to self-reported or independently coded justice behavior.

We are only aware of one study that has examined police supervisors’ orientations toward organizational justice. Wolfe, Nix, and Campbell (2018) recently analyzed self-report survey data from 211 command-level officers who attended a training program in the southern United States. They found that police managers with higher levels of self-control placed greater importance on exhibiting organizational justice when interacting with their subordinates. Wolfe et al. captured managers’ attitudes concerning the importance of using organizational justice while interacting with subordinate officers (e.g., How important is that… “I consider my employees’ viewpoints”). The scale had strong psychometric properties and was associated with key theoretical variables in the expected directions.

Organizational justice-related policing research has grown tremendously over the past decade and revealed that many beneficial outcomes stem from officers believing they have been treated with fairness from their supervisors. In our view, one area of research that deserves more attention is police managers’ orientations toward or actual use of organizational justice. The organizational behavior literature supports the potential importance of doing so but also provides methodological caution when moving forward. One way to capture police managers’ organizational justice focuses on perceived importance (as Wolfe and colleagues did) and another on actual usage (through manager self-report, employee evaluations, or independent coding of behavior). To provide guidance for future research on police managers’ organizational justice, it is necessary to test the extent to which operationalization of the concept impacts study results. The organizational behavior research reviewed above suggests that measuring mangers’ perceived importance of using organizational justice compared to their self-reported use of justice may yield different results. Furthermore, although Azjen’s (1991) work suggests that behavioral intentions are highly correlated with actual behavior, work by scholars such as Nobel Laureate Richard Thaler (2005) demonstrate that our self-reported attitudes do not always neatly predict our actual behavior.

Current Focus

Given the increasing interest in organizational justice research within our field and the need for more research on supervisors’ views of the managerial technique, we need sound evidence to base our operationalization strategies. Furthermore, given the potential impracticality of using a social systematic observation (SSO) methodology to code managers’ use of organizational justice, most research in the future will likely rely on survey-based designs and self-reports. Toward this end, the current study sought to address a simple question: when administering a survey to police managers, is it best to measure organizational justice using attitudinal questions tapping into perceived importance or self-reported behavioral measures? Next, we discuss our methodology, present our results, and conclude with a discussion of the practical research and theoretical implications of the findings.



Data for this study come from a survey administered to a national probability sample of police executives in the winter of 2018. We used the National Directory of Law Enforcement Administrators to obtain the mailing address for 12,039 municipal police departments. Using a sampling strategy similar to that used by the Bureau of Justice Statistics for its Law Enforcement Management and Administrative Statistics (LEMAS) survey (see also Strom & Hickman, 2010), we placed each agency into one of four stratum based on the number of sworn officers they employed (i.e., 0 to 24, 25 to 49, 50 to 99, and 100 or more). A random sample of 624 agencies was drawn from each stratum which provided a stratified random sample of 2,496 departments.

We used a modified Dillman method to elicit participation in the study (Dillman et al., 2009). First, in February 2018, we mailed a notification postcard to each agency in our sample. The postcards were addressed to the chief executive at each department and notified them of an upcoming mail questionnaire. A link to an online version of the survey was also available on the postcard. We requested that the chief executive complete the survey. If s/he was not able to do so, we requested that another high-ranking, command-level officer complete the survey. One week after the postcard mailing, we sent a survey packet to each chief executive in our sample (containing a questionnaire, cover letter, and self-addressed, postage-paid return envelope). The online survey option was again provided if respondents preferred to complete the questionnaire in this manner. Finally, two weeks later we mailed another survey and reminder letter to agencies that had not yet responded. We received 675 surveys (369 by mail and 306 online), resulting in a 27% response rate.1 While the response rate is lower than we hoped, research reveals that survey response rate and non-response bias are only weakly correlated (Peytcheva, 2013; Pickett et al., 2018).


Experimental Manipulation

All respondents were randomly assigned to Group A or Group B, asked to consider their interactions with employees in their agency, and presented with six questions pertaining to organizational justice (Colquitt 2001; Wolfe et al. 2018). On the one hand, Group A’s questions elicited respondents’ perceived importance of using organizationally fair management tactics with their employees. Specifically, they were asked to indicate how important (1=very unimportant to 5=very important) it is to: (1) consider employees’ viewpoints, (2) treat employees with kindness and consideration, (3) clearly explain the reasons for your decisions, (4) clearly explain the reasons your agency makes policy changes, (5) treat employees with respect, and (6) make decisions that have the agency’s best interest in mind. On the other hand, Group B was asked to self-report how often (1=never, 2=rarely, 3=sometimes, 4=often, 5=always) they engaged in each of these same six behaviors. Factor analysis indicated that for each group, responses to the six questions loaded onto a single factor with acceptable loadings (Group A eigenvalue=4.614, loadings >.85; Group B eigenvalue=2.053, loadings >.47). As such, we averaged each group’s responses to the six items to generate mean scales (Group A Cronbach’s alpha=.95; Group B Cronbach’s alpha=.74). Table 1 provides descriptive statistics for these and each of the other variables used in the analyses that follow.

Outcome Variable

Our dependent variable was a scale reflecting respondents’ perceived quality of relationships with other officers in their agency. Research demonstrates that supervisors who support the use of organizational justice or engage in fair managerial practices are more likely to cultivate higher quality interpersonal relationships with their subordinates (Connell, Ferres, &Travaglione, 2003; Koopman et al., 2015; Zapata et al., 2013). Accordingly, assessing managers’ views on the quality of relationships they have with subordinates allows use to test the predictive validity of each operationalization of organizational justice. All respondents reported their level of agreement (1=strongly agree to 5=strongly disagree) with the following statements: (1) I have a good working relationship with the officers in my department, (2) I feel that officers in this department trust me, (3) I feel supported by the officers in my department, (4) Officers in this department treat me with respect, (5) My views about what is right and wrong in police work are similar to the views of other officers in the department, and (6) Other officers in the department come to me for advice. Exploratory factor analysis indicated that for each group, the items loaded onto a single factor (Group A eigenvalue =2.90, loadings >.52; Group B eigenvalue =3.14, loadings >.56). As such, we reverse coded the items so that higher scores reflected greater agreement and averaged responses to create mean scales (Group A Cronbach’s alpha =.84; Group B Cronbach’s alpha =.86).


We controlled for respondents’ years of experience in law enforcement with a continuous variable. We also used dummy variables to control for education (1 = Master’s degree or higher), race/gender (1 = White male), the size of the respondent’s agency (1 = Large agency, i.e., 100 or more sworn officers), and rank (1 = Chief).

Analytic Strategy

To test whether the wording of our organizational justice measures is associated with our outcome, perceived quality of relationships with employees, we ran ordinary least squares (OLS) regression models using a split ballot design. OLS is appropriate here as our outcome is a continuous variable that approximates a normal distribution within each group.2 Because we oversampled large agencies (i.e., those with 100+ sworn officers), we applied inverse-probability weights to our models to ensure more representative estimates.


Table 1 indicates that the group means on organizational justice were virtually identical (Group A = 4.525, Group B = 4.527). In other words, the randomized prompt and response options did not have a noticeable impact on the distribution of responses on a 1 to 5 Likert scale. But importantly, this tells us nothing about whether the two scales are in fact measuring the same construct of interest – organizationally fair managerial behavior. Table 2 presents the results of a series of OLS regression models predicting respondents’ perceived quality of relationships with their employees. Models 1 and 2 indicate that perceived importance of using organizational justice is not significantly associated with the outcome. On the contrary, Models 3 and 4 demonstrate that self-reported use of organizational justice is strongly associated with the outcome. Even after controlling for experience, education, race/gender, agency size, and rank, executives who report using organizational justice more frequently during interactions with their employees are significantly more likely to report higher quality relationships with them (b = .554, p < .001). The Paternoster et al. (1998) coefficient-equality test reveals that there is a highly significant difference (Z = 5.758, p < .001) between the organizational justice coefficients in the two experimental groups (Model 2 vs. 4). Also of note is the considerable improvement in the explanatory power of Model 4 (R2 = .264) relative to Model 2 (R2 = .022). Collectively, our results suggest that the wording of survey items meant to capture organizational justice matters a great deal.


Although always instructive methodologically, measurement effects are sometimes theoretically illuminating as well. The methodological implication of our findings is straightforward: researchers studying the effects of organizational justice, whether in policing or in other management contexts, should either measure the concept using behavioral reports instead of importance perceptions, or should randomize its measurement (more on this shortly). Our experiment shows that the use of organizational justice, but not its perceived importance, is associated with a key theoretical outcome, high-quality manger-subordinate relationships.

There are several reasons why we may have observed this finding. For starters, police managers may be more apt to agree that it is “important to use” organizational justice than to actually report using fair managerial practices when interacting with their employees. This suggests that the former measurement strategy may be vulnerable to social desirability bias, while measuring self-reported organizational justice may invoke more honest reflection. In this way, operationalization of the concept as perceived importance of using organizational justice may be inadequate for distinguishing police managers in terms of beneficial outcomes such as the perceived quality of relationships they have with their subordinates.

A second explanation of the results suggests the perception that organizational justice is important apparently is not enough to guarantee its use and improve workplace bonds. This possibility raises a critical, new theoretical question, with great policy relevance: why might mangers who believe organizational justice is important fail to use it? Perhaps mangers’ ability to adopt valued management strategies depends on their personal resources, such as their level self-control (Matta et al., 2017; Whiteside & Barclay, 2018; Wolfe et al., 2018). It seems likely that mangers who are impulsive and hot tempered would have difficulty using organizational justice—managing fairly, respectfully, and with interactional turn-taking—especially when faced with workplace adversity. Alternatively, the perceived importance-usage gap may reflect inadequate training. Maybe managers who believe organizational justice is important would be more likely to use it if they only knew how. If so, training on how to put managerial values into practice, or about specific strategies for implementing organizational justice, may be useful.

Future studies should explore these and other theoretical possibilities. More generally, we encourage researchers to devote greater theoretical and empirical attention to the perceived importance-usage gap. It seems crucial to understand why mangers adopt management strategies that differ from the managerial values they themselves believe are important. Obviously, it would also be useful for subsequent studies to examine whether the perceived importance-usage gap emerges in different managerial contexts, and with other outcome variables, like employee job satisfaction or workplace deviance.

We close by emphasizing the importance of experimental measurement studies for understanding interpersonal interactions in the workplace. Even a cursory review of the literature on organizational justice and procedural justice will show that one notable absence is research randomizing the measurement of key concepts. Not only does this increase the risk of mono-operation and mono-method bias (Shadish et al., 2002), but it also risks locking away key theoretical insights, like the perceived importance-usage gap, behind the closed door of measurement invariance. We cannot know that measurement matters unless we test it, and we are unlikely to analyze why it matters without first knowing that it does. For this reason, we suggest future research examining organizational justice use an experimental approach that randomizes the measurement of the concept, in order to examine the causal effects—both main and interactive—of measurement on the outcomes and relationships of theoretical interest.


Ajzen, I. (1991). The theory of planned behavior. Organizational behavior and human decision processes50(2), 179-211.

Ambrose, M. L., & Schminke, M. (2009). The role of overall justice judgments in organizational justice research: A test of mediation. Journal of applied psychology94(2), 491.

Bechtoldt, M. N., Welk, C., Zapf, D., & Hartig, J. (2007). Main and moderating effects of self-control, organizational justice, and emotional labour on counterproductive behaviour at work. European Journal of Work and Organizational Psychology16(4), 479-500.

Bradford, B., & Quinton, P. (2014). Self-legitimacy, police culture and support for democratic policing in an English constabulary. British journal of criminology54(6), 1023-1046.

Bradford, B., Quinton, P., Myhill, A., & Porter, G. (2014). Why do “the law” comply? Procedural justice, group identification and officer motivation in police organizations. European Journal of Criminology, 11(1), 110–131.

Brockner, J., Wiesenfeld, B. M., Siegel, P. A., Bobocel, D. R., & Liu, Z. (2015). Riding the fifth wave: Organizational justice as dependent variable. Research in Organizational Behavior35, 103-121.

Carr, J. D., & Maxwell, S. R. (2018). Police officers’ perceptions of organizational justice and their trust in the public. Police Practice and Research, 19(4), 365–379.

Cohen-Charash, Y., & Spector, P. E. (2001). The role of justice in organizations: A meta-analysis. Organizational behavior and human decision processes86(2), 278-321.

Colquitt, J. A. (2001). On the dimensionality of organizational justice: A construct validation of a measure. Journal of applied psychology86(3), 386.

Connell, J., Ferres, N., & Travaglione, T. (2003). Engendering trust in manager-subordinate relationships: Predictors and outcomes. Personnel Review32(5), 569-587.

Cornelis, I., Van Hiel, A., De Cremer, D., & Mayer, D. M. (2013). When leaders choose to be fair: Follower belongingness needs and leader empathy influences leaders' adherence to procedural fairness rules. Journal of Experimental Social Psychology49(4), 605-613.

Donner, C., Maskaly, J., Fridell, L., & Jennings, W. G. (2015). Policing and procedural justice: A state-of-the-art review. Policing, 38(1), 153–172.

Fox, S., Spector, P. E., & Miles, D. (2001). Counterproductive work behavior (CWB) in response to job stressors and organizational justice: Some mediator and moderator tests for autonomy and emotions. Journal of Vocational Behavior59(3), 291-309.

Haas, N. E., Van Craen, M., Skogan, W. G., & Fleitas, D. M. (2015). Explaining officer compliance: The importance of procedural justice and trust inside a police organization. Criminology & Criminal Justice, 15(4), 442–463.

Koopman, J., Matta, F. K., Scott, B. A., & Conlon, D. E. (2015). Ingratiation and popularity as antecedents of justice: A social exchange and social capital perspective. Organizational Behavior and Human Decision Processes131, 132-148.

Korsgaard, M. A., Roberson, L., & Rymph, R. D. (1998). What motivates fairness? The role of subordinate assertive behavior on manager's interactional fairness. Journal of Applied Psychology83(5), 731.

Lim, V. K. (2002). The IT way of loafing on the job: Cyberloafing, neutralizing and organizational justice. Journal of Organizational Behavior23(5), 675-694.

Matta, F. K., Scott, B. A., Colquitt, J. A., Koopman, J., & Passantino, L. G. (2017). Is consistently unfair better than sporadically fair? An investigation of justice variability and stress. Academy of Management Journal60(2), 743-770.

McFarlin, D. B., & Sweeney, P. D. (1992). Distributive and procedural justice as predictors of satisfaction with personal and organizational outcomes. Academy of Management Journal35(3), 626-637.

Myhill, A., & Bradford, B. (2013). Overcoming cop culture? Organizational justice and police officers’ attitudes toward the public. Policing: An International Journal of Police Strategies & management36(2), 338-356.

Nix, J., & Wolfe, S. E. (2016). Sensitivity to the Ferguson effect: The role of managerial organizational justice. Journal of Criminal Justice, 47(1), 12–20.

Paternoster, R., Brame, R., Mazerolle, P., & Piquero, A. (1998). Using the correct statistical test for the equality of regression coefficients. Criminology, 36(4), 859-66.

Rosenbaum, D. P., & McCarty, W. P. (2017). Organizational justice and officer “buy in” in American policing. Policing, 40(1), 71–85.

Scott, B. A., Colquitt, J. A., & Zapata-Phelan, C. P. (2007). Justice as a dependent variable: Subordinate charisma as a predictor of interpersonal and informational justice perceptions. Journal of Applied Psychology92(6), 1597.

Sweeney, P. D., & McFarlin, D. B. (1993). Workers′ evaluations of the" ends" and the" means": An examination of four models of distributive and procedural justice. Organizational Behavior and Human Decision Processes55(1), 23-40.

Tankebe, J. (2014). The making of ‘democracy’s champions’: Understanding police support for democracy in Ghana. Criminology & Criminal Justice14(1), 25-43.

Thaler, R. H. (Ed.). (2005). Advances in behavioral finance (Vol. 2). Princeton University Press.

Trinkner, R., Tyler, T. R., & Goff, P. A. (2016). Justice from within: The relations between a procedurally just organizational climate and police organizational efficiency, endorsement of democratic policing, and officer well-being. Psychology, Public Policy, and Law22(2), 158.

Van Craen, M., & Skogan, W. G. (2017a). Achieving fairness in policing: The link between internal and external procedural justice. Police Quarterly, 20(1), 3–23.

Van Craen, M., & Skogan, W. G. (2017b). Officer support for use of force policy: The role of fair supervision. Criminal Justice and Behavior, 44(6), 843–861.

Whiteside, D. B., & Barclay, L. J. (2018). When wanting to be fair is not enough: The effects of depletion and self-appraisal gaps on fair behavior. Journal of Management44(8), 3311-3335.

Wolfe, S. E., & Nix, J. (2017). Police officers’ trust in their agency: Does self-legitimacy protect against supervisor procedural injustice? Criminal Justice and Behavior, 44(5), 717–732.

Wolfe, S. E., & Piquero, A. R. (2011). Organizational justice and police misconduct. Criminal Justice and Behavior, 38(4), 332–353.

Wolfe, S. E., Nix, J., & Campbell, B. A. (2018). Police managers’ self-control and support for organizational justice. Law and Human Behavior, 42(1), 71-82.

Wolfe, S. E., Rojek, J., Manjarrez Jr, V. M., & Rojek, A. (2018). Why does organizational justice matter? Uncertainty management among law enforcement officers. Journal of Criminal Justice54, 20-29.

Zapata, C. P., Olsen, J. E., & Martins, L. L. (2013). Social exchange from the supervisor’s perspective: Employee trustworthiness as a predictor of interpersonal and informational justice. Organizational Behavior and Human Decision Processes121(1), 1-12.

No comments here
Why not start the discussion?