Type I errors galore…

Legatti-Maddox (2019) explored the moderating effect of two leadership styles, transformational and transactional, on the relationship between four types of humor and Organizational Citizenship Behavior. The sample for this study were 42 MBA students.

First, the sample size (N = 42) concerned me since it seemed to be a bit small to find a practical (moderate) effect (f2 = 0.15). Using the pwr.f2.test() function from R’s pwr package (Champerly, 2020), it appears a sample of at least 73 would be required with three independent variables and a minimum power of .80 (see below).

          u = 3
          v = 72.70583
         f2 = 0.15
  sig.level = 0.05
      power = 0.8

So, it appears the study was underpowered by design. With underpowered studies there is a low probability of finding true effects and any effects could be false. Let’s move forward…

Second, when performing a moderation analysis, one has to enter both independent variables along with the interaction (see below)

y = X1 + X2 + X1*X2

If the moderating variable (X1*X2) is significant, then the interaction is explored and the independent variables generally lose their value from a research perspective. However, since no independent or interactive variable’s p-values were reported, no moderation evidence was provided by the emerging scholar. Instead. p-values of unmoderated and moderated models were compared, and an increase in the F-statistic reported as evidence of a moderating effect. That’s a flawed approach and, when the null hypothesis is rejected based on that approach, a Type I error ensues.

In a prior post, I discuss the uses of P-P Plots vs. Q-Q Plots and how it’s a default option in regression under SPSS. This emerging scholar used this plot (from SPSS) and stated that the homoscedasticity assumption was met.

Figure 1. Normal P-P Plot of Regression Standardized Residual (p. 70)

However, there is no reference to the independent variables or which model. I wonder what would have happened if her faculty advisor challenged her and said the residuals are hetroskedastic?

Finally, a quick look at a summary table in the study (Table 19 below)

A learned faculty should have counseled this student that the p-values would need to be adjusted for potential family-wise errors as the student’s premise is that all nine models are true. The widely-cited Bonferroni correction would result in a new p-value of 0.0055 (.05/9). If applied, only Model 9 may have met the criteria. However the focus of the study was not on whether a model could be constructed, but whether the interaction of humor and leadership explained the relationship better than the direct effects. Thus, more Type I errors.

The interaction of humor and leadership may influence OCB, but this study provides no evidence. The results of this study should be ignored.

Student Note: The way to approach this would be through some Structured Equation Model (SEM) that controls for Type I errors.

Reference:

Champely, S. (2020, March 16). pwr: Basic functions of power analysis. https://cran.r-project.org/web/packages/pwr/pwr.pdf

Legatti-Maddox, A. C. (2019). Humor style in the workplace as it relates to leadership style and organizational citizenship behavior (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (22622521)

Can different styles of leadership be performed simultaneously?

During my review of dissertations found in ProQuest, I came across a study on the influence of leadership styles on construction project success (Parson, 2020). In this study, the novice researcher examined the relationship between leadership styles and overall construction project success based on the self-assessment of the respondents (N = 78).

The independent variable, leadership styles (or behaviors), were measured by the widely used MLQ5X-Short Self-Rater Form (Avolio et al., 1999; Bass & Avolio, 2005). The form, a 36-item, 9-dimensional survey instrument, was provided to construction project managers to self-assess their perceived leadership behaviors. The instrument’s 9-dimensions are used to form three overall styles: Transformation, Transactional , and Laissez-faire. The dependent variable, overall construction success, was measured by the Project Implementation Profile (PIP), a 50-item, 10-dimension survey instrument developed by Slevin and Pinto (1986). Several control variables were also reported to be used: country and number of years as a construction project manager (p. 17), and age, state where respondent worked, type of construction industry, years worked in the construction industry, and gender (p. 23).

What interested me first was the reporting of the strength in the relationship between three IVs and DV (from Table 3 on p. 90):

  • Transformational Leadership -> PIP, r = .54, 95% CI (.39, .67), moderate-to-large effect size
  • Transactional Leadership -> PIP, r = .45, 95% CI (.28, .59), small-to-large effect size
  • Laissez-faire Leadership -> PIP, r = -.37, 95% CI (-53, -.19), small-to-large effect size

All three relationships appear to be statistically significant. Confidence intervals and effect size characterizations (Cohen 1988), were added here for informational purposes. Student Note #1: Report confidence intervals and effect sizes so a reader can make a judgments about the sample size of your study. The student also performed simple linear regression, which was unnecessary. By just squaring the r, the novice researcher can show, for example, 27% of the change in PIP can be attributed to a one unit change in transformational leadership (H1).

Next, I searched and could not find where the control variables were used. I connected with the author and who stated “most likely I used the incorrect terminology.” It appears she was using the term control variable when in fact she was using certain attributes to stratify her sample. Student Note #2: Make sure you know the terms used in your study.

Finally, the novice researcher did something unique: She entered multiple leadership styles simultaneously into a regression formula as IVs to predict PIP. I’ve never seen something like that done before. In the project management literature, there are discussions about using different leadership styles (a) at certain times of a project, (b) based on the type of project, and (c) with specific types of people assigned to a project. However, I couldn’t find any reference to an examination into multiple distinct leadership behaviors simultaneously influencing an outcome variable? I then started to think: Why did the student do regression at all?

Correlation does not equal causation. Had the novice researcher simply ended the study after performing a correlation analysis, the study would have been done. Not a very rigorous study, but done. When one builds a regression model, though, one is exploring prediction or forecasting of an outcome (PIP) based on related input predictors. Through this process, causation could be inferred. I would think that the experience of the project manager (perhaps using age as a proxy), the gender of the project manager, project length and complexity, and types of workers on the project could also influence PIP; but they are no where to be found. If so, I speculate the size of the leadership coefficient would be less as it gives way to other variables.

This novice research, in my opinion, under thought the study’s methodology and data analysis approach and the results should be reviewed with caution. The novice researcher’s committee also should have caught this. Perhaps a replication of this study by exploring different aspects of leadership and their influence on the different aspects of the project implementation process controlling for confounders.

Note: I corresponded with the novice researcher about sharing the data so we could perform an analysis looking at confounding variables; however, the University IRB denied the request. Since the data was collected by a third-party (QuestionPro), I suspect the raw data contains no personal identification information. This means the study was exempt from human subject research protocol, as defined by the US Federal Government. Thus, I don’t know why the University IRB would restrict a student from sharing their information in this situation.

References:

Avolio, B. J., Bass, B. M., & Jung, D. I. (1999). Re-examining the components of transformational and transactional leadership using the Multifactor Leadership Questionnaire. Journal of Occupational and Organizational Psychology, 72(4), 441-462. https://doi.org/10.1348/096317999166789

Bass, B. M., & Avolio, B. J. (1995). MLQ: Multifactor Leadership Questionnaire. MindGarden.

Cohen, J. B. (1988). Statistical power analysis for the behavioral sciences (2nd ed). Lawrence Erlbaum Associates.

Parson, S. J. (2020). Relationship between U.S. construction project managers’ leadership styles and construction project success (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (28028785)