DV not measured but part of the title and other issues…

In Transformational Leadership, Organizational Commitment and Taking Charge in Small Businesses in the Northwestern United States, Willis (2020) reports to have measured three dimensions –

Transformation Leadership
Organizational Commitment
Taking Charge

The emerging scholar used the MLQ-5X Short Form to measure transformational leadership, and the Three-Component Model (TCM) of commitment Employee Commitment Survey to measure organizational commitment (pp. 61-67). Both are widely used in social science research. I was intrigued about how the “Taking Charge” dimension would be measured. During my time reviewing student dissertations, I had never seen the phrase operationalized. However, I was surprised to not find a discussion about it in the methodology section, although it is prominently listed in RQ2: What is the relationship between transformational leadership and organizational employee take-charge behavior? (emphasis added).

After a search for the phrase in the document, I found an interesting passage –

The instruments used by Morrison and Phelps and Kim and Lui ultimately did not measure take charge behavior as defined in this study. No instrument was found to measure take charge behavior adequately.
Willis, 2020, p. 27

So, the phrase has been defined, operationalized, and written about in peer-reviewed journals by Morrison and Phelps (1999), and used and confirmed by Kim and Lui (2017); however, that didn’t match the definition adopted by the emerging scholar (and the committee)? Why not simply change the definition to match other researchers? At a minimum, create an instrument that matches the new operationalized definition? Regardless, why keep the phrase in the title for search engines to find? Post-publication review begins!

Willis cited Love and Dustin (2014) as the source for this definition of Taking Charge –

The efforts both voluntary and constructive in the nature and exertion of the individual employee’s desire to affect change within the organization about how tasks are executed
Willis, 2020, p. 18

If Love and Dustin’s definition differed from Morrison and Phelps and Kim and Lui, why not use the instrument that Love and Dustin used in their study? Guess what? They measured the Taking Charge dimension using the Morrison and Phelps instrument!

Mads Østberg on Twitter: "As Homer Simpson would have put it, DOH! 🙈🤦… " — I can’t believe somebody read my doctoral study

I don’t understand why the committee didn’t press the issue with the student (if they even read the study), and how the title and abstract, which states RQ2 was never answered, got through at least three reviewers. Now back to RQ1.

In RQ1, the emerging scholar explored the relationship between transformation leadership and organizational commitment. This is a common research question for doctoral students. To answer the question, one would follow a standard process –

Collect data via survey
Form the dimensions and subdimensions of inquiry by averaging items (e.g., add items a, b, c, & d and divide by 4). Don’t forget to reverse code when necessary!
Report descriptive statistics (M, SD, SE)
Perform exploratory data analysis such as examining outliers and making a decision about the distribution of each variable by looking at graphs and performing statistical tests
Perform a statistical test appropriate for interval variables such as Pearson Product-Moment Correlation or Spearman Rank-Order Correlation.

What was done? The emerging scholar did not report any descriptive statistics regarding the dimensions formed (if they were even formed), and there is no reporting on the variable’s distribution. A Chi-square test was used to reject the null hypothesis. A Chi-square test is used to examine the association of categorical variables; think of a 2X4 matrix of Gender (0/1) and Education(0:4). In other words, the wrong test was used. The emerging scholar merely reported the effect size (Cramer’s V), and p-value.

Who reviewed this study? The review process let this student down. As a result, the results of this study should be ignored.

Student Note: Here’s a great chance to do some research; just do it correctly.

References:

Liu, Y., Loi, R., & Lam, L. W. (2011). Linking organizational identification and employee
performance in teams: The moderating role of team-member exchange. International
Journal of Human Resource Management, 22(15),3187-3201. https://doi.org/10.1080/09585192.2011.560875

Love, M. S., & Dustin, S. L. (2014). An investigation of coworker relationship and psychological collectivism on employee propensity to take charge. The International Journal of Human Resource Managment, 25(9), 1208-1226. http://dx.doi.org/10.1080/09585192.2013.826712

Morrison, E. W., & Phelps, C. C. (1999). Taking charge at work: Extrarole efforts to initiate
workplace change. Academy of Management Journal, 42(4), 403–419. https://doi.org/10.5465/257011

Willis, S. G. (2020). Transformational leadership, organizational commitment and taking charge in small businesses in the Northwestern United States (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (27998399)

Can a government program recipient be led?

Robinson (2020) interviewed eight recipients of the U.S. Department of Agriculture’s Supplemental Nutrition Assistance Program (SNAP) for the purposes of understanding healthy food choice options and program leadership. What struck my interest was the inclusion of the term Transformational Leadership in the title. It appears the emerging scholar was attempting to connect leadership behavior by Federal government program administrators to food choices made by recipients in the program. I don’t see it. Three things popped into my head –

Were any of the program leaders executing transformational leadership qualities?
What influence does USDA leadership styles have on the leadership styles of regional administrators?
If transformational leadership falls under the umbrella of organizational theory, are recipients of of aid considered part of an organization?

When exploring the themes (p. 55), I came to the conclusion this was simply another example of an oral survey in which participants gave answers to questions –

Participants need better guidance in making healthier food choices
Participants make food choices based on various factors
Federal leaders should hold regional leaders accountable for informing participants about benefits under the program
Participants should be explained the consequences of purchasing certain types of foods

Note that none of these themes had anything to do with transformational leadership and were more related to program evaluation gap analysis. I wonder if the participants could name any of the regional leadership they were supposed to follow?

Ignore the results of this study.

Reference:

Robinson, A. M. (2020). Exploring transformational leadership of the Supplemental Nutrition Assistance Program (SNAP) – A qualitative study. ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (28031688)

Can different styles of leadership be performed simultaneously?

During my review of dissertations found in ProQuest, I came across a study on the influence of leadership styles on construction project success (Parson, 2020). In this study, the novice researcher examined the relationship between leadership styles and overall construction project success based on the self-assessment of the respondents (N = 78).

The independent variable, leadership styles (or behaviors), were measured by the widely used MLQ5X-Short Self-Rater Form (Avolio et al., 1999; Bass & Avolio, 2005). The form, a 36-item, 9-dimensional survey instrument, was provided to construction project managers to self-assess their perceived leadership behaviors. The instrument’s 9-dimensions are used to form three overall styles: Transformation, Transactional , and Laissez-faire. The dependent variable, overall construction success, was measured by the Project Implementation Profile (PIP), a 50-item, 10-dimension survey instrument developed by Slevin and Pinto (1986). Several control variables were also reported to be used: country and number of years as a construction project manager (p. 17), and age, state where respondent worked, type of construction industry, years worked in the construction industry, and gender (p. 23).

What interested me first was the reporting of the strength in the relationship between three IVs and DV (from Table 3 on p. 90):

Transformational Leadership -> PIP, r = .54, 95% CI (.39, .67), moderate-to-large effect size
Transactional Leadership -> PIP, r = .45, 95% CI (.28, .59), small-to-large effect size
Laissez-faire Leadership -> PIP, r = -.37, 95% CI (-53, -.19), small-to-large effect size

All three relationships appear to be statistically significant. Confidence intervals and effect size characterizations (Cohen 1988), were added here for informational purposes. Student Note #1: Report confidence intervals and effect sizes so a reader can make a judgments about the sample size of your study. The student also performed simple linear regression, which was unnecessary. By just squaring the r, the novice researcher can show, for example, 27% of the change in PIP can be attributed to a one unit change in transformational leadership (H1).

Next, I searched and could not find where the control variables were used. I connected with the author and who stated “most likely I used the incorrect terminology.” It appears she was using the term control variable when in fact she was using certain attributes to stratify her sample. Student Note #2: Make sure you know the terms used in your study.

Finally, the novice researcher did something unique: She entered multiple leadership styles simultaneously into a regression formula as IVs to predict PIP. I’ve never seen something like that done before. In the project management literature, there are discussions about using different leadership styles (a) at certain times of a project, (b) based on the type of project, and (c) with specific types of people assigned to a project. However, I couldn’t find any reference to an examination into multiple distinct leadership behaviors simultaneously influencing an outcome variable? I then started to think: Why did the student do regression at all?

Correlation does not equal causation. Had the novice researcher simply ended the study after performing a correlation analysis, the study would have been done. Not a very rigorous study, but done. When one builds a regression model, though, one is exploring prediction or forecasting of an outcome (PIP) based on related input predictors. Through this process, causation could be inferred. I would think that the experience of the project manager (perhaps using age as a proxy), the gender of the project manager, project length and complexity, and types of workers on the project could also influence PIP; but they are no where to be found. If so, I speculate the size of the leadership coefficient would be less as it gives way to other variables.

This novice research, in my opinion, under thought the study’s methodology and data analysis approach and the results should be reviewed with caution. The novice researcher’s committee also should have caught this. Perhaps a replication of this study by exploring different aspects of leadership and their influence on the different aspects of the project implementation process controlling for confounders.

Note: I corresponded with the novice researcher about sharing the data so we could perform an analysis looking at confounding variables; however, the University IRB denied the request. Since the data was collected by a third-party (QuestionPro), I suspect the raw data contains no personal identification information. This means the study was exempt from human subject research protocol, as defined by the US Federal Government. Thus, I don’t know why the University IRB would restrict a student from sharing their information in this situation.

References:

Avolio, B. J., Bass, B. M., & Jung, D. I. (1999). Re-examining the components of transformational and transactional leadership using the Multifactor Leadership Questionnaire. Journal of Occupational and Organizational Psychology, 72(4), 441-462. https://doi.org/10.1348/096317999166789

Bass, B. M., & Avolio, B. J. (1995). MLQ: Multifactor Leadership Questionnaire. MindGarden.

Cohen, J. B. (1988). Statistical power analysis for the behavioral sciences (2nd ed). Lawrence Erlbaum Associates.

Parson, S. J. (2020). Relationship between U.S. construction project managers’ leadership styles and construction project success (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (28028785)