Keyword search vs Abstract search…

The purpose of placing keywords in an abstract is to allow a search engine or another researcher to easily identify main topics in your research. For additional thoughts, see link and link.

As I continue digging through doctoral studies to identify patterns of concern or mistakes, I began reviewing studies from a University that uses a case study method for many students. I’ve identified problems in case studies here and here. I wanted to quickly see how many times the phrase “case study” appeared as a keyword or phrase. Using the R library tidyverse, and two commands (str_detect and table), I found only 4 instances in the keywords:

library(tidyverse)
str_detect(selected_university$keywords, "case study") %>% table()
.
FALSE  TRUE 
  233     4 

However, when I searched for the same string in the Abstract, I found 215 instances.

str_detect(selected_university$abstract, "case study") %>% table()
.
FALSE  TRUE 
   22   215 

This tells me that a specific research design is deemed not important enough to place as a keyword phrase. No problem.

Student Note: Don’t rely on keywords for finding similar types of research designs.

It also tells me that 90% of this University’s DBA graduates in 2019 used the same research design. Did I hear somebody say formulaic?

In writing about formulaic papers in organizational research, Alvesson and Gabriel wrote –

Formulaic papers are the products of a sequence of interrelated codified and standardized practices that involve formulaic research, a formulaic editorial process, formulaic reviewing, and more generally, formulaic mind-sets, that is, formulaic ways of thinking about what constitutes scholarship. Reliance on a formula is in itself not detrimental to quality, especially if the formula has yielded good results in the past. As we shall see presently, however, slavish adherence to formula renders researchers oblivious to potentially interesting possibilities that exist outside the formula,
eliminating the scope for serendipity and accidental discovery that have long been crucial factors in
scientific discovery and technological innovations

Alvesson & Gabriel, 2013, p. 247 (emphasis added)

I don’t have a problem with writing templates or standardized statistical approaches, but when 90% of a University’s doctoral studies relate to case study methodology, and issues have been identified in research from that University relating to the framing and execution of the case study method, what does that say about the quality of the formula?

Reference:

Alvesson, M., & Gabriel, Y. (2013). Beyond formulaic research: In praise of greater diversity in organizational research and publications. Academy of Management Learning & Education, 12(2), 245-263. https://doi.org/10.5465/amle.2012.0327

When the first interview question equals the research question, why ask for more information?

Using a multiple-case study approach (N = 3), Uhuegbulem (2019) explored strategies for retiring oil and gas assets in Canada. The emerging scholar did not describe the three organizations under study. Instead, one ‘business leader’ of selected organizations was used as a proxy. The term business leader is not described; there is no evidence this leader was the President, Owner, or Managing Partner of the organization. There is also no evidence of a review of organizational documents substantiating how retired oil and gas assets are retired, tracked, and management.

What intrigued me was the research question –

What strategies do asset managers in small- and medium-sized O&G companies use to manage retired O&G assets effectively to increase organizational sustainability?

Uhuegbulem, 2019, p. 5

I wondered how the emerging scholar was going to determine how retired O&G asset management would lead to organizational sustainability or anything else, which is a cause-and-effect issue. I guess the first item of the interview guide would answer my question on how the emerging scholar would answer the research question –

What strategies do you (the participant) use to track, monitor, and manage retired O&G assets effectively?

Uhuegbulem, 2019, p. 5 & Appendix B (emphasis added)

The emerging scholar wasn’t going to do it…the participant would do it.

The Easy Button - Creations - paint.net Forum

Reference:

Uhuegbulem, I. (2019). Strategies for oil and gas asset retirement sustainability in Alberta, Canada (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (13864356)

Examining influence in a qualitative study?

It’s very important to use the correct terminology associated with a research method and design. For example, the word influence is widely-associated with quantitative research. Influence can be measured by examining the change in the Y variable when the X variable is manipulated or involving a third variable (Z). Quantitative research is more scientific, less subjective, is repeatable, and can be generalized. Qualitative research is based on the knowledge and skill of the researcher. There are times when an experienced researcher will explore influence in a qualitative study but that is few and far between, generally related to specific disciplines (e.g., medical, social work), and is supported with significant academic research (see here, here, and here). I don’t recommend emerging scholars perform qualitative research. Besides skill, the time needed to complete a qualitative study is much longer than the time needed to complete a quantitative study.

Grant (2019) is an example of why emerging scholars shouldn’t do qualitative research. This emerging scholar explored the influence of leadership behaviors on two dimensions: employee engagement and collaboration (the organization is not germane to this discussion). To perform this study, the emerging scholar created a 7-item open-ended survey and distributed it anonymously to 10 people in an organization exceeding 3,800 people. The emerging scholar would interpret the responses and categorize them to answer the following two research questions –

  • What leadership styles and behaviors are being utilized at [organization]?
  • What is the influence of existing leadership styles and behaviors on employee engagement and collaboration?

Yin (2018) describes five situations where a single case study would be appropriate: critical, unusual, common, revelatory, or longitudinal (pp. 48-50). In addition, Yin describes two types of single case studies: holistic and embedded (pp. 51-53). When reviewing the dissertation, the researcher is attempting a build a common, holistic single-case study. Common because leadership is an everyday situation. Holistic because the organization appears to have a single purpose. However, a case study focuses on “how” or “why” a situation occurred (perhaps leadership style evolution); not “what” style is prevalent or which specific styles influence two outcomes. With an anonymous survey, there is no way to follow up with a participant to clarify their responses. To quote a colleague –

Who’s the researcher? Carnac the Magnificent?

Name withheld

As a result, the research method (QUAL) and design (case study) doesn’t appear to align with the research questions. The results of the study should be ignored. However, I wanted to discuss the themes identified by Grant –

  • A collaborative, or transformational, leadership style is present
  • Organizational leaders are engaging
  • Unfair hiring practices have become standard

First, are collaborate and transformational the same? They’re close, but I believe some scholars would say they’re different. Second, what does the organization’s hiring practices have to do with leadership in an organization? Plus, how can one generalize to an organization of 3,800+ from a sample of 10? Do the math: That’s a 95% CI of nearly 31 points! Even if 90% of the sample described an organization leaders as collaborative, as interpreted by the researcher, that means the 95% CI would between 60% and Inf. What are the other 40%? Non-collaborative?

Reference:

Grant, R. M. (2019). Investigating the influence of leadership behaviors on employee engagement and collaboration in a Federal organization (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (22615969)

Yin, R. K. (2018). Case study research and applications: Design and methods (6th Ed.). SAGE Publications.

When do numbers count in a QUAL study?

Gary (2019) wrote –

This research method (qualitative) addresses “how” questions – rather than “how many” through the perspective of those studied – informants.

Gary, 2019, p. 31

Makes sense…qualitative studies are not about “how many” but the words used by participants to describe their experiences and the interpretation of those words by the researcher based on their worldview and theoretical framework.

So, why report this?

Figure 1. Thematic Coding (Gary, 2019, p. 64).

I guess the experiences are important when proposing the study; however, its important to “demonstrate why one should have confidence in the findings” (Hannah & Lautsch, 2011, p. 16). Hannah and Lautsch call his credentialing counting. Who cares if a theme was framed from the responses of 10/10 or 9/10 of participants? Isn’t the theme more important?

There are other problems with this research (e.g., 7 formal “questions” vs an interview guide, no research question of any kind to guide the study), but this counting issue just bugs me. I agree with Sutton (2017): put the numbers in the closet.

References:

Gary, M. E. (2019). Managing toxic leaders: An exploration of human resources management’s role in mitigating the impact of leader imposed toxicity on organization, individuals, and other stakeholders (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection (13897507)

Hannah, D. R., & Lautsch, B. A. (2011). Counting in qualitative research: Why to conduct it, when to avoid it, and when to closet it. Journal of Management Inquiry, 20(1), 14-22. https://doi.org/10.1177/1056492610375988

Sutton, R. I. (1997). The virtues of closet qualitative research. Organizational Science, 8(1), 97-106.

Type I errors galore…

Legatti-Maddox (2019) explored the moderating effect of two leadership styles, transformational and transactional, on the relationship between four types of humor and Organizational Citizenship Behavior. The sample for this study were 42 MBA students.

First, the sample size (N = 42) concerned me since it seemed to be a bit small to find a practical (moderate) effect (f2 = 0.15). Using the pwr.f2.test() function from R’s pwr package (Champerly, 2020), it appears a sample of at least 73 would be required with three independent variables and a minimum power of .80 (see below).

          u = 3
          v = 72.70583
         f2 = 0.15
  sig.level = 0.05
      power = 0.8

So, it appears the study was underpowered by design. With underpowered studies there is a low probability of finding true effects and any effects could be false. Let’s move forward…

Second, when performing a moderation analysis, one has to enter both independent variables along with the interaction (see below)

y = X1 + X2 + X1*X2

If the moderating variable (X1*X2) is significant, then the interaction is explored and the independent variables generally lose their value from a research perspective. However, since no independent or interactive variable’s p-values were reported, no moderation evidence was provided by the emerging scholar. Instead. p-values of unmoderated and moderated models were compared, and an increase in the F-statistic reported as evidence of a moderating effect. That’s a flawed approach and, when the null hypothesis is rejected based on that approach, a Type I error ensues.

In a prior post, I discuss the uses of P-P Plots vs. Q-Q Plots and how it’s a default option in regression under SPSS. This emerging scholar used this plot (from SPSS) and stated that the homoscedasticity assumption was met.

Figure 1. Normal P-P Plot of Regression Standardized Residual (p. 70)

However, there is no reference to the independent variables or which model. I wonder what would have happened if her faculty advisor challenged her and said the residuals are hetroskedastic?

Finally, a quick look at a summary table in the study (Table 19 below)

A learned faculty should have counseled this student that the p-values would need to be adjusted for potential family-wise errors as the student’s premise is that all nine models are true. The widely-cited Bonferroni correction would result in a new p-value of 0.0055 (.05/9). If applied, only Model 9 may have met the criteria. However the focus of the study was not on whether a model could be constructed, but whether the interaction of humor and leadership explained the relationship better than the direct effects. Thus, more Type I errors.

The interaction of humor and leadership may influence OCB, but this study provides no evidence. The results of this study should be ignored.

Student Note: The way to approach this would be through some Structured Equation Model (SEM) that controls for Type I errors.

Reference:

Champely, S. (2020, March 16). pwr: Basic functions of power analysis. https://cran.r-project.org/web/packages/pwr/pwr.pdf

Legatti-Maddox, A. C. (2019). Humor style in the workplace as it relates to leadership style and organizational citizenship behavior (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (22622521)