Book Review: Evaluating Research in Academic Journals: A Practical Guide to Realistic Evaluation (Pyrczak & Tcherni-Buzzeo, 2019)

I’m back! I’ve taken off a few months to read, recharge, and frame some study proposals with colleagues. I have a lot of things in the hopper…

During my break, I had a chance to help a PhD student graduate. It took some heavy lifting (another blog post by itself), but he made it! During my discussion with the emerging scholar, he asked me a question –

Which books do you own that could help me?

Hmmm…I have many, but let me start a list.

First up: Evaluating Research in Academic Journals: A Practical Guide to Realistic Evaluation

The most recent 7th Edition is written by Maria Tcherni-Buzzeo of New Haven University and pays homage to the originator of the book, the late Fred Pyrczak (1945-2014). The authors focus on how to read everything from Abstracts, Introductions, and Literature Reviews through the Analysis and Results section, ending in the Discussion section. In a checklist/rubric format, the book provides items (with example narrative in most places) such as –

  • Are primary variables mentioned in the title? (p. 17)
  • Does the introduction move from topic to topic instead of from citation to citation? (p. 43)
  • If the response rate was low, did the researcher make multiple attempts to contact potential participants? (p. 67)
  • If any differences are statistically significant but substantively small, have the researches noted they are small? (p. 123)

There are also specific sections in QUAN, QUAL, and MM research, which I have found invaluable.

This book is great for emerging scholars as they can apply it to learn how to critique academic research. It’s also great for chairpersons and people like me that critique research all day. It’s a must read (and buy!).

Advertisement

Keyword search vs Abstract search…

The purpose of placing keywords in an abstract is to allow a search engine or another researcher to easily identify main topics in your research. For additional thoughts, see link and link.

As I continue digging through doctoral studies to identify patterns of concern or mistakes, I began reviewing studies from a University that uses a case study method for many students. I’ve identified problems in case studies here and here. I wanted to quickly see how many times the phrase “case study” appeared as a keyword or phrase. Using the R library tidyverse, and two commands (str_detect and table), I found only 4 instances in the keywords:

library(tidyverse)
str_detect(selected_university$keywords, "case study") %>% table()
.
FALSE  TRUE 
  233     4

However, when I searched for the same string in the Abstract, I found 215 instances.

str_detect(selected_university$abstract, "case study") %>% table()
.
FALSE  TRUE 
   22   215

This tells me that a specific research design is deemed not important enough to place as a keyword phrase. No problem.

Student Note: Don’t rely on keywords for finding similar types of research designs.

It also tells me that 90% of this University’s DBA graduates in 2019 used the same research design. Did I hear somebody say formulaic?

In writing about formulaic papers in organizational research, Alvesson and Gabriel wrote –

Formulaic papers are the products of a sequence of interrelated codified and standardized practices that involve formulaic research, a formulaic editorial process, formulaic reviewing, and more generally, formulaic mind-sets, that is, formulaic ways of thinking about what constitutes scholarship. Reliance on a formula is in itself not detrimental to quality, especially if the formula has yielded good results in the past. As we shall see presently, however, slavish adherence to formula renders researchers oblivious to potentially interesting possibilities that exist outside the formula,
eliminating the scope for serendipity and accidental discovery that have long been crucial factors in
scientific discovery and technological innovations

Alvesson & Gabriel, 2013, p. 247 (emphasis added)

I don’t have a problem with writing templates or standardized statistical approaches, but when 90% of a University’s doctoral studies relate to case study methodology, and issues have been identified in research from that University relating to the framing and execution of the case study method, what does that say about the quality of the formula?

Reference:

Alvesson, M., & Gabriel, Y. (2013). Beyond formulaic research: In praise of greater diversity in organizational research and publications. Academy of Management Learning & Education, 12(2), 245-263. https://doi.org/10.5465/amle.2012.0327

When the first interview question equals the research question, why ask for more information?

Using a multiple-case study approach (N = 3), Uhuegbulem (2019) explored strategies for retiring oil and gas assets in Canada. The emerging scholar did not describe the three organizations under study. Instead, one ‘business leader’ of selected organizations was used as a proxy. The term business leader is not described; there is no evidence this leader was the President, Owner, or Managing Partner of the organization. There is also no evidence of a review of organizational documents substantiating how retired oil and gas assets are retired, tracked, and management.

What intrigued me was the research question –

What strategies do asset managers in small- and medium-sized O&G companies use to manage retired O&G assets effectively to increase organizational sustainability?

Uhuegbulem, 2019, p. 5

I wondered how the emerging scholar was going to determine how retired O&G asset management would lead to organizational sustainability or anything else, which is a cause-and-effect issue. I guess the first item of the interview guide would answer my question on how the emerging scholar would answer the research question –

What strategies do you (the participant) use to track, monitor, and manage retired O&G assets effectively?

Uhuegbulem, 2019, p. 5 & Appendix B (emphasis added)

The emerging scholar wasn’t going to do it…the participant would do it.

The Easy Button - Creations - paint.net Forum

Reference:

Uhuegbulem, I. (2019). Strategies for oil and gas asset retirement sustainability in Alberta, Canada (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (13864356)

Examining influence in a qualitative study?

It’s very important to use the correct terminology associated with a research method and design. For example, the word influence is widely-associated with quantitative research. Influence can be measured by examining the change in the Y variable when the X variable is manipulated or involving a third variable (Z). Quantitative research is more scientific, less subjective, is repeatable, and can be generalized. Qualitative research is based on the knowledge and skill of the researcher. There are times when an experienced researcher will explore influence in a qualitative study but that is few and far between, generally related to specific disciplines (e.g., medical, social work), and is supported with significant academic research (see here, here, and here). I don’t recommend emerging scholars perform qualitative research. Besides skill, the time needed to complete a qualitative study is much longer than the time needed to complete a quantitative study.

Grant (2019) is an example of why emerging scholars shouldn’t do qualitative research. This emerging scholar explored the influence of leadership behaviors on two dimensions: employee engagement and collaboration (the organization is not germane to this discussion). To perform this study, the emerging scholar created a 7-item open-ended survey and distributed it anonymously to 10 people in an organization exceeding 3,800 people. The emerging scholar would interpret the responses and categorize them to answer the following two research questions –

  • What leadership styles and behaviors are being utilized at [organization]?
  • What is the influence of existing leadership styles and behaviors on employee engagement and collaboration?

Yin (2018) describes five situations where a single case study would be appropriate: critical, unusual, common, revelatory, or longitudinal (pp. 48-50). In addition, Yin describes two types of single case studies: holistic and embedded (pp. 51-53). When reviewing the dissertation, the researcher is attempting a build a common, holistic single-case study. Common because leadership is an everyday situation. Holistic because the organization appears to have a single purpose. However, a case study focuses on “how” or “why” a situation occurred (perhaps leadership style evolution); not “what” style is prevalent or which specific styles influence two outcomes. With an anonymous survey, there is no way to follow up with a participant to clarify their responses. To quote a colleague –

Who’s the researcher? Carnac the Magnificent?

Name withheld

As a result, the research method (QUAL) and design (case study) doesn’t appear to align with the research questions. The results of the study should be ignored. However, I wanted to discuss the themes identified by Grant –

  • A collaborative, or transformational, leadership style is present
  • Organizational leaders are engaging
  • Unfair hiring practices have become standard

First, are collaborate and transformational the same? They’re close, but I believe some scholars would say they’re different. Second, what does the organization’s hiring practices have to do with leadership in an organization? Plus, how can one generalize to an organization of 3,800+ from a sample of 10? Do the math: That’s a 95% CI of nearly 31 points! Even if 90% of the sample described an organization leaders as collaborative, as interpreted by the researcher, that means the 95% CI would between 60% and Inf. What are the other 40%? Non-collaborative?

Reference:

Grant, R. M. (2019). Investigating the influence of leadership behaviors on employee engagement and collaboration in a Federal organization (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (22615969)

Yin, R. K. (2018). Case study research and applications: Design and methods (6th Ed.). SAGE Publications.