What could be found with 16 IVs and an N = 31?

At the time of writing this post, I’m looking at a recently introduced colleagues doctoral dissertation. The faculty member went to the same school as me, but several years later. It’s always interesting looking at people’s dissertations because it can show a person’s beginning academic success. This faculty member will be mentoring emerging scholars so I always hope more knowledge has been acquired. Why do I say that? Well…

This faculty, as a doctoral student, explored the influence of, what are described as, critical business variables on solo criminal law practitioner success. In the study, the faculty utilized an instrument that purports to have been validated many times in many countries. I don’t feel like exploring that claim so I skipped to the data analysis plan and results.

First, when determining a sample size, one estimates the desired effect size. An effect size is commonly based on prior research, but can include other factors (e.g., practicality). In this study, the then emerging scholar reported the size of the population (N = 530) but stated that an a priori sample size-based effect size wasn’t needed since the faculty was planning on performing a census. What? Surveying a population is fine; however, one should have some idea on the expected effect size so if less than the required number of samples are returned, follow-up efforts can be initiated to reach the desired level of practicality. This issue will rear its ugly head later.

Second, a total of 16 research hypotheses were explored. Of the 16, 8 IVs were ordinal, 4 were interval, and 4 were nominal. The DV was titled “Degree of Success” and was treated as an ordinal variable. In the study, the then emerging scholar decided to perform the analyses using the Kendall Rank-Order Correlation Coefficient (tau-b). For a quick review of this technique, see link. This is fine and appropriate if two variables are not normally distributed. However, what about the 3 IVs that are nominal? Kendall is not the right solution. These relationships should have been explored by one-sample t-tests; parametric or non-parametric. Wrong test…

Third, 16 IVs and 1 IV? Bonferroni anyone? With 16 IVs, a p-value of .003125 (.05/16) would be required so as not to make a Type I error caused by cumulative, family-wise hypothesis testing issues.

So, what happened? Well, let’s count the issues –

Only 31 responses were received. Was this expected? I bet not. Normally, a social science researcher shoots for an 80% SP (1-(4*p-value)). This study was severely under-powered to start (SP ~ .09). Perhaps if the then emerging scholar had calculated a sample size a priori… How big of sample was needed? Well, to identify a moderate effect size with a p-value of .003125, about 155 observations were needed. I wonder how much time it would have taken to get the N = 31 closer to N = 155?
A p-value of .05 was used, not a p-value of .003125. Thus, any statistically significant results reported had a high probability of being a Type I error.
Guess what? One statistical result was reported (tau-b = .322, p = .037). I’m not going to list it since it should be ignored
There may be something to report in this study, but since descriptive statistics were NOT reported relating to the sample, it’s hard to tell.

I’m thinking about asking for this faculty’s data. Perhaps there is something there…who knows?

Does turnover affect an organization or an organization’s executive?

As part of a larger undertaking on exploring how case study research designs are applied by doctoral students and approved by faculty at some universities, I came across an interesting viewpoint on employee turnover. Onyenacho (2019) stated the following problem that led to her research –

Physician turnover is costly for outpatient health care executives (Fibuch &
Ahmed, 2015). Outpatient health care executives lose 2 to 3 times the physician’s annual
salary when replacing a physician (Shanafelt, Goh, & Sinsky, 2017; Shanafelt &
Noseworthy, 2017), and lost revenue for physician replacement is $990,000 per full-time
equivalent physician, with an organizational cost to replace a physician ranging from
$500,000 to $1,000,000 (Shanafelt et al., 2017). The general business problem was that
some outpatient health care executives experienced high physician turnover, which led to
increased cost. The specific business problem was that some outpatient health care
executives lacked strategies to reduce physician turnover.
Onyenacho, 2019, p. 3 (emphasis added )

My question: Does an individual experience turnover or does an organization experience turnover?

I suppose if a specific health care executive’s compensation or continued employment with an organization is connected to physician turnover rates within the organization, then the first statement would be true. But as I read on, I realized the emerging researcher is attributing or assigning physician turnover to a specific person in an organization rather viewing employee turnover as an organizational artifact. That’s a unique perspective.

I have an idea on why she phrased her problem statement that way. It relates to her selected research design, but that’s a different discussion.

Reference:

Onyenacho, M. A. (2019). Strategies outpatient health care executives use to reduce physician turnover (Doctoral Dissertation). PQDT Open.

Does usage = success in marketing?

I stumbled across Davis (2020) because the title intrigued me. There are many different types of small businesses, and I wondered how the student was going to assess success in a qualitative, multiple-case study approach. Skipping to the interview guide, I see the problem. The novice researcher is having the participants answer the research question (my comments are in blue) –

What are your primary marketing strategies that have helped you grow and sustain your business beyond the first 5 years of operation? First, the novice researcher appears to want the participant to do researcher’s job. Why not start with “What types of marketing strategies to do you utilize?” From there, a researcher can explore how much is spent, the components of the strategy, the timing of the expenditure, how results are measured, etc. Second, shouldn’t the researcher look at all marketing strategies used, not just the ones that “helped them grow?” That’s the purpose of a multiple case study; to compare how different units (companies, in this case) operate?
What strategies were most effective toward your marketing efforts? I would think the novice researcher would do this. What is the role of the researcher in this study then?
How did you address the key barriers you encounter when implementing marketing strategies? The researcher “primed” the participants to discuss obstacles without learning if they had any. Why do I say that? Because this interview guide was created a priori. I could see discussing obstacles, such as lack of funds, knowledge or experience, in the context of a discussion about marketing; however, how could one see into the future to “know” a barrier was encountered. Student Note: Know your research method and design before you start your data collection.
How did the business skills you possess facilitate the effective implementation of the successful marketing strategies? Again, this would be researcher’s job to answer this question once the question of effectiveness was examined. For example, if a business owner thinks Strategy X is effective, but upon investigation and analysis of data it is learned that Strategy X is not more effective than Strategy Y and Z, then perhaps a business owner’s knowledge of marketing or the measurement should be included.

According to the novice researcher, four themes ’emerged’. Below are the themes and anecdotal quotes cited in the study. Again, my comments are in blue –

Social media
- “A majority of business owners use social media as part of their branding and marketing strategy” (p. 66). So…if a majority of businesses use something, it must be effective, right?
- P3 states “I find that I get the greatest type of traction with Intragram” What does traction mean? What information does the participant use to assess traction? How does the participant measure effectiveness with Instagram? Do the number of hits or likes on a web site translate to an increase in foot traffic, purchases, or sales? Where’s the data to support a conclusion that social media is effective? The novice researcher states this theme aligns with the work of He et al (2017), another QUAL study, but based on the supporting information its pure speculation.
Collaborations
- P1 described their relationship with a local humane society (see Incentive Marketing below).
- P3 described how they leave their business cards with local businesses. I wouldn’t call that a partnership. But based on these two items, the novice researcher made this conclusion: “Small business owners who develop partnerships with other small business owners in their community are an effective marketing strategy in spreading brand awareness” (p. 71). Studies examining the antecedents of brand awareness have found that organizational inducements via the marketing mix, marketing inducements (e.g., word of mouth advertising, promotion), and consumer experiences have statistically significant effects. These concepts were not covered by the novice researcher in the review of the literature. In fact, the term “brand awareness” was used only once in the study.
Incentive marketing
- P1 states “So every new adopter at the local human society receives a $10 gift certificate to my store, and I’d donate that gift certificate. That was also one of the main strategies I used.” How does one know if a strategy is successful until it’s examined and measured?
- P3 stated that “promotions, giveaways, and asking to tag three or more friends is very, very effective.” Not effective, but very, very effective(!). Since P3 probably doesn’t have a terminal degree in marketing, I would think someone exploring marketing in a terminal degree program, like this novice researcher, would dig deeper and ask questions about how effectiveness was measured. How long does the effect last? Is a ‘pulsing’ strategy used so as not to waste promotional funds when an effect is still active? I guess we’ll never know.
Word-of-Mouth
- P3 stated that “word of mouth was being leveraged better than the digital boards.” Here was an opportunity to explore with the business owner the depth of their word of mouth strategy, and validate the effect with data found in the corporate records (e.g., do they track referrals in their sales/accounting system?).

Using the novice researcher’s own words, these four themes were “primary strategies used by participants that play a key role in enhancing their marketing strategies for their business” (p. 66), and “the results of the study provided marketing strategies that small business owners can use to help sustain their businesses beyond 5 years” (p. 75). Perhaps the title of the study should have been: Research Strategies used by Four Small Business Owners in the Midwestern United States. But, who would read a study, let alone award a terminal degree, for a simple list.

A multiple case study research design was proposed, but not executed correctly. No information was provided about each business (e.g., products or services sold, revenue, date of origin), nor how much is spent on marketing by type of strategy (either by absolute $ or as a % of revenue). The interview guide was poorly developed. As a result, the novice researcher didn’t explore marketing strategies that were successful for each business in the case; she simply asked each business owner a series of questions about marketing activities they performed. This is another example of a non-case, case study.

I can’t believe the committee would have let this study go through, so I examined the terminal degree of each member. I’ll let you decide if this University (my alma mater) or committee did the student a disservice:

Chairperson: Terminal degree in organizational leadership
Committee member: Terminal degree in education
Committee member: Terminal degree in management

With nobody on the committee with a marketing degree, the faculty were probably doing the best they can and relying on the novice researcher to be an expert in the discipline. The novice researcher may be an expert in marketing, but did not demonstrate that expertise, nor how to perform a multiple case study research design, in this study. Note to Students: The results of this study should be viewed with caution, it not ignored in their entirety.

Reference:

Davis, J. (2020). Successful marketing strategies for small business sustainability (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (28086062)

He, W., Wang, F. K., Chen, Y., & Zha, S. (2017). An exploratory investigation of social media adoption by small businesses. Information Technology and Management, 18(2), 149-160. An exploratory investigation of social media adoption by small businesse

Converting a qualitative interview guide to a survey?

Recently, I reviewed a doctoral proposal where a student cited results from a peer-reviewed article. In the article Startup Success Trends in Small Business Beyond Five-Years: A Qualitative Research Study (Perry, Rahim, & Davis, 2018), the authors described how they interviewed 20 hair salon owners in New Jersey to explore how their businesses survived beyond five years. What caught my eye in this study was a reference to using “six questions” and SurveyMonkey. This piqued my interest, so I read deeper. After completing my reading, I began to think that this article read like a mini-dissertation. I found the lead author’s dissertation published in ProQuest (Perry, 2012), and it mirrored the three-author article. The second-listed author was the student’s chairperson. I don’t know the role of the third author of the study.

I began my first read of the Perry dissertation. He represented that he was extending research of another doctoral student – Schorr (2008). Rather than bias my view of Perry’s dissertation too much, I shifted to reading Schorr. Schorr performed a phenomenological inquiry into the “essence” of a successful entrepreneur by interviewing 10 entrepreneurs. Schorr included 100 pages (!!!) of contextual description and interview notes in his Appendix H. In my opinion, he performed a qualitative study properly in that he obtained deep, rich narrative descriptions from his participants. He used those descriptions as a basis for his thematic development. He integrated quotes from the descriptions in his study so others can assess the quality of the themes. Now that I read Schorr, I moved back to Perry.

Well…what a difference! Let me list and comment on some problems I have with this study –

Perry makes reference to obtaining Schorr’s approval to use his interview “questions.” It always amazes me that students try to use the same requests for information (I hate the term ‘questions’) and expect the same results in a qualitative inquiry. Can a qualitative researcher extend another qualitative researcher’s work by merely mirroring the same starting point? Wouldn’t follow-up inquiries unique to each participant’s narrative, individual researcher interpretation of each interview, and researcher observational notes render an extension impossible? It might be acceptable in some disciplines to use the same starting point as another researcher in a qualitative study; however, based on the response from the participant, this is where each interview (and study) can and, most likely, will diverge. Student Note #1: Become a near-expert in your selected methodology before you begin data collection.

It appears Perry didn’t go beyond the initial interview ‘starter’ questions. Qualitative research is about making sense of deep, rich narratives provided by participants via interviews; interviews that could take hours to complete. Not to drain the energy of the participant or the researcher, interviews often occur over a series of days or weeks. See Student Note #1.

The researcher used SurveyMonkey to distribute an open-ended ‘survey’ to participants. What?!?!? How can a researcher obtain deep, rich narrative descriptions in a Q&A format? Perry, citing McCoyd and Kerson (2006) as the source for this type of approach, wrote “an electronic open-ended survey allowed for more convenience for participants than face-to-face interviews and is just as reliable and accurate as face-to-face interviews” (pp. 51-52; emphasis added). I read the article cited and, in my opinion, Perry misrepresented the substance of the article. McCoyd and Kerson were exploring computer-mediated communication in qualitative research and were comparing email interviews with face-to-face interviews in social work. Surveys were not part of their research.

I reached out to Dr. Judith McCoyd, the co-author of the referenced study and Associate Professor at the Rutgers University School of Social Work, to get her opinion on the author’s characterization. She responded to my email with the following comments –

In the 2018 article by Perry, Rahim and Davis, it is asserted that my article with Toba Kerson “noted that an electronic open-ended survey allows for more convenience for participants, but is just as reliable and accurate as face-to-face interviews.” That is not accurate at all.
Our article compared the experience of long, multi-occasion, prolonged interview engagement by email with bereaved women to single face-to-face or telephone interviews and found that the data were much richer and more nuanced, as well as lengthier, when collected over an extended period of time and in multiple interactions tailored to explore the respondent’s earlier answers. Under NO circumstances would a one-shot Survey Monkey (or any survey method) be able to do the same thing.
I also find the assertion of phenomenological understandings (from a survey!) unbelievable on their face. Phenomenology requires sustained and engaged interaction to gain a deep understanding of the phenomena being explored. Qualitative research methodologists are clear about this. Again, there is no way to do that with a survey. Although the authors may have gotten some degree of detail in some of the responses to their survey, that is NOT the same thing as an interview process that is iterative and allows clarification and deepening of responses. Narrative data is qualitative, but it can never be fully developed without some degree of interaction or iterative ability.
Small sample sizes are common in qualitative research using ethnographic methods or intensive interviews, perhaps. However, a Survey Monkey instrument is not the same as an intensive interview protocol, regardless of the authors’ astonishingly uninformed claims. Interviews require interaction and probing to get to the heart of how the phenomena under study unfold. These involve fully exploring a respondent’s initial response in order to understand any complexity, ambivalence, or nuance more fully.
I am distressed that my findings were misrepresented and concerned that peer reviewers did not catch this error. Further, there are such obvious methodological problems that I am surprised that this was published.
I feel saddened for a scholar who was not mentored well enough to know that all cited literature should be correctly portrayed. Additionally, this is a study that needed to be framed in much more humble ways. A survey of 20 may suggest common features that allowed those hair salons to thrive where others failed, but it is certainly not phenomenological, conclusive, nor generalizable.
personal correspondence with J. L. M. McCoyd, September 8, 2020

Student Note #2: Read carefully so as not to mischaracterize another’s research.

What does this all mean? The results of Perry et al. (2018), which are simply the results of Perry (2012) repackaged, should be ignored due to a lack of internal validity caused by a mischaracterization of research which led to a poor research design; specifically, failing to perform in-depth interviews.

Reference:

McCoyd, J. L. M., & Kerson, T. S. (2006). Conducting intensive interviews using email: A serendiphttps://cran.r-project.org/web/packages/tm/tm.pdfitous comparative opportunity. Qualitative Social Work, 5(3), 389-406. https://doi.org/10.1177/1473325006067367

Perry, A. S. (2012). Determining the keys to entrepreneurial sustainability beyond the first 5 years (Doctoral dissertation). ProQuest Dissertations and Theses database. (UMI No. 3552459)

Perry, A., Rahim, E., & Davis, B. (2018). Startup success trends in small business beyond five-years: A qualitative research study. International Journal of Sustainable Entrepreneurship and Corporate Social Responsibility, 3(1), Article 1. https://doi.org/10.4018/ijsecsr.2018010101

Schorr, F. (2008). Becoming a successful entrepreneur: A phenomenological study (Doctoral dissertation). ProQuest Dissertations and Theses database. (UMI No. 3326847)

N = 2 in a non-case study Case Study

A student recently referenced a dissertation that focused on strategies that could be used to promote a sustainable business beyond 5 years (Johnson-Hilliard, 2015). What struck my interest was the size of sample: 2!!! I get it. In qualitative research, it’s not about the number of participants per se but the depth data collection and analysis. As I read on, the student frames the study as a multiple case study. In a multiple case study, an N = 2 could be appropriate where two businesses are compared and contrasted. However, upon further reading, the novice researcher conducted semi-structured interviews with the owners of the business. Reviews of financial statements, market-related factors (e.g., location, competition), or marketing-related artifacts (clients acquired by quarter, advertising mediums, customer lifetime value analyses), dimensions associated with success in business literature, were not performed. She did mention that she reviewed each company’s business plan to “verify if they are on the right path to potential risks or rewards” (p. 54). Besides the Unit of Focus changing from business to business owner, I guess the researcher will also be able to ‘see the future’ regarding potential risks and rewards.

The overarching research question in this study was – What strategies do salon business owners need to succeed in business beyond 5 years? Next, let’s look at the the interview guide (My thoughts are in blue)

What strategies do you use to enhance growth of your business? (The novice researcher is requiring each participant to provide a list of strategies. Do they know what a strategy is? I suppose it’s easier to ask a participant to provide a list rather than dig through documents and transcripts to determine which strategies were key)
How important is having a strategy to you as a small business owner? (Wait! First, tell me what strategies (if any) you employ to enhance growth, then tell me their importance? Does that mean some strategies are not important? Which strategies didn’t enhance growth?)
How do you compete with larger salons? (The size of the competitors salon was just primed by the novice researcher. The participant is now instructed to ignore small- or equal-sized competitors, potentially next door, and discuss how they compete with larger competitors. Hopefully, this line of inquiry leads to a series of marketing-related themes)
What are the causes of negative challenges in salons? (What’s a negative challenge? It’s not defined by the researcher. I searched for the term in some academic literature and couldn’t find anything. If not defined by the researcher, who knows how this will be interpreted by the participants. Taking that into consideration, is the researcher now having the participants speak for the entire industry? Did the scope of the inquiry just change?)
What are some gains or losses of being a successful business owner? (Another swerve from the identification of success factors in operating a salon for 5+ years to entrepreneurship rewards and sacrifices)
What additional information can you provide to assist me in understanding successful salon operations? (A throw-away request for information. With no follow-up to any of the other five items, who cares at this point…)

Before I dig into the themes, I find it troublesome when no details are provided about each business. How long have they been in business? What is their revenue? How many employees? Where are they located in Savannah, generally speaking? How many clients do they see in an average week? What is the average sale? Nothing to tell the reader anything about the businesses so they can decide whether to ignore the study and its results or potentially apply it to different situations.

Now, the themes –

Theme 1: Key Strategies for Salon Owners to Succeed in Business beyond 5 years – Yes, the first theme was the research question. Regardless of the questionable title (Who reviewed this study?), the researcher listed three strategies: Education, Training, and Skills. This makes sense since one has to be licensed by the State of Georgia to practice and maintain a record of Continuing Education. But should a “key strategy” be to make sure you are licensed in the State? It would appear that licensure would be the entrance to the field.

Theme 2: Effective Strategies for a Successful Business – Again, a questionable title; however, three strategies were listed: Customer Service, Niche Marketing, and Technology. The novice researcher reported that P1 stated “she employs excellent customer service in her establishment to all customers” (p. 70). What does “excellent customer service” mean? Isn’t that a self-serving statement? Is somebody going to say they don’t provide excellent customer service? This is an example of a novice researcher “reporting” what people say and calling it a theme rather than a participant describing the customer interaction process and the researcher characterizing the level of customer service. Next, niche marketing. The novice researcher describes how P2 appeared to have a niche market in hair molds and pieces for clients that have lost their hair to cancer, etc. However, there is no reference to the % of sales attributed to this service. Finally, technology. P1 stated she doesn’t use technology to schedule appointments while P2 does. In a 50-50 situation, I don’t understand why this was included in the study.

Theme 3: Determination and Dedication – Both participants identified their own determination and dedication, in what could be described as “self-serving” statements, so the novice researcher “reported” it (pp. 72-73). I guess we’ll never know the components of determination and dedication the two business-owners displayed.

Theme 4: Professionalism – When reading the analysis, the comments made by the participants align with Theme 3. But I’m speculating that because both participants commented about “providing professional environment, service, and attitude” (p. 74), it appears this was another case of extracting words used by the participant and making it a theme.

Yin (2018) describes two types of multiple-case study designs: holistic and embedded. A holistic design focuses on a single unit of analysis, while an embedded design involves looking at multiple units (p. 48). Without an explanation on the size and complexity of the two businesses, its difficult to determine which would have been appropriate; however, simply asking the business owners their thoughts fails both design models.

I don’t know whether to place responsibility on the quality (or lack thereof) of this study on the student, chairperson, committee, or University (my alma mater). This is an example of how the peer-review process can fail an emerging researcher. Regardless, the results of this study should be ignored due to internal validity issues.

References:

Johnson-Hilliard, M. (2015). Small business sustainability in the salon industry (Doctoral dissertation). ProQuest Dissertations Publishing. (3736144).

Yin, R. K. (2018). Case study research and applications: Design and methods (6th Ed.). SAGE Publications.