Measuring perception…

A week ago, a doctoral business student was referred to me for a discussion about a specific research design. While the primary research question was somewhat convoluted, once edited the focus was on measuring perception of a behavior in a group through the lens of another (the sample), then relating that perception of the group to the sample’s perception of project performance. If one were to ask a survey participant a single question, it would be “What do you think the relationship is between A and B?” This type of question falls into the category (loosely) of (public) opinion research, not business research.

Research into perceptions (and attitudes) are often studied using a qualitative research methodology. The perception of something or someone is generally explored via interviews, where the researcher collapses groups of thoughts into themes that answer the research question. This type of approach is often covered in some depth in many research methods and design textbooks.

When it comes to quantitative research, though, measuring perception is focused on the self-assessment of a sample. For example, the Perceived Stress Scale for measuring the perception of stress, or the Buss-Durkee Hostility Inventory for measuring aspects of hostility and guilt; both instruments developed by psychologists.

Using a subject’s perception of another person is problematic due to cognitive bias. This type of bias involves the systematic error in one’s perception of others. Within cognitive bias, there are three groups –

  • Fundamental Attribution Error, which involves labeling people without sufficient information, knowledge or time
  • Confirmation Bias, which is widely referred to as a common judgmental bias. Research has shown that people trick their minds on focusing on a small piece of information to confirm already developed belief
  • Self-serving Bias, which involves perceiving a situation in a manner so to plays the one perceiving in a more positive light

How would you measure validity in the proposed study? Have the sample assess behaviors in people, measure the behaviors of the people, compare the two assessments for accuracy, and factor that accuracy into the study? Seems like a long way to go, and all you are really doing is measuring the assessment ability of the sample.

I don’t know who’s at fault here for not identifying this type of fundamental issue before the student’s research proposal was developed. It may have been identified by faculty along the way and ignored by the student? It could be faculty didn’t really understand what the student was proposing due to how the research question was formed?

Examining influence in a qualitative study?

It’s very important to use the correct terminology associated with a research method and design. For example, the word influence is widely-associated with quantitative research. Influence can be measured by examining the change in the Y variable when the X variable is manipulated or involving a third variable (Z). Quantitative research is more scientific, less subjective, is repeatable, and can be generalized. Qualitative research is based on the knowledge and skill of the researcher. There are times when an experienced researcher will explore influence in a qualitative study but that is few and far between, generally related to specific disciplines (e.g., medical, social work), and is supported with significant academic research (see here, here, and here). I don’t recommend emerging scholars perform qualitative research. Besides skill, the time needed to complete a qualitative study is much longer than the time needed to complete a quantitative study.

Grant (2019) is an example of why emerging scholars shouldn’t do qualitative research. This emerging scholar explored the influence of leadership behaviors on two dimensions: employee engagement and collaboration (the organization is not germane to this discussion). To perform this study, the emerging scholar created a 7-item open-ended survey and distributed it anonymously to 10 people in an organization exceeding 3,800 people. The emerging scholar would interpret the responses and categorize them to answer the following two research questions –

  • What leadership styles and behaviors are being utilized at [organization]?
  • What is the influence of existing leadership styles and behaviors on employee engagement and collaboration?

Yin (2018) describes five situations where a single case study would be appropriate: critical, unusual, common, revelatory, or longitudinal (pp. 48-50). In addition, Yin describes two types of single case studies: holistic and embedded (pp. 51-53). When reviewing the dissertation, the researcher is attempting a build a common, holistic single-case study. Common because leadership is an everyday situation. Holistic because the organization appears to have a single purpose. However, a case study focuses on “how” or “why” a situation occurred (perhaps leadership style evolution); not “what” style is prevalent or which specific styles influence two outcomes. With an anonymous survey, there is no way to follow up with a participant to clarify their responses. To quote a colleague –

Who’s the researcher? Carnac the Magnificent?

Name withheld

As a result, the research method (QUAL) and design (case study) doesn’t appear to align with the research questions. The results of the study should be ignored. However, I wanted to discuss the themes identified by Grant –

  • A collaborative, or transformational, leadership style is present
  • Organizational leaders are engaging
  • Unfair hiring practices have become standard

First, are collaborate and transformational the same? They’re close, but I believe some scholars would say they’re different. Second, what does the organization’s hiring practices have to do with leadership in an organization? Plus, how can one generalize to an organization of 3,800+ from a sample of 10? Do the math: That’s a 95% CI of nearly 31 points! Even if 90% of the sample described an organization leaders as collaborative, as interpreted by the researcher, that means the 95% CI would between 60% and Inf. What are the other 40%? Non-collaborative?

Reference:

Grant, R. M. (2019). Investigating the influence of leadership behaviors on employee engagement and collaboration in a Federal organization (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (22615969)

Yin, R. K. (2018). Case study research and applications: Design and methods (6th Ed.). SAGE Publications.

Ethnicity: M = 1.26, SD = .529?

I understand that one has to “numericize” categories for quantitative analysis, but any student and faculty should understand the numbers mean nothing when compared (see Figure 1).

Figure 1: Barplot of Ethnicity (Race) with Distribution overlay (Deonarinesingh, 2019, p. 59).

A chairperson has to perform a lot of reading when reviewing a student’s dissertation. A committee member can hopefully help. But this type of error appeared on every chart in this study’s Chapter 4; regardless of the type of variables (e.g., categorical, interval). Did the faculty not know, or did they simply not read the study?

Student Note: Understand your variables and how best to display them. Don’t rely on your committee; they might not know or remember.

This study will return in a later post…stay tuned.

Reference:

Deonarinesingh, S. (2019). The effect of cultural intelligence upon organizational citizenship behavior, mediated by openness to experience (Doctoral dissertation). ProQuest Dissertations & Theses Global: The Humanities and Social Sciences Collection. (13880805)

N = 2 in a non-case study Case Study

A student recently referenced a dissertation that focused on strategies that could be used to promote a sustainable business beyond 5 years (Johnson-Hilliard, 2015). What struck my interest was the size of sample: 2!!! I get it. In qualitative research, it’s not about the number of participants per se but the depth data collection and analysis. As I read on, the student frames the study as a multiple case study. In a multiple case study, an N = 2 could be appropriate where two businesses are compared and contrasted. However, upon further reading, the novice researcher conducted semi-structured interviews with the owners of the business. Reviews of financial statements, market-related factors (e.g., location, competition), or marketing-related artifacts (clients acquired by quarter, advertising mediums, customer lifetime value analyses), dimensions associated with success in business literature, were not performed. She did mention that she reviewed each company’s business plan to “verify if they are on the right path to potential risks or rewards” (p. 54). Besides the Unit of Focus changing from business to business owner, I guess the researcher will also be able to ‘see the future’ regarding potential risks and rewards.

The overarching research question in this study was – What strategies do salon business owners need to succeed in business beyond 5 years? Next, let’s look at the the interview guide (My thoughts are in blue)

  1. What strategies do you use to enhance growth of your business? (The novice researcher is requiring each participant to provide a list of strategies. Do they know what a strategy is? I suppose it’s easier to ask a participant to provide a list rather than dig through documents and transcripts to determine which strategies were key)
  2. How important is having a strategy to you as a small business owner? (Wait! First, tell me what strategies (if any) you employ to enhance growth, then tell me their importance? Does that mean some strategies are not important? Which strategies didn’t enhance growth?)
  3. How do you compete with larger salons? (The size of the competitors salon was just primed by the novice researcher. The participant is now instructed to ignore small- or equal-sized competitors, potentially next door, and discuss how they compete with larger competitors. Hopefully, this line of inquiry leads to a series of marketing-related themes)
  4. What are the causes of negative challenges in salons? (What’s a negative challenge? It’s not defined by the researcher. I searched for the term in some academic literature and couldn’t find anything. If not defined by the researcher, who knows how this will be interpreted by the participants. Taking that into consideration, is the researcher now having the participants speak for the entire industry? Did the scope of the inquiry just change?)
  5. What are some gains or losses of being a successful business owner? (Another swerve from the identification of success factors in operating a salon for 5+ years to entrepreneurship rewards and sacrifices)
  6. What additional information can you provide to assist me in understanding successful salon operations? (A throw-away request for information. With no follow-up to any of the other five items, who cares at this point…)

Before I dig into the themes, I find it troublesome when no details are provided about each business. How long have they been in business? What is their revenue? How many employees? Where are they located in Savannah, generally speaking? How many clients do they see in an average week? What is the average sale? Nothing to tell the reader anything about the businesses so they can decide whether to ignore the study and its results or potentially apply it to different situations.

Now, the themes –

Theme 1: Key Strategies for Salon Owners to Succeed in Business beyond 5 years – Yes, the first theme was the research question. Regardless of the questionable title (Who reviewed this study?), the researcher listed three strategies: Education, Training, and Skills. This makes sense since one has to be licensed by the State of Georgia to practice and maintain a record of Continuing Education. But should a “key strategy” be to make sure you are licensed in the State? It would appear that licensure would be the entrance to the field.

Theme 2: Effective Strategies for a Successful Business – Again, a questionable title; however, three strategies were listed: Customer Service, Niche Marketing, and Technology. The novice researcher reported that P1 stated “she employs excellent customer service in her establishment to all customers” (p. 70). What does “excellent customer service” mean? Isn’t that a self-serving statement? Is somebody going to say they don’t provide excellent customer service? This is an example of a novice researcher “reporting” what people say and calling it a theme rather than a participant describing the customer interaction process and the researcher characterizing the level of customer service. Next, niche marketing. The novice researcher describes how P2 appeared to have a niche market in hair molds and pieces for clients that have lost their hair to cancer, etc. However, there is no reference to the % of sales attributed to this service. Finally, technology. P1 stated she doesn’t use technology to schedule appointments while P2 does. In a 50-50 situation, I don’t understand why this was included in the study.

Theme 3: Determination and Dedication – Both participants identified their own determination and dedication, in what could be described as “self-serving” statements, so the novice researcher “reported” it (pp. 72-73). I guess we’ll never know the components of determination and dedication the two business-owners displayed.

Theme 4: Professionalism – When reading the analysis, the comments made by the participants align with Theme 3. But I’m speculating that because both participants commented about “providing professional environment, service, and attitude” (p. 74), it appears this was another case of extracting words used by the participant and making it a theme.

Yin (2018) describes two types of multiple-case study designs: holistic and embedded. A holistic design focuses on a single unit of analysis, while an embedded design involves looking at multiple units (p. 48). Without an explanation on the size and complexity of the two businesses, its difficult to determine which would have been appropriate; however, simply asking the business owners their thoughts fails both design models.

I don’t know whether to place responsibility on the quality (or lack thereof) of this study on the student, chairperson, committee, or University (my alma mater). This is an example of how the peer-review process can fail an emerging researcher. Regardless, the results of this study should be ignored due to internal validity issues.

References:

Johnson-Hilliard, M. (2015). Small business sustainability in the salon industry (Doctoral dissertation). ProQuest Dissertations Publishing. (3736144).

Yin, R. K. (2018). Case study research and applications: Design and methods (6th Ed.). SAGE Publications.