Quantizing qualitative data outside of a mixed methods study?

As part of an upcoming paper on the quality of doctoral research in DBA programs, I came across a study where the emerging scholar did one of my pet peeves: quantizing qualitative information outside of a mixed method research design. As a result, I employ techniques performed in a prior post to illustrate how this approach can be detrimental to the effect of a study.

Wagner (2019) explored the effectiveness of the Department of Defense’s Transition Assistance Program (TAP) as it related to California veterans. According to the author’s sources, California is home to over 1.8M veterans; 230,000 of those serving post-9/11. The emerging scholar used a questionnaire, rather than an interview guide, to collect data. From information collected via questionnaire (N = 10), the scholar coded participants responses into three levels: high, moderate, and low regarding the participants confidence in a service provided by the TAP. For example, relating to preparedness to transition from the military to the private sector (Appendix E, p. 121), the scholar coded responses into three categories –

  • High – Veteran utilized resources provided by TAP for financial health, relocation, and career search
  • Moderate – Veteran able to locate financial, networking, and relocation resources, but did not use
  • Low – Veteran did not know where to (sic) locate essential transition resources.

Once coded, two faculty reviewed the coding and confirming the ‘classification’ (pp. 52-53). Student Note: This type of step is needed for internal validity in some research designs.

I chose one item for illustration purposes, but the other items are similar in form and content. Participants were asked about drafting a basic resume (see pp. 53-57). From their responses, the emerging scholar classified them into the three levels of confidence and created a cross tabulation of the results. Based on these results, the scholar stated –

The majority of respondents (60%) had a high degree of confidence in drafting a basic resume and cover letter after participating in TAP.

Wagner, 2019, p. 53

Later, the scholar wrote –

High confidence was exhibited by participants who felt empowered by the TAP workshop and were capable of drafting a basic resume. Moderate confidence was demonstrated by veterans who obtained skills to create a basic resume while veterans with low confidence struggled to translate their military career to a basic resume or lacked focus

Wagner, 2019, p. 54 (emphasis added)

By simply describing and interpreting the responses, and not quantizing them into levels, the emerging scholar’s analysis may have had more influence on a reader. There are still questions about the depth of inquiry (e.g., questionnaire vs. in-depth interviews), but that’s hard to explore without obtaining transcripts. However, when quantizing comes into play a reader has to consider the writer’s level of confidence in a “majority of respondents (60%)” statement based on a sample size of 10.

Using the information in Table 3 of the study, I added 95% CI error bars that equate to an N = 230,000 and an n = 10 (CI = 31%; Figure 1) –

Figure 1. Confidence in Drafting a Resume after receiving TAP training with 95% CI = 31% (N = 10)

As one can see, each level’s confidence interval covers the other levels, and detracts from the effect of the study. Just for the record, there was no statistical difference between the three groups, X2(2) = 0.9722, p = .615, due to the small sample size.

This was not a qualitative study; this was merely a quantitative descriptive study. Using the author’s words, the analysis was based on data elicited from “17 formalized questions used during the interview process” (p. 89). Later, the author used the phrase ‘general consensus’ when describing how pre-2011 Veterans Opportunity Work Act participants felt that TAP was a “check in the box as part of out-processing” (p. 51), and TAP provided “adequate support to draft a basic resume and cover letter” (p. 93). Had the scholar simply reported the descriptive statistics based on a larger population, he may have had something; however, would the University had granted a doctorate for that level of rigor?

From a management perspective, if you were in a position to redirect the TAP program and read this study, would you act on these types of results?

Student Note: Make sure you clearly align your research question, research method, and research design. Also, make sure you speak to several faculty members at your university who perform research to get their view on your proposed study methodology. Some faculty focus on only one type of method and try to stuff every study into that mold…right or wrong. Some faculty focus on certain types of QUAN or QUAL. Others only know the method they performed when they did their study. Heck, they may have done their study incorrectly…Remember: It’s your study and it will become a public record.

Reference:

Wagner, J. D. (2019). Effectiveness of the Transition Assistance Process (TAP) in building career self-efficacy for California post-9/11 veterans (Doctoral dissertation). ProQuest LLC. (13865682)

Luck, inadvertent omission, or lack of knowledge?

Johnson (2018) explored the willingness to hire people who were convicted of drug crimes. The scope of the study was limited to the Central Virginia region. To answer the first research question (How does the willingness to hire returning citizens by Central Virginia employers differ by position/job role in private sector, for-profit business firms?), the emerging scholar used descriptive measures (rather than inferential statistics).

Johnson stated that the null hypothesis could be rejected for three types of jobs: Unskilled, Semi-skilled Labor, and Skilled Labor. I suppose that the decision was based on point estimates above 50% (Figure 1) –

Figure 1. Willingness to hire by position/job role (Johnson, 2018, p. 56)

However, when rejecting null hypotheses based on sample data, confidence intervals must be considered. Based on the information provided by the emerging scholar in his study, there were 653,193 businesses in the sample frame. A quota sample of 635 was chosen (p. 35). Using R and the samplingbook package (Manitz, 2017), that equates to having a 95% CI of 3.89% (see below).

sample.size.prop(e = .0389, N = 653193, level = .95)

sample.size.prop object: Sample size for proportion estimate
With finite population correction: N=653193, precision e=0.0389 and expected proportion P=0.5

Sample size needed: 635

I then recreated the graphic using the ggplot2 package (Wickham et al., 2020), and added the 95% CI (Figure 2).

Figure 2. Recreated Willingness to hire by position/job role with 95% CI = 3.89%

Okay. I see it. However, only 105 complete responses were obtained, not the target sample of 635. Using the same method to calculate the 95% CI above, I backed into a 9.6% 95% CI (see below):

sample.size.prop(e = .096, N = 653193, level = .95)

sample.size.prop object: Sample size for proportion estimate
With finite population correction: N=653193, precision e=0.096 and expected proportion P=0.5

Sample size needed: 105

Thus, the 95% CI changed from a planned 3.83% to an actual 9.6%; a 2.5x increase in interval width. When overlaying the new 95% CI on the data, new perspectives emerge (Figure 3).

Figure 3 Willingness to hire by position/job role (Johnson, 2019, p. 56) with 95% CI = 9.6%

Visually, one can see that the emerging scholar is correct when stating that Semi-skilled Labor and Skilled Labor fall above the 50% line; even when accounting for the 95% CI. However, the error bar for Unskilled Labor (a) drops below 50%, and the error bar for Clerical Labor (d) rises above 50%. Should Unskilled Labor be omitted from the rejection? Should Clerical Labor be included in the rejection of the null hypothesis? It appears both a Type I and a Type II error occurred.

One note: The emerging scholar reported his findings as being similar to research performed in Sweden by Ahmed and Lang (2017). The authors wrote –

We found that ex-offenders were discriminated against in the occupations accounting clerk, cleaner, preschool teacher, restaurant worker, sales person, and software developer. However, we did not observe any statistically significant discrimination against ex-offenders in the occupations auto mechanic, enrolled nurse, and truck driver. 

Ahmed & Lang, 2017, p. 17

Well, they don’t now. Also…Virginia = Sweden? That may be a stretch…

Student Note: Descriptive statistics are not inferential statistics. Know the difference.

References:

Ahmed, A., & Lang, E. (2017). The employability of ex-offenders: A field experiment in
the Swedish labor market. IZA Journal of Labor Policy, 6(1), Article 6. https://doi.org/10.1186/s40173-017-0084-2

Johnson, R. (2018). Willingness of employers to hire individuals convicted of drug crimes in Central Virginia (Doctoral dissertation). ProQuest LLC. (13421921)

Manitz, J. (2017, May 21). samplingbook: Survey sampling procedures. https://cran.r-project.org/web/packages/samplingbook/samplingbook.pdf

Wickham, H., Chang, W., Henry, L., Pederson, T. L., Takahashi, K., Wilke, C., Woo, K., Yutani, H., Dunningham, D., & RStudio (2020, June 19). ggplot2: Create elegant data visualizations using the Grammar of Graphics. https://cran.r-project.org/web/packages/ggplot2/ggplot2.pdf