As part of an upcoming paper on the quality of doctoral research in DBA programs, I came across a study where the emerging scholar did one of my pet peeves: quantizing qualitative information outside of a mixed method research design. As a result, I employ techniques performed in a prior post to illustrate how this approach can be detrimental to the effect of a study.
Wagner (2019) explored the effectiveness of the Department of Defense’s Transition Assistance Program (TAP) as it related to California veterans. According to the author’s sources, California is home to over 1.8M veterans; 230,000 of those serving post-9/11. The emerging scholar used a questionnaire, rather than an interview guide, to collect data. From information collected via questionnaire (N = 10), the scholar coded participants responses into three levels: high, moderate, and low regarding the participants confidence in a service provided by the TAP. For example, relating to preparedness to transition from the military to the private sector (Appendix E, p. 121), the scholar coded responses into three categories –
- High – Veteran utilized resources provided by TAP for financial health, relocation, and career search
- Moderate – Veteran able to locate financial, networking, and relocation resources, but did not use
- Low – Veteran did not know where to (sic) locate essential transition resources.
Once coded, two faculty reviewed the coding and confirming the ‘classification’ (pp. 52-53). Student Note: This type of step is needed for internal validity in some research designs.
I chose one item for illustration purposes, but the other items are similar in form and content. Participants were asked about drafting a basic resume (see pp. 53-57). From their responses, the emerging scholar classified them into the three levels of confidence and created a cross tabulation of the results. Based on these results, the scholar stated –
The majority of respondents (60%) had a high degree of confidence in drafting a basic resume and cover letter after participating in TAP.
Wagner, 2019, p. 53
Later, the scholar wrote –
High confidence was exhibited by participants who felt empowered by the TAP workshop and were capable of drafting a basic resume. Moderate confidence was demonstrated by veterans who obtained skills to create a basic resume while veterans with low confidence struggled to translate their military career to a basic resume or lacked focus
Wagner, 2019, p. 54 (emphasis added)
By simply describing and interpreting the responses, and not quantizing them into levels, the emerging scholar’s analysis may have had more influence on a reader. There are still questions about the depth of inquiry (e.g., questionnaire vs. in-depth interviews), but that’s hard to explore without obtaining transcripts. However, when quantizing comes into play a reader has to consider the writer’s level of confidence in a “majority of respondents (60%)” statement based on a sample size of 10.
Using the information in Table 3 of the study, I added 95% CI error bars that equate to an N = 230,000 and an n = 10 (CI = 31%; Figure 1) –
As one can see, each level’s confidence interval covers the other levels, and detracts from the effect of the study. Just for the record, there was no statistical difference between the three groups, X2(2) = 0.9722, p = .615, due to the small sample size.
This was not a qualitative study; this was merely a quantitative descriptive study. Using the author’s words, the analysis was based on data elicited from “17 formalized questions used during the interview process” (p. 89). Later, the author used the phrase ‘general consensus’ when describing how pre-2011 Veterans Opportunity Work Act participants felt that TAP was a “check in the box as part of out-processing” (p. 51), and TAP provided “adequate support to draft a basic resume and cover letter” (p. 93). Had the scholar simply reported the descriptive statistics based on a larger population, he may have had something; however, would the University had granted a doctorate for that level of rigor?
From a management perspective, if you were in a position to redirect the TAP program and read this study, would you act on these types of results?
Student Note: Make sure you clearly align your research question, research method, and research design. Also, make sure you speak to several faculty members at your university who perform research to get their view on your proposed study methodology. Some faculty focus on only one type of method and try to stuff every study into that mold…right or wrong. Some faculty focus on certain types of QUAN or QUAL. Others only know the method they performed when they did their study. Heck, they may have done their study incorrectly…Remember: It’s your study and it will become a public record.
Reference:
Wagner, J. D. (2019). Effectiveness of the Transition Assistance Process (TAP) in building career self-efficacy for California post-9/11 veterans (Doctoral dissertation). ProQuest LLC. (13865682)
Dr. James Ready 