DOC 723 Week 2 Evaluating Studies Example

Not All Evidence Is Equal: Evaluating Studies for a Review on Customer Onboarding

[Student Name]

University of Phoenix

DOC/723: Doctoral Seminar II

Week 2 Assignment

[Instructor Name]

[Date]

The learner, the company and the studies described as composites are written for a model paper; cited works are real.

Week 1 planned the literature search for the onboarding dissertation at the invented Texas software firm. The pilot searches have produced a first set of candidate studies to judge. Before synthesizing them, the learner must decide how much each deserves to be trusted and how much each bears on her questions, since a review that treats every source alike will mislead its readers.

Why Appraise

Rousseau et al. (2008) argued that management research is often used without systematic attention to its quality and relevance, and called for evidence-based management in which practitioners and scholars gather, appraise and synthesize the best available evidence for specific questions. Briner et al. (2009) clarified that evidence-based management combines the best available research evidence with organizational data, practitioner expertise and stakeholder concerns, and that it requires critical appraisal rather than uncritical use of published findings. For a dissertation whose findings could guide how a company spends money on customer success, uncritical use of weak evidence would be a real risk.

Appraisal Criteria

Quantitative studies: design strength, from experiments through quasi-experiments and longitudinal designs to cross-sectional surveys; sample size and representativeness; measure reliability and validity; control of alternative explanations; and appropriate analysis.

Qualitative studies: fit between question and method; transparency about sampling, data and analysis; credibility strategies; and depth of interpretation.

Conceptual and review articles: coherence of argument; grounding in prior research; clarity of propositions; and, for reviews, systematic methods.

All sources: relevance to the review's questions, especially subscription or business-to-business settings and small firms; and independence from commercial interests.

Appraising Five Representative Sources

Ascarza et al. (2018), a review of customer retention management by leading marketing scholars. Strengths: broad synthesis, clear framework, attention to targeting interventions. Limits: draws mostly on consumer and telecommunications contexts; not systematic in method. Quality: high for a conceptual review. Relevance: high for framing the problem; moderate for small-business software.

Bhattacherjee (2001), a model of information systems continuance tested with survey data from users of online banking. Strengths: theory-grounded model linking confirmation of expectations, perceived usefulness and satisfaction to continued use; widely replicated. Limits: cross-sectional, intention rather than actual continuance, individual consumers rather than firms. Quality: moderate to high. Relevance: high for theory of why customers keep using software.

Payne et al. (2008), a conceptual framework for managing value co-creation between suppliers and customers. Strengths: clear processes for supplier, customer and encounter; strong grounding in service-dominant logic. Limits: conceptual, not tested. Quality: high as conceptual work. Relevance: moderate to high, since onboarding is a supplier-led encounter process.

A composite example of a cross-sectional survey of 140 software customers recruited through a vendor's newsletter, reporting that onboarding satisfaction correlates with renewal intention. Strengths: directly on topic. Limits: convenience sample, self-reported intention, single source, vendor sponsorship. Quality: low. Relevance: high. It will be noted, not relied on.

A composite example of a qualitative study of eighteen small-business owners' experiences adopting cloud accounting software. Strengths: rich accounts of time pressure and reliance on outside help. Limits: one country, adoption rather than continued use. Quality: moderate to high on qualitative criteria. Relevance: moderate.

The most relevant study on the list was the weakest one, which is exactly why relevance and quality must be rated separately.

The Evaluation Matrix

Each source has one row with columns for citation, purpose, theory, design, sample and setting, measures, key findings, quality rating, relevance rating, notes on limitations and how the source will be used: anchor, support or context. The matrix is kept in a spreadsheet so it can be sorted by quality, relevance or concept area as the synthesis develops, and so the committee can quickly see which sources carry the review's main claims and which serve only as background.

How Appraisal Shapes the Synthesis

High-quality, high-relevance sources anchor claims in the review. High-quality but less relevant sources, such as consumer retention studies, inform theory and methods but are presented with attention to the difference in setting. Low-quality sources are cited only when nothing better exists, with their limits stated in the same sentence. Where strong and weak studies disagree, the review will give more weight to the stronger evidence and explain its reasons openly.

Common Weaknesses in This Literature

Appraising the first forty sources revealed patterns. Many quantitative studies of retention measure intention to continue rather than actual renewal, which overstates links because people who say they will stay do not always stay. Many use single-source surveys, in which the same respondent reports both the predictor and the outcome, inflating correlations. Studies of customer success practices are mostly descriptive or conceptual, with few that contrast customers given a practice with customers not given it. And small businesses appear mainly in adoption research, rarely in retention research. These patterns matter for the review's argument: they show not only what is known but how weakly some of it is known.

Strengths Worth Building On

The literature also has strengths the dissertation can use. Contractual churn research in telecommunications and banking has developed careful methods for modeling time to cancellation with large administrative data sets. Continuance theory offers a tested explanation for why users keep using software after adopting it. And value co-creation frameworks describe what suppliers can do in early encounters. The review will draw on these strengths while noting their settings.

Recording Uncertainty

Some appraisals are uncertain, especially for studies that report methods briefly. Rather than force a rating, the matrix includes a confidence column, so a study rated moderate quality with low confidence can be revisited if its findings become central to the argument.

Practitioner Evidence

Industry surveys and vendor reports on customer success are numerous. They are appraised on a separate scale for transparency of methods, sample description and independence from vendors, and they appear in the review only to describe practice, never as evidence that a practice works.

Why Rate Relevance Separately

A study can be excellent and still say little about the dissertation's questions. Large retention studies of mobile phone subscribers are methodologically strong, but consumers choosing phone plans differ from small-business owners deciding whether to keep software that runs their appointments and billing. Rating relevance separately keeps such studies in their proper role: valuable for methods and theory, cautious for conclusions about small business customers.

Consistency

To check consistency, a classmate will appraise ten sources independently using the same criteria. Differences will be discussed, and criteria will be clarified where ratings diverged, with each clarification noted in the matrix guide so later ratings follow the same rule.

Conclusion

Clear criteria for different kinds of research, applied specifically to each source and recorded in a sortable matrix, let the review weigh evidence by quality and relevance. The five examples showed that the most relevant evidence is not always the strongest evidence available, which will shape how the review presents what is known. Week 3 turns to synthesizing the appraised literature by theme.

References

Ascarza, E., Neslin, S. A., Netzer, O., Anderson, Z., Fader, P. S., Gupta, S., Hardie, B. G. S., Lemmens, A., Libai, B., Neal, D., Provost, F., & Schrift, R. (2018). In pursuit of enhanced customer retention management: Review, key issues, and future directions. Customer Needs and Solutions, 5(1-2), 65-81. https://doi.org/10.1007/s40547-017-0080-0

Bhattacherjee, A. (2001). Understanding information systems continuance: An expectation-confirmation model. MIS Quarterly, 25(3), 351-370. https://doi.org/10.2307/3250921

Briner, R. B., Denyer, D., & Rousseau, D. M. (2009). Evidence-based management: Concept cleanup time? Academy of Management Perspectives, 23(4), 19-32. https://doi.org/10.5465/amp.23.4.19

Payne, A. F., Storbacka, K., & Frow, P. (2008). Managing the co-creation of value. Journal of the Academy of Marketing Science, 36(1), 83-96. https://doi.org/10.1007/s11747-007-0070-0

Rousseau, D. M., Manning, J., & Denyer, D. (2008). Evidence in management and organizational science: Assembling the field's full weight of scientific knowledge through syntheses. Academy of Management Annals, 2(1), 475-515. https://doi.org/10.5465/19416520802211651