DOC 715 Week 6 Sampling and Access Example

Getting the Right Records the Right Way: Sampling and Data Access for an Onboarding Study

[Student Name]

University of Phoenix

DOC/715: Doctoral Seminar I

Week 6 Assignment

[Instructor Name]

[Date]

The learner, the company, its customers and all figures are composites written for a model paper.

Week 5 argued for propensity score matching as the core design, with a difference-in-differences check. This paper defines whose records will be studied, checks that there are enough and plans how the learner will obtain them properly from the company where she works.

Population and Frame

The target population is small service businesses that subscribe to scheduling and billing software. The accessible population is Hill Country Software's small service business customers. The sampling frame is all customer accounts for businesses with fewer than 20 employees in the eleven service categories defined in Week 3 that began a paid subscription between January 1, 2023, and June 30, 2025, about 12,000 accounts. The end date allows a full twelve months of follow-up for every account before data extraction begins.

Inclusion and Exclusion

Included: paid subscriptions in the frame. Excluded: test and internal accounts; accounts closed within thirty days for nonpayment, which reflect billing failures rather than retention decisions; franchise locations onboarded centrally by their franchisor, about 400 accounts, whose onboarding differs from both study conditions; and accounts reactivated after a prior cancellation, since their history differs. Each exclusion will be reported with counts in a flow diagram.

Sampling Approach

The study uses the full frame rather than a sample drawn from it. All eligible onboarded customers and all eligible automated-only customers enter the propensity score model; matching then selects the comparison group. Using the full frame avoids sampling error at the selection stage and maximizes the chance of finding close matches. Stuart (2010) notes that matching performs best when the pool of potential comparison units is large relative to the treated group, since close matches are then easier to find; here, about two automated-only customers exist for each onboarded one.

Sample Size and Power

About 3,900 eligible customers received proactive onboarding. Week 5 anticipated that some would lack close matches; a preliminary propensity model on anonymized data suggests about 3,600 matched pairs.

Faul et al. (2007) describe G*Power, which estimates power for many tests. For RQ1, a difference in twelve-month retention from 69 to 74 percent, a modest effect the company would consider worth acting on, requires about 1,300 customers per group for power of .80 at alpha .05; 3,600 pairs give power above .99. For RQ2, a difference of a quarter of a feature in mean adoption requires far fewer customers.

RQ3 is the constraint. Only about 900 matched pairs involve businesses with one to four employees. An interaction test has less power than a main effect test; simulation using the preliminary data suggests power of about .70 to detect a difference in effect of five percentage points between size groups. The learner will report this limitation in advance and interpret a nonsignificant interaction cautiously, as unsettled rather than as proof that size does not matter.

A sample of 7,200 customers is large for the main question and only moderate for the question that matters most for targeting.

What Happens if Matching Yields Fewer Pairs

If the final matched sample falls well below 3,600 pairs, because the propensity model reveals less overlap than expected, the main questions will still have ample power down to about 1,300 pairs. The size subgroup would suffer more. In that case the learner will consider a slightly wider caliper, report the trade-off between closeness of matches and sample size and present the weighted analysis from Week 5 as a check that uses all customers.

Site Permission

The company's chief operating officer will sign a site permission letter stating that the company permits use of de-identified customer records for the dissertation, that the learner may publish results without identifying customers, that the company may review drafts for confidential business information but may not alter findings and that the company may be named or anonymized at its choice.

Data Use Agreement and De-identification

A data use agreement will specify the fields to be shared, security requirements and destruction of data after the dissertation is complete. An analyst in the company's data team, outside the customer success department, will extract records, replace account numbers with study codes, remove names, addresses, email addresses and phone numbers and convert city to region. The learner will receive only the de-identified file, stored on encrypted company equipment and never on personal devices.

Responsible Use of Customer Data

Zook et al. (2017) proposed rules for responsible research with large data sets, including acknowledging that data are people and can do harm, recognizing that privacy is more than a binary value, guarding against re-identification and considering the context and expectations under which data were collected. Customers agreed to the company's privacy policy, which permits use of data to improve services, not to be research participants. The study's use falls within service improvement, but the learner will minimize fields, aggregate small groups and never report results for any group smaller than 20 accounts, so no customer can be identified.

The Learner's Insider Role

The learner leads the customer success team whose work is being evaluated. Risks: she could be tempted to shape data toward a favorable result, and managers on her team could feel their performance is being judged. Protections: the analysis plan is fixed before outcome data are released; extraction is done by the data team; manager identities are coded and no manager-level results will be reported to the company; and the plan goes to the university's review board, including whether the study requires a determination that it involves only de-identified data.

Data Quality Checks

Before matching, the learner will check the extracted file for duplicate accounts, impossible dates such as cancellations before start dates and missing values in each covariate. Week 4 found key fields complete for 96 percent of accounts; records missing a covariate will be handled with a missing-indicator approach in the propensity model and the pattern of missingness will be reported. Feature use logs will be compared against a sample of accounts reviewed manually to confirm that the logging captures actual use.

Timeline for Access

Site permission and the data use agreement will be signed before the proposal defense. After ethics board approval, the data team will need about three weeks to extract and de-identify records. The learner has confirmed the team's availability for that window.

Can the Sample Answer Every Question?

RQ1 and RQ2: yes, with ample power. RQ3: yes, but with moderate power, a limitation stated in advance and repeated in the results. All variables defined in Week 2 are available in the frame.

Conclusion

The full frame of about 12,000 eligible accounts, with stated exclusions, should yield about 3,600 matched pairs, ample for the main questions and moderate for the size interaction. Signed permission, a data use agreement, de-identification by an outside analyst, rules for responsible data use and protections for the learner's insider role make access ethical. Week 7 assesses feasibility, scope and boundaries.

References

Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175-191. https://doi.org/10.3758/BF03193146

Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25(1), 1-21. https://doi.org/10.1214/09-STS313

Zook, M., Barocas, S., boyd, d., Crawford, K., Keller, E., Gangadharan, S. P., Goodman, A., Hollander, R., Koenig, B. A., Metcalf, J., Narayanan, A., Nelson, A., & Pasquale, F. (2017). Ten simple rules for responsible big data research. PLoS Computational Biology, 13(3), e1005399. https://doi.org/10.1371/journal.pcbi.1005399