Research design, statistical tests and program evaluation for counselors
Home / Guides / NCE knowledge guides
Short answer. NCE research questions ask you to identify a design, spot the threat to validity and pick the right statistical test. Random assignment makes a true experiment; without it the design is quasi-experimental. To compare two group means use a t-test, three or more means a one-way ANOVA, counts in categories a chi-square, and a relationship between two continuous variables a correlation or regression. Program evaluation separates needs assessment, formative and summative evaluation, and process and outcome questions.
Source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs..
What research designs does the NCE expect you to recognize?
The first question to ask of any study is whether participants were randomly assigned to conditions (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). Random assignment spreads unknown differences evenly across groups, which is what lets a researcher claim the treatment caused the change. Campbell and Stanley sorted designs into three families.
| Family | Examples | What it can claim |
|---|---|---|
| Pre-experimental | One-shot case study; one-group pretest-posttest; static-group comparison | Very little about cause; many threats uncontrolled |
| True experimental | Pretest-posttest control group; posttest-only control group; Solomon four-group | Causal inference, because of random assignment |
| Quasi-experimental | Nonequivalent control group (intact classrooms or clinics); time series; interrupted time series | Tentative causal claims; selection is the main threat |
The Solomon four-group design adds two groups without a pretest so the researcher can test whether taking the pretest changed the results. Correlational designs measure variables as they occur and describe relationships, not causes. A correlation between hours of counseling and lower symptoms may reflect motivated clients attending more.
How do single-subject designs work?
A single-subject (single-case) design measures one person, or a small number, repeatedly across phases, so each participant serves as their own control (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). A is a baseline phase with no treatment, B is the treatment phase.
- AB: baseline then treatment. Weak, because anything that happened at the switch could explain the change.
- ABA and ABAB (withdrawal or reversal): treatment is removed and restored. If the behavior improves with each B phase and slips back in A, the treatment is the likely cause. ABAB ends on treatment, which is better ethically.
- Multiple baseline: treatment starts at staggered times across behaviors, settings or people. Use it when withdrawing treatment would be unethical or the skill cannot be unlearned.
- Changing criterion: the target level is raised in steps, and behavior should track each new criterion.
Results are usually judged by visual inspection of the graph (level, trend, variability) rather than by significance tests. The repeated measure is often a brief symptom scale or a behavior count; the assessment instruments page lists common measures.
What are the main qualitative traditions?
Qualitative research studies meaning and experience in depth with small, purposefully chosen samples. Phenomenology asks what an experience is like for those who live it, grounded theory builds a theory from coded interview data, ethnography studies a culture from inside, a case study examines a bounded case in detail and narrative research works with life stories (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). Trustworthiness takes the place of validity: credibility (member checks, triangulation), transferability (thick description), dependability (audit trail) and confirmability (reflexivity). The research practice page has a short table of these terms.
What threatens internal and external validity?
Internal validity asks whether the treatment caused the effect; external validity asks whether the result generalizes to other people, settings and times (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). Campbell and Stanley's lists are still the reference.
| Threat | Type | Example in a counseling study |
|---|---|---|
| History | Internal | A campus suicide during a prevention study changes everyone's mood |
| Maturation | Internal | Children's social skills improve over a school year with or without the group |
| Testing | Internal | Clients score better on the posttest because they remember the pretest |
| Instrumentation | Internal | Raters become stricter over time, or the questionnaire version changes |
| Statistical regression | Internal | Clients chosen for extreme scores drift toward the mean on retest |
| Selection | Internal | The treatment clinic serves more motivated clients than the comparison clinic |
| Experimental mortality (attrition) | Internal | The most distressed clients drop out of the treatment group |
| Selection-maturation interaction | Internal | One group was already changing faster when the study began |
| Reactive effect of testing | External | A pretest sensitizes participants so results apply only to pretested people |
| Interaction of selection and treatment | External | The program works for volunteers but not for mandated clients |
| Reactive arrangements | External | Participants act differently because they know they are being studied (Hawthorne effect) |
| Multiple-treatment interference | External | Clients received three interventions in a row, so effects cannot be separated |
Some textbooks list the Hawthorne effect under internal validity. Campbell and Stanley placed reactive arrangements under external validity, so read the options carefully. Other threats often named in counseling texts are diffusion of treatment (control group members learn the intervention) and experimenter bias, controlled with blind or double-blind procedures.
How do sampling methods differ?
Probability samples give every member of the population a known chance of selection, which supports generalization (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). Simple random sampling gives everyone an equal chance; stratified sampling draws randomly within subgroups (for example, by year in school) so each is represented; cluster sampling randomly selects whole groups such as schools; systematic sampling takes every nth name from a list. Non-probability samples (convenience, purposive, snowball, quota) are easier to collect and common in qualitative work, but generalization is limited. Keep two ideas apart: random selection supports external validity; random assignment supports internal validity.
Which statistical test fits which research question?
Start with three facts: how many groups or variables there are, what level of measurement the outcome has, and whether the same people are measured twice (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs).
| Research question | Outcome data | Test | Nonparametric alternative |
|---|---|---|---|
| Do two independent groups differ? | Interval or ratio | Independent-samples t-test | Mann-Whitney U |
| Did the same group change from pre to post? | Interval or ratio | Paired (dependent) t-test | Wilcoxon signed-rank |
| Do three or more groups differ on one factor? | Interval or ratio | One-way ANOVA (then post hoc tests) | Kruskal-Wallis |
| Effects of two or more factors and their interaction? | Interval or ratio | Factorial ANOVA | |
| Group differences after controlling for a covariate (such as pretest)? | Interval or ratio | ANCOVA | |
| Group differences on two or more outcomes at once? | Interval or ratio | MANOVA | |
| Are two categorical variables related? | Frequencies (nominal) | Chi-square test of independence | |
| Do observed counts match expected counts? | Frequencies (nominal) | Chi-square goodness of fit | |
| How strongly are two continuous variables related? | Interval or ratio | Pearson r | Spearman rho (ranks) |
| Predict an outcome from one or more predictors? | Interval or ratio outcome | Simple or multiple regression | Logistic regression for a yes/no outcome |
Why not run several t-tests instead of an ANOVA? Each test carries its own chance of a false positive, so running many inflates the overall Type I error rate. ANOVA tests all the means at once, and post hoc tests (such as Tukey) then locate the difference.
What are Type I and Type II errors, power and effect size?
A Type I error (alpha) is rejecting a null hypothesis that is true: concluding a program works when it does not (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). A Type II error (beta) is failing to reject a null hypothesis that is false: missing a real effect. Power is 1 minus beta, the chance of detecting an effect that exists. Power rises with a larger sample, a larger true effect, a less strict alpha and less measurement error. Lowering alpha from .05 to .01 cuts Type I risk but raises Type II risk.
Statistical significance tells you a result is unlikely under the null hypothesis; effect size tells you how big it is. Cohen's d expresses a difference between means in standard deviation units, and Cohen's conventions call d of 0.2 small, 0.5 medium and 0.8 large (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). With a very large sample, a trivial difference can be significant, which is why journals ask for effect sizes. For the math of standard scores and reliability, see assessment and statistics.
What is program evaluation and how is it done?
Program evaluation judges whether a counseling program is needed, delivered as planned and effective (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). The questions change over the life of a program.
| Type | Question it answers | When | Example |
|---|---|---|---|
| Needs assessment | What problems exist and for whom? | Before the program | Survey of students about stress and service gaps |
| Formative | How can we improve the program while it runs? | During | Mid-semester feedback from group members |
| Process (implementation) | Is the program delivered as designed, to the intended people? | During | Attendance logs, fidelity checklists |
| Summative | Should the program continue, expand or end? | After a cycle | Year-end report to the funder |
| Outcome | Did participants change on the targeted goals? | After | Pre/post symptom scores, graduation rates |
Two frameworks show up often. A logic model links inputs, activities, outputs and short- and long-term outcomes, so the evaluator can see what to measure at each step. Stufflebeam's CIPP model evaluates context, input, process and product. Accountability data from evaluations also support ethical practice: the ACA Code asks counselors to monitor their effectiveness (C.2.d) (source: 2014 ACA Code of Ethics, checked 2026-11-14).
Worked example: pick the test for four scenarios
- Scenario. A counselor compares anxiety scores (interval) of 40 clients who got CBT with 40 who got waitlist. Reasoning. Two independent groups, one interval outcome. Test. Independent-samples t-test.
- Scenario. A school tests whether graduation status (graduated or not) is related to participation in mentoring (yes or no). Reasoning. Both variables are categorical; the data are counts in a 2 by 2 table. Test. Chi-square test of independence.
- Scenario. A clinic compares depression scores across three group formats: CBT group, support group and psychoeducation. Reasoning. One factor with three levels, interval outcome; several t-tests would inflate Type I error. Test. One-way ANOVA followed by post hoc comparisons.
- Scenario. A researcher asks whether hours of supervision and caseload size predict counselor burnout scores. Reasoning. Two continuous predictors, one continuous outcome, a prediction question. Test. Multiple regression.
A fifth twist: if the CBT clients in scenario 1 were measured before and after treatment with no comparison group, the same people are measured twice, so the answer becomes a paired t-test, and the design is a weak pre-experimental one-group pretest-posttest.
Frequently asked questions
What is the difference between random selection and random assignment?
Random selection is how participants are drawn from the population and supports generalization; random assignment is how they are placed into conditions and supports causal claims (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs).
When should I use a nonparametric test?
When the outcome is ordinal, or the data badly violate the assumptions of a parametric test (for example, a small, very skewed sample) (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs).
Is the ABAB design ethical if treatment is withdrawn?
It can raise concerns when the behavior is dangerous, which is why a multiple baseline design is often preferred in that case (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs).
How much research is on the NCE?
Research sits within the NCE content areas; see NCE content areas for how the outline is organized.
Check yourself: 6 practice questions
Original questions written for this page. Pick your answer, then open it to check.
- Designs
A researcher compares two intact sixth-grade classrooms, one receiving a bullying-prevention curriculum and one not, with no random assignment. The design is:
- True experimental
- Single-subject
- Correlational
- Quasi-experimental
Show answer
Answer: D. Comparison groups without random assignment make a quasi-experimental (nonequivalent control group) design. A true experiment would need random assignment of students to conditions.
- Validity threats
Clients selected because they scored extremely high on a depression screen score lower on retest even without treatment. This threat is:
- History
- Statistical regression
- Instrumentation
- Diffusion of treatment
Show answer
Answer: B. Extreme scores tend to move toward the mean on retest, which is statistical regression. History would require an outside event during the study.
- Choosing a test
Which test fits a study of whether gender (female, male, nonbinary) is related to preferred counseling format (individual, group, online)?
- Chi-square test of independence
- One-way ANOVA
- Pearson r
- Paired t-test
Show answer
Answer: A. Both variables are categorical and the data are counts, so chi-square is correct. ANOVA would need an interval outcome.
- Errors
A study concludes that a new intervention has no effect when it actually does. This is a:
- Type I error
- Alpha level
- Type II error
- Effect size
Show answer
Answer: C. Missing a real effect (failing to reject a false null) is a Type II error. A Type I error is a false positive.
- Single-subject
A counselor wants to test a self-monitoring intervention for a child's outbursts but cannot ethically remove the intervention once it works. The best design is:
- ABAB withdrawal
- Multiple baseline across settings
- One-shot case study
- Solomon four-group
Show answer
Answer: B. A multiple baseline design staggers the start of treatment and does not require withdrawal. ABAB requires removing the treatment.
- Program evaluation
Halfway through a new grief group, the leader collects feedback to adjust the sessions. This is:
- Summative evaluation
- Needs assessment
- Outcome evaluation
- Formative evaluation
Show answer
Answer: D. Evaluation used to improve a program while it runs is formative. Summative evaluation judges a program after a cycle to decide its future.
The Intakewise app has original NCE-style questions in all eight core areas and 30 NCMHCE-style clinical sessions you walk from intake to treatment.
Get it onGoogle Play Practice free first
iPhone app coming soon. Until then, the free practice works in any phone browser.
Sources
- Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs. Reference work cited by title.
- 2014 ACA Code of Ethics (PDF), American Counseling Association. Checked 2026-11-14.
Exam formats and licensing rules change. Confirm details with NBCC, CCE and your state board.
How this page was made: Opened the 2014 ACA Code of Ethics (C.2.d) on 2026-11-14; designs, validity threats, test selection and evaluation types are our own synthesis of the standard research references listed.