Intakewise Get the app

Research design, statistical tests and program evaluation for counselors

Last updated ยท Published by Certloom Learning

Home / Guides / NCE knowledge guides

Short answer. NCE research questions ask you to identify a design, spot the threat to validity and pick the right statistical test. Random assignment makes a true experiment; without it the design is quasi-experimental. To compare two group means use a t-test, three or more means a one-way ANOVA, counts in categories a chi-square, and a relationship between two continuous variables a correlation or regression. Program evaluation separates needs assessment, formative and summative evaluation, and process and outcome questions.

Source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs..

What research designs does the NCE expect you to recognize?

The first question to ask of any study is whether participants were randomly assigned to conditions (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). Random assignment spreads unknown differences evenly across groups, which is what lets a researcher claim the treatment caused the change. Campbell and Stanley sorted designs into three families.

FamilyExamplesWhat it can claim
Pre-experimentalOne-shot case study; one-group pretest-posttest; static-group comparisonVery little about cause; many threats uncontrolled
True experimentalPretest-posttest control group; posttest-only control group; Solomon four-groupCausal inference, because of random assignment
Quasi-experimentalNonequivalent control group (intact classrooms or clinics); time series; interrupted time seriesTentative causal claims; selection is the main threat

The Solomon four-group design adds two groups without a pretest so the researcher can test whether taking the pretest changed the results. Correlational designs measure variables as they occur and describe relationships, not causes. A correlation between hours of counseling and lower symptoms may reflect motivated clients attending more.

How do single-subject designs work?

A single-subject (single-case) design measures one person, or a small number, repeatedly across phases, so each participant serves as their own control (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). A is a baseline phase with no treatment, B is the treatment phase.

Results are usually judged by visual inspection of the graph (level, trend, variability) rather than by significance tests. The repeated measure is often a brief symptom scale or a behavior count; the assessment instruments page lists common measures.

What are the main qualitative traditions?

Qualitative research studies meaning and experience in depth with small, purposefully chosen samples. Phenomenology asks what an experience is like for those who live it, grounded theory builds a theory from coded interview data, ethnography studies a culture from inside, a case study examines a bounded case in detail and narrative research works with life stories (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). Trustworthiness takes the place of validity: credibility (member checks, triangulation), transferability (thick description), dependability (audit trail) and confirmability (reflexivity). The research practice page has a short table of these terms.

What threatens internal and external validity?

Internal validity asks whether the treatment caused the effect; external validity asks whether the result generalizes to other people, settings and times (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). Campbell and Stanley's lists are still the reference.

ThreatTypeExample in a counseling study
HistoryInternalA campus suicide during a prevention study changes everyone's mood
MaturationInternalChildren's social skills improve over a school year with or without the group
TestingInternalClients score better on the posttest because they remember the pretest
InstrumentationInternalRaters become stricter over time, or the questionnaire version changes
Statistical regressionInternalClients chosen for extreme scores drift toward the mean on retest
SelectionInternalThe treatment clinic serves more motivated clients than the comparison clinic
Experimental mortality (attrition)InternalThe most distressed clients drop out of the treatment group
Selection-maturation interactionInternalOne group was already changing faster when the study began
Reactive effect of testingExternalA pretest sensitizes participants so results apply only to pretested people
Interaction of selection and treatmentExternalThe program works for volunteers but not for mandated clients
Reactive arrangementsExternalParticipants act differently because they know they are being studied (Hawthorne effect)
Multiple-treatment interferenceExternalClients received three interventions in a row, so effects cannot be separated

Some textbooks list the Hawthorne effect under internal validity. Campbell and Stanley placed reactive arrangements under external validity, so read the options carefully. Other threats often named in counseling texts are diffusion of treatment (control group members learn the intervention) and experimenter bias, controlled with blind or double-blind procedures.

How do sampling methods differ?

Probability samples give every member of the population a known chance of selection, which supports generalization (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). Simple random sampling gives everyone an equal chance; stratified sampling draws randomly within subgroups (for example, by year in school) so each is represented; cluster sampling randomly selects whole groups such as schools; systematic sampling takes every nth name from a list. Non-probability samples (convenience, purposive, snowball, quota) are easier to collect and common in qualitative work, but generalization is limited. Keep two ideas apart: random selection supports external validity; random assignment supports internal validity.

Which statistical test fits which research question?

Start with three facts: how many groups or variables there are, what level of measurement the outcome has, and whether the same people are measured twice (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs).

Research questionOutcome dataTestNonparametric alternative
Do two independent groups differ?Interval or ratioIndependent-samples t-testMann-Whitney U
Did the same group change from pre to post?Interval or ratioPaired (dependent) t-testWilcoxon signed-rank
Do three or more groups differ on one factor?Interval or ratioOne-way ANOVA (then post hoc tests)Kruskal-Wallis
Effects of two or more factors and their interaction?Interval or ratioFactorial ANOVA
Group differences after controlling for a covariate (such as pretest)?Interval or ratioANCOVA
Group differences on two or more outcomes at once?Interval or ratioMANOVA
Are two categorical variables related?Frequencies (nominal)Chi-square test of independence
Do observed counts match expected counts?Frequencies (nominal)Chi-square goodness of fit
How strongly are two continuous variables related?Interval or ratioPearson rSpearman rho (ranks)
Predict an outcome from one or more predictors?Interval or ratio outcomeSimple or multiple regressionLogistic regression for a yes/no outcome

Why not run several t-tests instead of an ANOVA? Each test carries its own chance of a false positive, so running many inflates the overall Type I error rate. ANOVA tests all the means at once, and post hoc tests (such as Tukey) then locate the difference.

What are Type I and Type II errors, power and effect size?

A Type I error (alpha) is rejecting a null hypothesis that is true: concluding a program works when it does not (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). A Type II error (beta) is failing to reject a null hypothesis that is false: missing a real effect. Power is 1 minus beta, the chance of detecting an effect that exists. Power rises with a larger sample, a larger true effect, a less strict alpha and less measurement error. Lowering alpha from .05 to .01 cuts Type I risk but raises Type II risk.

Statistical significance tells you a result is unlikely under the null hypothesis; effect size tells you how big it is. Cohen's d expresses a difference between means in standard deviation units, and Cohen's conventions call d of 0.2 small, 0.5 medium and 0.8 large (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). With a very large sample, a trivial difference can be significant, which is why journals ask for effect sizes. For the math of standard scores and reliability, see assessment and statistics.

What is program evaluation and how is it done?

Program evaluation judges whether a counseling program is needed, delivered as planned and effective (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs). The questions change over the life of a program.

TypeQuestion it answersWhenExample
Needs assessmentWhat problems exist and for whom?Before the programSurvey of students about stress and service gaps
FormativeHow can we improve the program while it runs?DuringMid-semester feedback from group members
Process (implementation)Is the program delivered as designed, to the intended people?DuringAttendance logs, fidelity checklists
SummativeShould the program continue, expand or end?After a cycleYear-end report to the funder
OutcomeDid participants change on the targeted goals?AfterPre/post symptom scores, graduation rates

Two frameworks show up often. A logic model links inputs, activities, outputs and short- and long-term outcomes, so the evaluator can see what to measure at each step. Stufflebeam's CIPP model evaluates context, input, process and product. Accountability data from evaluations also support ethical practice: the ACA Code asks counselors to monitor their effectiveness (C.2.d) (source: 2014 ACA Code of Ethics, checked 2026-11-14).

Worked example: pick the test for four scenarios

  1. Scenario. A counselor compares anxiety scores (interval) of 40 clients who got CBT with 40 who got waitlist. Reasoning. Two independent groups, one interval outcome. Test. Independent-samples t-test.
  2. Scenario. A school tests whether graduation status (graduated or not) is related to participation in mentoring (yes or no). Reasoning. Both variables are categorical; the data are counts in a 2 by 2 table. Test. Chi-square test of independence.
  3. Scenario. A clinic compares depression scores across three group formats: CBT group, support group and psychoeducation. Reasoning. One factor with three levels, interval outcome; several t-tests would inflate Type I error. Test. One-way ANOVA followed by post hoc comparisons.
  4. Scenario. A researcher asks whether hours of supervision and caseload size predict counselor burnout scores. Reasoning. Two continuous predictors, one continuous outcome, a prediction question. Test. Multiple regression.

A fifth twist: if the CBT clients in scenario 1 were measured before and after treatment with no comparison group, the same people are measured twice, so the answer becomes a paired t-test, and the design is a weak pre-experimental one-group pretest-posttest.

Frequently asked questions

What is the difference between random selection and random assignment?

Random selection is how participants are drawn from the population and supports generalization; random assignment is how they are placed into conditions and supports causal claims (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs).

When should I use a nonparametric test?

When the outcome is ordinal, or the data badly violate the assumptions of a parametric test (for example, a small, very skewed sample) (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs).

Is the ABAB design ethical if treatment is withdrawn?

It can raise concerns when the behavior is dangerous, which is why a multiple baseline design is often preferred in that case (source: Campbell, D. T., & Stanley, J. C. (1963). Experimental and Quasi-Experimental Designs for Research; Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.); Stufflebeam, D. L. CIPP evaluation model; Kazdin, A. E. Single-Case Research Designs).

How much research is on the NCE?

Research sits within the NCE content areas; see NCE content areas for how the outline is organized.

Check yourself: 6 practice questions

Original questions written for this page. Pick your answer, then open it to check.

  1. Designs

    A researcher compares two intact sixth-grade classrooms, one receiving a bullying-prevention curriculum and one not, with no random assignment. The design is:

    1. True experimental
    2. Single-subject
    3. Correlational
    4. Quasi-experimental
    Show answer

    Answer: D. Comparison groups without random assignment make a quasi-experimental (nonequivalent control group) design. A true experiment would need random assignment of students to conditions.

  2. Validity threats

    Clients selected because they scored extremely high on a depression screen score lower on retest even without treatment. This threat is:

    1. History
    2. Statistical regression
    3. Instrumentation
    4. Diffusion of treatment
    Show answer

    Answer: B. Extreme scores tend to move toward the mean on retest, which is statistical regression. History would require an outside event during the study.

  3. Choosing a test

    Which test fits a study of whether gender (female, male, nonbinary) is related to preferred counseling format (individual, group, online)?

    1. Chi-square test of independence
    2. One-way ANOVA
    3. Pearson r
    4. Paired t-test
    Show answer

    Answer: A. Both variables are categorical and the data are counts, so chi-square is correct. ANOVA would need an interval outcome.

  4. Errors

    A study concludes that a new intervention has no effect when it actually does. This is a:

    1. Type I error
    2. Alpha level
    3. Type II error
    4. Effect size
    Show answer

    Answer: C. Missing a real effect (failing to reject a false null) is a Type II error. A Type I error is a false positive.

  5. Single-subject

    A counselor wants to test a self-monitoring intervention for a child's outbursts but cannot ethically remove the intervention once it works. The best design is:

    1. ABAB withdrawal
    2. Multiple baseline across settings
    3. One-shot case study
    4. Solomon four-group
    Show answer

    Answer: B. A multiple baseline design staggers the start of treatment and does not require withdrawal. ABAB requires removing the treatment.

  6. Program evaluation

    Halfway through a new grief group, the leader collects feedback to adjust the sessions. This is:

    1. Summative evaluation
    2. Needs assessment
    3. Outcome evaluation
    4. Formative evaluation
    Show answer

    Answer: D. Evaluation used to improve a program while it runs is formative. Summative evaluation judges a program after a cycle to decide its future.

The Intakewise app has original NCE-style questions in all eight core areas and 30 NCMHCE-style clinical sessions you walk from intake to treatment.

Get it onGoogle Play Practice free first

iPhone app coming soon. Until then, the free practice works in any phone browser.

Sources

Exam formats and licensing rules change. Confirm details with NBCC, CCE and your state board.

How this page was made: Opened the 2014 ACA Code of Ethics (C.2.d) on 2026-11-14; designs, validity threats, test selection and evaluation types are our own synthesis of the standard research references listed.