Statistical Conclusion Validity
Statistical conclusion validity
refers to the appropriate use of statistics for data analyses. Examples
of threats to statistical conclusion validity include:
1. Low statistical power
2. Violated assumptions of statistical
tests
3. Error rate
4. Reliability of outcome
measure procedures/tests
I have discussed reliability of outcome measures in a previous post. I
will be discussing statistical power, statistical test violations, and error
rate in future posts.
Internal Validity
Internal validity refers to the
potential for confounding factors to interfere with the relationship between
the independent and dependent variables.
I have divided threats to internal validity into three categories.
1. Single-Group Threats
2. Multiple-Group Threats
3. Social Threats
Single-group threats include:
a) history, b) maturation, c) attrition, d) testing, e) instrumentation, and f)
regression.
History is
where the observed study results may be
explained by events or experiences (confounding variables), other than the
intervention/treatment. For example, participation in other physical
activities may effect the outcome of an exercise training study.
Maturation is a threat that is internal to the individual
participant. It is possible that mental or physical changes occur within
the study participants that could account for the study results, simply due to
the passage of time. For example, a study that investigates the effect of
a treatment on pain in patients with an acute orthopaedic injury may observe an
improvement in pain due to the normal healing process and not the treatment.
Attrition (also referred to as drop-outs, withdraw, and experimental
mortality) is a threat related to participants withdrawing from a study before
it is completed. If participants withdraw from a study, randomization is
negatively impacted and data analyses cannot be performing on all pre-treatment
and post-treatment data.
Testing (especially multiple testing) can have a potential effect on
a dependent measure. The effect of conducting multiple tests can result
in improvement in an outcome measure that is due to a testing effect and not
the effect of an intervention. Because of this potential testing effect,
researchers should use tests that are considered reliable.
Instrumentation is a possible threat if an instrument is unreliable and provides
data that are unstable and prone to measurement error. Observed
changes seen between observation points (ie. pre-test and post-test) may also
be due to changes in the testing procedure, including the instrument that is
being used to collect data.
Regression (or regression toward the mean) is also related to the reliability of a
test. When an unreliable test is used for data collection, a statistical
phenomenon sometimes occurs when extreme pre-intervention scores (for example,
very high or very low scores) regress toward the group mean at
post-intervention. Again, a reliable test minimizes the threat of
regression. Previous research has identified this statistical phenomenon.
Multiple-group threats to internal validity are related to any variables other
than the experimental intervention that can have an impact of the
post-intervention difference in outcomes between the groups (sometimes referred
to as selection interaction), making the groups not
comparable. Below is a description of different selection interaction
threats.
The threat of selection-history is
when one group of study participants has different experiences than the
other group and these different experiences can influence the outcome of the
study.
Selection-maturation occurs
when the experimental group experiences changes in the dependent variable at a
different rate than the control or comparison group. For example, a group
of 2-year-old children are likely to experience a different rate of change in
development than a group of 10-year-old children.
Selection-testing is
when pre-intervention testing affects the groups differently.
Selection-instrumentation occurs
when the test (outcome measurement procedures) are performed differently
between the groups.
Selection-regression is a
concern when participants are assigned to groups based on extreme scores.
Social
threats to internal validity refer to the social pressures in the research
context that can lead to post-intervention differences that are not
directly caused by the treatment. The following are possible social
threats to internal validity.
Social threats can occur because study participants
in one group are aware of the treatment that the other group is receiving. Below are examples of socials threats.
Diffusion
or imitation of treatment occurs when a
comparison or control group learns about the treatment that the experimental
group receives and tries to imitate the treatment.
Compensatory
rivalry is where the comparison or control group
knows which intervention that the experimental group is receiving and develops
a competitive attitude toward the experimental group.
Compensatory
equalization of treatments occurs when the one(s)
who are delivering the experimental treatment administer the treatment to the
control or comparison group because the treatment is considered more favorable
for treating the study participants.
Resentful
demoralization can be thought of the opposite of
compensatory rivalry. For example, the comparison or control group
discovers the treatment that the experimental group is receiving. In this
case, instead of developing a rivalry, the comparison group becomes resentful
and thus, post-intervention outcome scores may be lower. Such an impact
may be observed when subjective outcome measures (like a questionnaire) are
being used. This threat can result in false, exaggerated differences
between groups, making the treatment appear more effective than it actually is.
Construct Validity
of Cause and Effect
Construct validity
are abstract behaviors or events than cannot be directly observed, but can
impact the interpretation of the cause-and-effect relationship. The
threat related to construct validity that I would like to discuss is experimental
bias. This threat occurs when biases are
introduced into a study by investigators or the study participants. For
example, participants may desire to fulfill the expectations of the
investigators. Thus, the participants "try harder" to perform
better on post-intervention outcomes or adhere more strictly to the study
protocol. Investigators can incorporate this type of bias by making study
participants aware of their expectations. This phenomenon is often
referred to as the Hawthorne effect. Click on the following link for the history of the Hawthorne effect.
External Validity
The
findings of a clinical research study must be translational to environments
outside of a “lab” in order to be applicable.
External validity is related to the degree to which the results of a
study can be generalized beyond the laboratory environment. I will discuss three threats to external
validity.
1. Interaction of Treatment and Selection
2. Interaction of Treatment and Setting
3. Interaction of Treatment and History
When
designing a clinical research study, investigators should include a sample of study
participants from a target population.
The threat of interaction of treatment and selection concerns when the
treatment does not apply to the entire target population. For example, consider a study that includes a
sample of patients with chronic low back pain.
The study findings may indicate that an intervention was effective for treating
the sample of study participants.
However, sub-classifications of patients with chronic low back pain
exist (patients have chronic low back pain with different causes). So, the treatment may not be applicable for
every patient with chronic low back pain.
The
threat of interaction of treatment and setting occurs when the findings of a
study cannot be applied to an environment outside the “lab”. For example, a treatment may be shown to be
effective in a controlled laboratory, but the clinical environment is significantly
different and therefore, the results of the study cannot be observed at another
site. This threat can be minimized by
replicating the study at multiple locations to determine if the study findings
are observed in various settings.
The
threat of interaction of treatment and history concerns the ability to
generalize the findings of a study to different points in time. For example, the findings of an older study
may have indicated that a drug was effective for reducing hypertension, however
the study did not control for confounding variables such as diet and exercise. Since the results of more recent studies provide
evidence that diet and exercise can improve hypertension, the findings of the
older study may not presently apply.
