Experimental Design in Educational Research | 10 Common Threats to Internal Validity and Useful Methods to Minimize Them

Do you wonder how to design an experiment in education? If you are looking for useful practical tips and examples and basic principles for experimental design, here is the topic: 10 common threats to internal validity of an experimental educational research and the methods to minimize the threats. It’s useful for educational psychology, educational measurement and instrumentation as well. The tips have advantages and disadvantages or limitations. To reduce the flaws, these tips are recommended. Randomization and control group are prominent tips.

If you want to watch this useful article as a video, here: https://youtu.be/omwLbSDFJZ0

If you prefer to read, slide the screen down.

Hi and hola para amigos,

Experimental research in educational field provides valuable information. It’s valuable since experiments allow us to see the relation between cause and effect. To conduct an experimental research, researchers need to use rigorous methods. There are some critical points to pay more attention. These are common threats to the validity of your study. I will present threats to internal validity of education experiments. If you stick around until the end, you will explore not only the threats but also how to minimize them.

The pretty flamingo is excited to contribute to your research.

Subject characteristics and selection bias are the first threats to the internal validity of education research experiments. In experiments, everything must be exactly same at the beginning. But education research does not allow to organize them exactly same. We are not living in a laboratory; the world we live in is totally complex. Participants are unique humans and differences among participants at the beginning of the study harmful for the validity. So, even if we could not get rid of all the differences, we still are able to minimize them. If so, how? To minimize it, you can randomize the participants. If you assign the participants to the groups randomly, all of the groups can be considered as the same. Also you can be sure that selection bias is as low as possible.

Randomization is a solution for another threat: Maturation. Maturation refers to the changes in participants over time unrelated to particular events. If you randomize the participants, the maturation effect applies all of the participants. The chance is equal for all of the participants. So, you minimize the maturation threat in this way.

History is another threat. History means that unplanned events unrelated to the intervention that might impact outcome. This kind of events impact the outcome and it may hinder or boost the outcome falsely. To save the study form this threat, a control group along with experiment group could be used. If there is a control group, it has equal chance to go through unplanned events that might impact outcome. So the conditions become the same for both group.

The same strategy, which is assigning a control group, could be used to minimize the other two threats: Instrumentation and regression to the mean. Instrumentation threat refers to the changes in scoring rubric or instrument calibration, including rater fatigue. The other threat, regression to the mean is an interesting concept. I have no space to explain it detailed but I recommend you to watch the Veritasium’s video on YouTube to understand this concept: The title of the video is How We’re Fooled By Statistics. I left the link of that video. So, to minimize this threat, control group is a useful solution.

Next threat! Testing. If you conduct a pretest, the participants would be familiar to the questions. Recall is a problem. So the performance on the posttest could be biased. Addition to that, pretest could lead to stimulate learning to the test, also increase awareness of intent of the study. These are really important problems for experimental design.

The solution is “no pretest”. If you don’t conduct a pretest, all of these problems disappear. But if we don’t carry out a pretest, how can we determine whether the participants improved themselves? By comparing the posttest performance of experiment group and control group. If we randomly assigned the participants to the groups at the beginning, we can assume that the two groups were equal at the initial conditions. So we can trust the comparison of the posttest performance of the two groups.

Fraenkel and Wallen state that

‘‘The randomized posttest-only design is perhaps the best of all designs to use in an experimental study, provided there are at least forty participants in each group.’’

Another threat is mortality or in other words, loss to follow-up. Sorry but I cannot provide a miraculously solution for that. You simply need to prevent loss. Next threat to internal validity of education research studies is location. It refers to the differences between groups in the environment or available resources. We should minimize as much as possible but differences would still exist since we cannot conduct the study in a laboratory setting. So, we should collect information on potential difference.

So, next threat: Participant attitude and motivation.

Learners involved in something they consider novel, or who are being observed, tend to be more motivated, conversely, those in comparison group may be demotivated. To minimize this threat, blind the participants to study hypothesis. If they are not aware of the intention of the study, their motivation would not be affected by it.

The last threat to the internal validity of education research studies is implementation. Variation in the learning experience e.g. differences in the expertise of the instructors, the opinions of instructors regarding the efficacy of the intervention, or the actual amount of instruction received. Learning outside the curriculum (how much learners studied on the topic beyond that intended by the intervention) falls into this category as well. It is hard to reduce the implementation threat zero, but to minimize it, careful planning of study interventions; collecting information on actual experiences, both within and without the study, are among useful methods.

As a summary, a comparison group can help mitigate many of these threats, but randomization is only required to control threats from selection and maturation. On the other hand, randomization cannot control for mortality, location, attitude, or implementation threats. Thus, the mere presence of randomization does not guarantee a study’s validity. In addition to that, pretest mostly weaken the study design.

If you learned something from this article, consider to like and subscribe. I left the source of the information, which is the article from Cook and Beckman, at the description below the video: https://youtu.be/omwLbSDFJZ0

See you and adios para amigos.

And also, don’t forget the flamingo.

For more information
Follow me on Twitter: https://twitter.com/MedEdFlamingo
Subscribe to my YouTube channel: https://www.youtube.com/channel/UCyOlOFLZTPFTBsH8PeLyitw