Login

Welcome, Guest. Please login or register.

March 29, 2024, 01:59:24 am

Author Topic: Dependency between validity, reliability and accuracy  (Read 6806 times)  Share 

0 Members and 1 Guest are viewing this topic.

RuiAce

  • ATAR Notes Lecturer
  • Honorary Moderator
  • Great Wonder of ATAR Notes
  • *******
  • Posts: 8814
  • "All models are wrong, but some are useful."
  • Respect: +2575
Dependency between validity, reliability and accuracy
« on: February 21, 2017, 10:58:11 pm »
+8
Hello there,

By now, many of you will know that whilst there are many things to address in scientific discussions, the main 3 are those listed above. You must ALWAYS consider to what extent your investigation was reliable, valid and accurate.

However, there were two things that I bumped into often during my HSC studies. The first is the fact that students tended to juggle between all the terms at once and hence mixing up which is what. However this lead to the second observation - the way students interpreted the terms lead them to believe that they must somehow be correlated, if not depend on each other. In this post, I would like to explore the possibilities of achieving only one or two of the three main criteria.

Let's review some of the more obvious ones:

The Extreme Cases

How can an experiment be valid, reliable AND accurate?
To achieve all three, you essentially have the perfect experiment and it is basically PERFECT to test the aim (validity). It has been repeated multiple times (reliability) to ensure that virtually the same results are produced, and your equipment is well chosen to reflect better data as well as being close to the theoretical values (accuracy).

How can an experiment be invalid, unreliable AND inaccurate?
One way is to simply design an experiment that has nothing to do with the aim (invalid), choose not to repeat it (unreliable) and use simplistic or broken instruments that may yield seemingly contradictory results (inaccurate). Of course, there are tons of ways of creating the worst possible experiment ever.

It becomes apparent that trying to achieve all 3 objectives is considerably more difficult than achieving none of them. These should make sense on the fly. But now, let's try considering how we may succeed at only ONE of these:

Attaining One Objective

How can an experiment be unreliable, inaccurate but still valid?
Design a method that actually addresses the aim (valid). Plan out everything so that if properly done, you should expect your experiment to produce useful results. But then choose to do it only once and accept it (unreliable) and, for a variety of reasons (e.g. human reaction delay, lack of good instruments) produce results that you would not have foreseen (inaccurate).

How can an experiment be invalid, unreliable but still accurate?
With great difficulty, to be honest! You would choose not to really repeat the experiment (unreliable), and in fact design an experiment that really lacks effectiveness in testing what you wish to prove (invalid), but design it barely sufficiently well so that when you conduct it, you conduct it so carefully that your results seem to make sense (accurate).

How can an experiment be invalid, inaccurate but still reliable?
Design an experiment that doesn't do what you want it to do (invalid) and obtain some nonsensical results (inaccurate). However, perform your invalid experiment the same way over and over again so that every time you do it, you get pretty much the same nonsensical results.

So there's actually a lot of possibility here! Now let's try two at a time:

Attaining Two Objectives

How can an experiment be valid, reliable but inaccurate?
Design an experiment that properly addresses the aim, being careful of almost every factor (valid) and repeating it over and over again and obtaining the same results (reliable). However, for some reason, an error should happen when you actually perform the experiment, or when you do the maths, so that your results are no longer what you'd expect (inaccurate). Usually the error occurs because the experiment written was perfect but something had occurred when actually performing it.

How can an experiment be valid, accurate but unreliable?
Design an experiment that properly addresses the aim, being careful of almost every factor (valid) and performing it so that you get excellent results on the fly (accurate). But then choose not to do it any more than two times (unreliable) and thus subjecting yourself to possible experimental fallacy.

How can an experiment be accurate, reliable but invalid?
Again, this will require some difficulty. To do this, you must have somehow designed a poor experiment that you wouldn't anticipate to help give you what you want (invalid), but then obtain the same excellent results despite these issues (accurate, reliable).

The Deal

If you observe the above scenarios carefully, you will notice some trends (or possibly, an absence of trends) occurring. These trends will lead to the analysis of how much the factors are interrelated.

Reliability and Validity
In general, only a tiny bit related. It's quite possible to design an experiment that's seemingly wrong, but your results collected are the same because you performed the experiment the same way. These two factors usually have nothing to do with each other.

Reliability and Accuracy
Essentially unrelated. There are many ways you can achieve the same wrong results over and over again.

Validity and Accuracy
The relationship between these two is interesting. Validity really does not depend on accuracy at all, but there's evidence to show that accuracy may somewhat depend on validity (i.e. the converse may be true). If your method is what appears wrong, then usually you'd anticipate that your results (which come after the method) to be wrong as well. Contradicting this is no easy task.

But then on the other hand, your results may be wrong despite using a correct method. Who knows what could possibly have happened after you wrote the method down and actually carried everything out? If anything, if your results are correct it's usually because your method was correct as well.


So this may be worth considering when you write a discussion to your experiment. This may give you an idea of what to expect when you write a discussion. The only thing worth considering is the fact that validity may affect accuracy, but nothing beyond this really occurs.

Before ending the post, I will mention that similar analyses can be made for second-hand investigations. However I won't make a post about them; if you have questions regarding those (or anything here really) just comment below!

jamonwindeyer

  • Honorary Moderator
  • Great Wonder of ATAR Notes
  • *******
  • Posts: 10150
  • The lurker from the north.
  • Respect: +3108
Re: Dependency between validity, reliability and accuracy
« Reply #1 on: February 22, 2017, 12:01:58 am »
+1
Great resource, nice work Rui ;D