Video The Scientific Method, hypothesis testing, pitfalls

0:00 <Untitled Chapter 1>

0:04 welcome back to the last unit of the lecture part of comp 335 we'll talk about empirical science and statistics today because this is how you properly evaluate ideas and hypotheses and theories in in education so we'll touch a bit on that

0:29 Vocabulary

0:29 my goals are to give you some vocabulary again similar to the theory chapter on on motivation and learning theories there are just a few terms that will pop up if you read papers about this that you should well at least know where to look it up um and yeah we'll see how much statistics you've seen before in a second i think there's some principles that you can say about this qualitatively without getting into much of of probability theory that are useful to know maybe this will be a repetition where some of you will see and i'll point out some pitfalls that relates or ties into the pre-session activity

1:13 i would say because computer science is not really a natural science like physics chemistry biology uh we don't have a proper module about empirical science the scientific method all these methodological things in our curriculum it kind of just fell off i think or maybe it was never really part of computer science

1:39 also one reason is that computer science is a weird discipline in that it is it is half half one part is a structural science like mathematics where you wonder what is what is possible in principle and how can you make this happen in in a theoretical space um and there it's it's a lot about proving theorems and uh everything is on on a mathematical model to start with so you don't need empirical science in that branch too much and the other part is at the other end of the spectrum if you want in engineering where you're one step behind the natural sciences you apply certain tools and techniques and then people teach best practices i mean we have software engineering that literally has this in the in the title but um other subfields are similar machine learning is maybe a reason why it might make sense to put a bit more emphasis on empirical research in computer science because it's growing in importance and that has more more similarities to other natural sciences maybe but yeah as far as as i know the curricula of computer science courses they don't really talk much about this

2:58 and i do think there's there's some something lost in not covering it i asked you how much you've seen about this uh in the past and i don't know does the the one who said yes i want to disclose what module that was um in what context yeah

3:25 i did on psychology which had like massive central school research all this sort of stuff yep that's great so just for repeating for the recording uh you did a levels in psychology and that hasn't had a more substantial part on research methodology

3:47 that's true especially the the software parts the humanities or social sciences they much more rely on this and also because they often deal with much more expensive things generating data is much more expensive in these areas they have to deal with smaller sample sizes because they can't bring a thousand people in their lab to test the little hypothesis and so they have to be even more rigorous on the methodology side and are hit harder by the the replication crisis in a sense especially psychology was the the field that's always mentioned first in these things and yeah i'm not trying to point this out as a field where things are done wrong more often than in other fields this happened to all fields basically but it makes sense in psychology to talk explicitly about this i think it makes sense talking computer science about it if you evaluate algorithm performance you might as well do this empirically as opposed to just proving big old bounds which in many cases are not so useful for predicting what's good in practice anyways but okay this is this is more my plea and it doesn't help to tell you this maybe you want to take this forward into school curricular if you ever end up designing a curriculum for universities

4:18 Replication Crisis

5:15 but let's today just do a a quick overview of of this space and i hope it's not not too superficially boring for you given your prior experience i want to just point out in the beginning we'll talk about all of these terms on the right in their scientific meaning today a bit i just want to bring this slide early to say be careful all of these terms are used in in everyday language but they mean a different thing uh we say theory if it's something that's not really clear it's just a hypothesis or some some suspicion that we might uh try to try to show and it has a much more concrete meaning in science uh same thing for experiment it could mean anything that you where you try something is an experiment that's not not true in the scientific sense uh statistics often in day-to-day speaking means you just summarize data in some way you compute the averages and standard deviations for the marks in your class and that's the statistics of the class or the statistics of a population you compare income or whatever you know all details from examples of this in in empirical science statistics really is about the testing part so you want to make concrete predictions about hypotheses whether they are

6:44 supported by the evidence or not so it has a lot more to do with probability theory with prior beliefs with inference um it's in a sense a narrower field that has more algorithmics in it all these statistical tests are basically telling you what to do to come to a certain conclusion and everyday language is more a vague collection of things you can compute and yeah significant we say significant if something's big or important in a general sense but statistically significant has a very specific technical meaning and especially in popular science reporting one has to be extra careful to keep those terms separate so that's also true for your essays i think your essays can be written in something like day to day lingo as i called it here but when you talk about experiments or about papers that talk about these these empirical scientific methods make sure you're separating the terms clearly i think it's often clear from context which one is meant but if you take a quotation out of the context then things get a bit hairy okay so let's um start from the beginning in a sense we'll talk about the scientific method scientific method is the philosophy behind science how we generate knowledge at least since the enlightening and uh before that there was often more

8:08 Scientific Method

8:20 guesswork and people felt this is supported by god or whatever um at some point people started to make this a bit more structured and the model that we still basically follow is is this uh you have some observation that you make about the real world and you well it uh gives you food for that for thought there's something um i will have a running example later but basically um maybe this is an observation you could have made in teaching if you give students a choice what to do in an assessment they do better or they like it better and then they do better maybe this is an observation you've made then you can formulate a theory why this is the case so we talked about motivation theories maybe this is something that one can use here but then you would like to the theory itself um is just in your in your head and it's not clear if uh if it's correct or not so to test the theory you derive a hypothesis that's a more concrete prediction if the theory was true certain things should follow from it and these things are measurable observable the important sacrifice here is that hypothesis in this sense they have to be falsifiable it must be possible to

9:39 refute them we'll come back to that in a second then we design a scientific experiment ideally a controlled experiment where you can infer the causation to test that hypothesis and well if you're if you're kind of lucky in a sense or if the experiment has good enough quality in the data you can support your hypothesis or you may fail to support it and then that might lead you to new observations and also the thing goes around round an important point is that the scientific method itself can never prove that the theory is correct beyond doubt it can only collect evidence and collect more evidence and some theories have been standing since decades despite researchers collecting more evidence and it's all in support of the theory that's great and then we can have confidence in the theory but you cannot prove it correct that's just a fundamental limitation in the sense of science in itself you can you can refute theories quite convincingly that's a bit of the the asymmetry here but you cannot prove a theory correct you can only support it with more evidence

9:43 Design a Scientific Experiment

11:01 Falsifiable Hypotheses

11:01 what's the deal with falsifiable hypotheses i just listed an example that's not falsifiable if you claim there is an almighty god somewhere i may try to find evidence for such a god maybe i don't find evidence so i can tell i can tell you i never found evidence for your god so yeah he or she doesn't exist but you can always say this just has been the will of god so uh he or she prefers to keep to to be out of sight because he's almighty he could do it um i can never i can never counter this argument because this almighty is kind of a cheat out of any any rational argument why i didn't find evidence you can always take this further and you replace god with magician or many other things work so there's no possible evidence that i could collect that would convince you that your hypothesis is wrong and that's a problem and it's not the problem of you it's the the hypothesis itself is just not amenable to the scientific method another thing that uh in practice will plague us usually is that the world is not black and white um evidence is usually not perfect and uh it it can usually only give us a certain tendency is this more in support of the fear of

12:11 The World Is Not Black and White

12:24 the hypothesis or more against it but there will always be a bit of fuzziness involved or a bit of personal belief that makes it sometimes hard to make definitive statements and science is all about being precise and accurate but then if you want to communicate it it's nice to have simple statements so there's a tension here

12:53 a way to make this quantitative reasoning rigorous is statistics it doesn't do away with the belief part as you'll see later but at least it gives us a solid foundation for making the quantitative part as reasonable as possible say

13:17 i started from the theory here and left out the observation but i gave that verbally before so say uh you believe in the theory that we talked about in the x in the motivation chapter the expectancy value theory you assign uh you're motivated to do something if you have reason to believe that you succeed and it appears valuable to you to do it

13:27 Expectancy Value Theory

13:43 and one one aspect of that is if you're if you have choice if you're in control of the process whatever you're doing learning in this case that improves your self-efficacy and that improves your your expectancy of support of succeeding it's a bit of a stretch but uh it's often counted as part of this theory uh and hopefully so this is another little stretch if you like i'm equating you're more motivated to do it so you will also be better that's another another claim that one would want to

14:23 to test in an experiment but let's ignore that for here because outcomes are easier to measure so it's clearer to talk about this data and from that we can derive a complete a concrete hypothesis which is if i give you a choice for your assessment you'll have better marks and usually if you're if you state such a claim in a scientific context you want to be very specific or very careful about the context usually the experiment you make will only make a statement for exactly the context in which you tested it so it might depend on what you're teaching what field what type of subject is it is it university is it school is it um a compulsory course that everyone has to take is it something that students do after school where it's just those that are interested all these things will have an influence so you have to take care to note this on the slides i will leave it out i will just use this hypothesis and in your head you will always duly add that it's specific to a certain environment that we're testing but let's just pretend we leave this out

15:43 um the gold standard for doing this is the controlled experiment and here again controlled it has a specific meaning it works as follows you take your subjects your class of students say and you randomly assign them into two groups and this random assignment is very important for the reason that statistics will pick up later but in a sense intuitively you don't want to end up with all the the gooder the better students who are more motivated anyways in one group and all the other students in another group then you already have a difference between the group you want the groups to be as equal as possible at the start and so the best we can do without having perfect knowledge about all your students is to randomly assign them and hope hope for the best so we have an experimental group and they do what you think should give them an advantage and you have the control group the experimental group in our case gets the choice which of two topics say they write about and the other group is randomly assigned some topic you probably want to control for how often each topic is picked so maybe you want to do give that group the choice first and then assign the other group the same

16:31 Experimental Group

16:36 Control Group

16:57 number of topic a same number of topic b randomly to people but the important difference is that the second group the control group doesn't have a choice they just get told what to do as is maybe it is still the standard then you do with both the same assessment you mark it in the same way this is the part where it's important to be to be meticulous to the methodology don't make any difference between the two groups ideally anonymize it so that it's blind for you you don't know which group you're you're marking at the moment

17:32 Double Blind Study

17:33 um you will have heard the term double blind study that's for for medic for medications uh if the patient knows they're getting the placebo or they're getting the actual treatment that influences how how they think about it how they feel and that again could be an influence on the study so you try to make it double blind neither the patient nor the doctor knows if they're giving the actual new medication or the placebo just to make sure that these psychological effects are excluded obviously in a case like this where we assign topics or give students a choice there's no way we can do this double blind we can do the marking blind but that's the best we can do and then well you look at your marks and if you feel that the marks in one group are better than the marks in the other group then you have found some interesting difference

18:26 concretely if our hypothesis was that choice makes the marks better than if that is indeed the case we support the hypothesis otherwise we refute it the catch here is that better is not so easy what exactly does that mean you will get a range of marks from both groups some well a list of marks and you can take the average or you could take the median or you take the smallest and largest or there's not a priory the right or wrong way to do it but a natural thing is to look at the average but even then if you find the averages these are real numbers say you give marks between 0 and 100 if the mark difference the average mark difference is 1 is that a finding or does that just mean this is the random variation that you get with a class of people who are not all identical in a perfect world you would just take your class clone it and have two identical copies for the experimental in the control group but alas this is not usually what you can do so you need statistics for judging if better is better enough or sufficiently better for your hypothesis so again same hypothesis as before now i'm one step back if uh

19:10 Average Mark Difference

19:52 i don't know if you've ever thought about this i said before the scientific method can never prove a theory correct uh you can only refute hypotheses convincingly um how can we apply this to our hypothesis if all we can do is refute something

20:19 well the solution is a very nice trick you apply the whole thing to the negation of what you want to prove and that's what people call the null hypothesis

20:23 The Null Hypothesis

20:27 so if you want to prove that there is a substantial difference okay i'll comment on this in a second uh if you want to prove there's a substantial difference between your two groups the one that got to choose the topic and the one that was assigned the topic the negation is there's no such difference and more concretely we could say the average mark in the two groups is equal that is the null hypothesis you kind of don't believe that right otherwise why would you do this experiment uh at least um you wouldn't gain much if it was true but we apply the scientific method to refute that null hypothesis and if we can successfully refute that clear evidence that contradicts the null hypothesis means evidence that supports the negation the alternative hypothesis and in our case the alternative hypothesis is well the means are not the same

21:29 so a slight nitpick originally from our theory of motivation we would say if there's a difference between the two groups then the one with the choice should do better but a priority is not clear maybe they do worse right who knows scientific testing statistical testing becomes much easier if your null hypothesis says equal as opposed to less or equal that's why i made this change here so our hypothesis now is there is a difference if you give people choice they do differently than if you if you don't give a choice and you can then um from the data often deduce if the the difference is positive or negative

22:16 okay so far so good now i've just shifted the problem how can we refute this null hypothesis again idealistic if we want to make a statement that's true for arbitrary humans maybe we should test them all and then we would know for sure in a sense statistics is only necessary because we can't do that if in some situation you're in the lucky position to be able to test anything and everything that you're making a statement about i mean by test including your experiment then you don't need any statistics but yeah again alas it's not what we can do and we would have to do the experiment twice with roles switched and all that so to keep things um yeah census is a nice example where we do actually have all data about the entire population up to people not responding etc but you're obliged to right so unless this is an option you can only take a sample of the population maybe in the class in the example i gave it's the class of students you already have and then you never quite know if your sample is large enough or if it's representative and so um we have to deal with the uncertainty that comes from this and it's also partly baked into the

23:36 controlled experiment i said randomly assign the two groups well there's a small chance that you were very very unlucky with your random assignment and it did create a very uneven distribution of academic ability in the two groups but we don't expect this to happen if you have really large groups it's absurdly unlikely but how much exactly is absurdly unlikely

24:03 Statistical Tests

24:05 and that's why we need statistical tests all right so what are statistical tests uh unfortunately this is where things become complicated because there's many of those tests but i tried to break it down to a simple abstract formula that's kind of true for all those and then there's the details maybe not so important for us because you're not trying to run your experiment next term you're just trying to understand existing experiments so that's a bit of a an easier task

24:41 the two hypotheses and how they are formulated like in our case h0 was the means are equal you could have many many statements of different types maybe you want to test the mean of one is smaller than the mean of the other or you can say the null hypothesis is both of these populations have a distribution over the marks and the distributions are the same but i don't know what the distribution is if you're talking about sample means you can use central limit theorems and assume if the sample is big enough that it's approximately normal distributed that's what most tests basically rely on we use not the entire data set in in every little bit but instead we take sample means and how they behave that's what most statistical tests are about but there's others so what what are the hypotheses how do they look like what forms do they have and then you get the observations from the two groups so i call this xc for control and xe for experimental group and the statistical test computes this this ominous p value which is the likelihood that you see this data if you repeat your experiment many many times including is the randomness of choosing people and assigning them to groups if you repeated your experiment many

26:01 many times what's the probability of seeing the data that you got if you assume that the null hypothesis is true okay so you make the assumption it's kind of a proof by induction experiment by induction or well reductio at absordum for the latin speaking people you try to assume the thing you don't believe is true and you try to derive a contradiction from that in our case contradiction means what we see is very very unlikely that's the contradiction that's the best we can get is only in terms of probabilities and that's what the p-value expresses i just want to stress again it's a weird quantity it's the likely it's the probability or yeah the proper term in in statistics is likelihood because it's not really probability but it's kind of the same it's how likely is the data that you've got assuming something that's maybe probably not true it's kind of a weird quantity and that's also why it's hard to interpret so what people usually do is they just set a cutoff if the p-value is smaller than a certain magic constant and that can be five percent in psychology one percent in some other fields in elementary physics statistical physics it might be much much smaller because they can produce much more data

27:27 but this is something that people have to decide on and there's no right or wrong it's just how much evidence do you want to see before you are inclined to believe in a hypothesis that this data is supporting

27:42 so the p-values are hard to interpret and some people do something simple just to cut off if it's below it's statistically significant and if it's not it's not and especially interpreting things that are not significant are tough i think i'll come back to that as well here's a few tests that you might have seen if you've done some statistics before there's many many more um and i don't feel this is useful for us because it depends on on exactly what you're observing and what you know about the data you get for example the the difference between the the z-test and the t-test is do you know the true variance in your population your observe marks and they will be if you pick a random person from your class that is a random mark that you get what's the distribution of that it has some mean some mean value some expected value and it has a variance and the difference between those two tests is do you know the variance for sure or do you not know the variants okay in our case we probably don't know it

28:55 yeah it's the variance of the test statistic i'm digressing i don't want i didn't want to talk about the tests so i won't talk about the specific tests much the basic principle for most tests looks like this though we compute some number and that number is taken we use our observed data and we subtract what we would expect if the null hypothesis was true that's some difference between what we actually have and what we would expect in this universe that we want to refute and we we divide this by the average variation so intuitively if you if you expect if you see a very big difference between what you actually got and what you expect if the null hypothesis is true that makes your test statistic big and a large test statistic means a small p-value so this is strong evidence if the variation though is really really large in your data then even a big difference might be meaningless because it's just random if the variation is huge a huge difference can also just come from random variation whereas if the variation is small it's again a bigger test statistic and the variation includes the sample size so the more people the more subjects you have in your study the lower you can push the variation

30:26 in our concrete example we probably want to use a t-test maybe i more want to focus on on saying what the data is in this abstract formulation the observed data in our case is not simply all the marks of the two groups it's more specifically the difference between the average marks because that's the quantity we're interested in we our hypothesis says the the average mark in the two groups is different and the null hypothesis said no it's the same so taking this difference between the averages that's really giving us the quantity we're looking at and the expected data in this case when when we assume h0 is just zero because the difference is not there okay and this is really often the case that this is zero and well average variation if you assume the variance in the two groups is independent the variance adds up and then what we need in the formula here is the standard deviation so it's the square root of that um well or you look it up in case you need this and that gives you the test statistic which is some number so that number by itself is really hard to interpret in this simple example you can work out the p-value kind of analytically there's a closed form in a sense

31:19 Average Variation

31:52 we're computing the p-value from the test statistic in other cases you can you can do this numerically it's not a such a big deal but there's a mapping between test statistic and p-value that they still have to do and um the test usually tells you how to do this and that's where the tests differ and the part that i don't want to talk about

32:18 okay so that's on one slide how statistical tests work or significance tests work and that's what most of the studies use in one variety or another the detail the devil's in the detail of using the right tests and coming to the right conclusions from it

32:39 and that brings me to what can possibly go wrong here's an obvious one that you will be familiar with but i felt i cannot not mention it if you cannot do controlled experiments your options are kind of limited um and sometimes controlled experiments are just uh not an option because it's it's absurdly unethical you may have heard of of this urban legend i think or yeah it's called the forbidden experiment the the theory is what what happens what's the the natural language for children if you don't talk to them if you raise a child without ever talking to them you've never heard about this there's a wikipedia article about it allegedly this experiment was done four times in history but uh historians questioned that it's ever been done and the results in all the four allegedly done experiments was so similar that people are suspicious they might have just copied and claimed it but the story goes pick your favorite historical king takes a few children puts them in the room he gives them everything they need physiologically like food and drink etc and there's a midwife etc they're taken care of but no one talks to them and the story goes they all die now we don't know if this is really true but it will definitely damage the

33:07 The Forbidden Experiment

34:06 children and no one wants to repeat the forbidden experiment so maybe we'll never know for sure this is a tough one to approach in any other way because well we don't have much evidence of people rising without any any outer influence of language and sometimes it's just absurdly expensive and we still don't want to do it i mean there's many examples of things we would like to know but a controlled experiment is just out of the question you can still do something you can do an observational study that's what what's done a lot just because it's cheap the data is there anyways um i don't know what's a nice example okay i listed listed our our whitening participation data set you can look at um you can look at post post codes there's data that shows you how many people go to uni there's data how many people from the post code have certain income and then you can try to connect these two is there a correlation between income of the postcode area and well the number of people going to uni i wouldn't be surprised if there was a correlation now what can you do with this and that's the problem with these observational studies you only ever get correlations so one variable and the

35:33 other have a a relationship if you increase one the other also goes up but this is not a causation it's just the pattern in the data whereas causation is if you change one thing the other will follow uh that's that's a subtle difference but yeah i mean you've probably seen this before so we can probably expect a correlation between income and higher education participation just for for many reasons unfortunate as it is but we cannot infer from this data anything that you know having more money brings people into university physically as a direct consequence or that university degrees generate higher income both is not unplausible but we cannot conclude this from this data and maybe you've seen this website

36:28 i don't have this as a link here well it doesn't matter uh we're running a bit out of time so it's a website that just goes through publicly available data sets and finds highly significant correlations between two things that most likely are just random for example the number of people who drown is highly significant highly significantly correlated to the number of movies with i think nicholas cage in that year so you look at the two curves and they nicely follow each other over law over many years okay it's probably just random random luck the thing is if you take enough pairs of things you will find some significant correlations that's almost almost guaranteed but that doesn't mean much

37:25 another thing that that could go wrong there's lots of lots of issues with um how studies are done and how empirical science is done i wouldn't be as cynical as this headline of the video suggests i think it's not quite as hopeless but definitely one has to be careful and there are some incentives that are seriously misaligned judging the the worth of scientists by how many papers they put out or how much waves and fuss the papers they put out create gives them an incentive to only publish these kinds of results and that bias alone is harmful enough

38:13 a big discussion has been as a result from this whether one would want to ditch entirely this notion of statistical significance especially if you pick a threshold like 0.05 you accept that with 5 your result is a false positive the null hypothesis could not actually be refuted well the null hypothesis is actually true but you your data refutes it because you were lucky enough to get one of the five percent outliers five percent is one in 20 is not so great maybe this is not not so useful um and yeah this this binary

38:53 classification of significant not significa or insignificant significant is maybe not as useful and yeah you can do all your statistics proper if your study is is flawed in the in the setup that won't help and yeah the video had a nice example this chocolate makes you lose weight where they just designed the study so that it leads to this outcome i think you've seen it all

39:25 maybe the point where we want to show this okay that was the explanation of the p-value and that's the one i wanted okay have people check each other yeah the bias that's that's absolutely that's partly done making sure people register studies so that even if they fail they get reported larger samples increases the cost so yeah i'm up for it yeah peer review in principle should work there's a lot of flawed incentives there as well it's a thankless task that is not paid for and usually not recognized much uh yep

40:31 okay yeah seems people are interested in this uh it's definitely it's definitely an important thing to fix yeah i think this is this is a big point as well one one has to live with the fact that not every study is new and exciting and we should reward this even if it's just for making scientific knowledge more rigorous all right the last thing i wanted to list again is misunderstanding what p-values give you because still even if you say we don't really want statistical significance as a binary concept the p values are useful but they're easy to misinterpret and so they're still the best thing the best tool we may have maybe uh together with bayesian inference but they need to be in interpreted carefully so just rephrasing same thing as before the p value is this weird thing it's how likely is it to see the data if you assume the hypothesis that you want to refute is really true so if that p-value is is large what that means is basically you haven't learned anything it doesn't mean that the null hypothesis is true in particular p the p value is not the likelihood that the hypothesis the null hypothesis is true it's just it's not it's not it's something different it's like it of your data

42:04 given the null hypothesis but there you already assume that i hope null hypothesis is true so you can never get a statement from there saying something about how likely it is that this is true if you already assume it

42:21 nevertheless you see this kind of as an implicit reasoning many times just don't do it the other thing is yeah be careful about what is exactly the null hypothesis and what does refuting it imply in our case um it said the mean is is equal so refuting it means it's it's not equal and you might have statistically significant evidence for that so your data is very strongly supporting that the means are different but it doesn't tell you how much different doesn't even tell you the direction of the of the difference so whether it's significant in the everyday language meaning is not clear and needs to be analyzed separately

43:10 okay i think that was a lot of material even though not much mathematical detail do you have any questions about this for now