Video The Scientific Method, hypothesis testing, pitfalls

0:00 <Untitled Chapter 1>

0:03 all right welcome back to communicating computer science uh today I want to give a little primer on what empirical science looks like and um a qualitative statement of what statistics is used for we won't dig very deep into that but I think to the extent that I you want to be able to read papers from empirical science in uh Humanities um I hope that this helps with it some of it may have been covered in school for some of you but my experience from the past was not so much actually for most so you I'll ask you afterwards how much of this was new uh so we have the scientific method as how do we learn things when there's no definitive truth um a little bit of statistics and then coming back to the pre-session task about what could potentially go wrong uh the goals are give you some vocabulary um specifically at the end for for papers in our contexts computer science education or education research in general uh yeah I think that might be useful um have some intuition about what statistical tools are used for and how they can be abused um and yeah leading into spotting these pitfalls hopefully now we'll do this at the very basic level but the reason why I'm doing it at all

1:38 is that you haven't probably seen it at least in University Computer Science is not a natural science it's a bit weird what it is actually is it a structural science more like mathematics or theoretical physics where we live in an abstract world that we construct ourselves and we just prove theorems in this abstract world for some part of computer science that's definitely true the part that I teach in my other class for example basically lifts detached from from reality for a long long time and then only touches back in some parts but of course he's also an engineering discipline and so it's it's one of these if you want you could make an argument that maybe physics was like this before and then it split into several fields and now you have mechanical engineering and electrical engineering ex-engineering which is no longer counted as part of physics and what's left in physics is more of the uh the pure science that looks for understanding and maybe also this structural bit in the theoretical branch of it computer science hasn't split we have all of that still under our umbrella what that usually means is that the curriculum at Uni is quite full and there's no no wiggle room for any things they're not not necessarily needed and the

3:00 scientific method is one of these things that are deemed to be not necessarily needed even though in some parts even in computer science I think it would be quite useful um the key difference often is in computer science it's comparatively cheap to get data if you compare that to a psychology experiment where you have to recruit people you have to go through an Ethics Community to get your experiments allowed in the first place then you have to run it you have to pay people you have to debrief them and it's a lot of effort before you even get started and you won't do this with a hundred thousand people normally no way it's too expensive you'll do it with much smaller numbers whereas computer science often can just run things on computers many more times so we're a little bit um spoiled in the sense that with more data you can repair what we lack in statistical finesse but a bit more structured methodology wouldn't heard in some some cases

4:08 okay some terms uh specifically those that differ from how we use them day to day and I'll I'll have to be careful myself to not mix the two up when I'm when I'm talking about this this lesson uh what's the theory if you just say that in in common parlance it's is something that you think may be true right but you don't know so specifically like a crime investigation could have the theory that this is the that the suspect is the murderer uh these are that's what we often call Theory uh in the scientific method that's quite different so their theory is also something that we haven't proven but it's uh much more theoretical much more comprehensive it's a set of ideas that hopefully explain how things work in the background uh but it's it's really um often a bit too abstract just to concretely uh test that all right next one is the experiment I'm definitely prone to doing that as well we say experiment if you just mock mess around right without much uh care so um experimental things could be you just try something out randomly that's not what a scientific experiment should be specifically in this scientific method empirical science an experiment is a very narrowly defined thing it's a specific test specifically designed to

5:09 Experiment

5:39 test a claim a hypothesis that you have and it has to be of a specific form otherwise it can't possibly give you the outcome you want so that's much more narrow on on the right uh yeah statistics is another yeah if we say statistics Kellogg really we just mean any sort of collection of numbers maybe if you do a survey the statistics is just the average happiness of students and you're you're happy with that as your statistics uh terminology

5:53 Statistics

6:11 but in science again statistics is more a deeper subject say it's a branch of mathematics that deals with how can you design mathematical tests that tell you with quantitative

6:35 with with in a quantitative way how much you can believe a certain hypothesis um we'll we'll talk about this in in much more detail um and the last one is is an important one uh significant commonly is just anything that's important but in this context of of Statistics usually it's called a statistically significant and that's again a very very narrow term that has to do with the p-values and we'll come back to that in more detail I wanted to list it here is that uh is this at all something you've seen the right hand side foreign

7:20 What's the Scientific Method

7:23 so what's the scientific method this is all about not talking about mathematics or things where you can prove things in deductively you can define something then postulates some theorem and come up with a formal proof then you don't need all this for everything else all the Natural Sciences the best we have is the scientific method and it's it's a cycle because it's not so clear um how to ever stop but you start with some observation in the in the real world like apples falling down so something seems to pull them down right so you come up with a theory maybe there's a thing like gravity a force that pulls them down uh then the tricky bit is you want to operationalize how to test that theory you want to make a a concrete statement that you can test and the trick here is you need a falsifiable hypothesis you need something that you can prove wrong with data so that you can learn something from it and the way to test the hypothesis is called an experiment and usually the experiment gives you some new data some new observations and then that returns to the beginning hopefully with a new observation

7:42 The Scientific Method

8:43 the annoying bit is you can't really definitively prove anything with the scientific methods because it's always it's always just one hypothesis that you uh that you refute um but that's that's that's hard it's hard to uh have any definitive answers you can only ever build up more evidence supporting your theory

9:09 but there's a few tricks how we can get a little bit more out of this a key a key thing about all of science um hopefully nothing new not everything that you can claim is is a scientifically testable hypothesis things that evoke something that's out of our universe in a way by definition can't be tested so in this example if you wanted to prove scientifically that there is a God you may never find evidence but someone can always say yeah but that's that's God's Almighty so he just hides very well and there's no way for you to to ever refute that that claim so we'll basically ignore or exclude any such statements we say this is outside of the realm of science there's still many important philosophical questions that we would like to get an inkling on but and not in this direct way sometimes you can break it down into some some other questions so if you if you think about God as a Creator then maybe you can ask questions about how did the universe come about or how did life start these are questions that relate to this but they become scientifically testable because they restrict the question and they take the out of this world part out of the claim

10:29 the more uh down to earth problem is it's not usually so clear-cut that things are true or false um in in several ways uh first of all we usually have some theory in mind that is simplification of what's really going on uh every physical physicist knows that what you put in formulas is an approximation and an oversimplification it might still be useful for engineering but it's not exactly the truth so you expect that there's some deviations and uh more on the on the Practical side you can't usually measure perfectly

11:10 so what we need is not just a way to say we refute this hypothesis but a quantitative way of doing that to what extent does the data refute the hypothesis and that's where statistics comes in

11:27 let's bring this to uh our our context um now flipping the units means that you haven't seen this so just ignore what the theory here is uh I'll I'll explain that next week I think so an example from from Computer Science Education could be this there's some theory that says people are motivated expectancy value theory is really a simple thing it says people are motivated when two things come together uh they have an expectation that they can succeed they can reasonably expect to succeed and they see a value in succeeding it's something that that means something to them if you have these two things together people are motivated and if if one or two drops to zero that's usually a problem uh the theory is as big as this there's no quantitative part of that because you can't easily put a number to either of these okay but uh that's a theory now you can't really test this uh but you could say one one implication of this could be if I give people a choice then they have more control over the task and so hopefully that enhances their performance because they're more motivated because they can more they can pick things that they care about and they can have a better uh feeling for

11:48 Motivated Expectancy Value Theory

12:47 whether they succeed all right that's uh how you can come to an hypothesis so the concrete statement here is this that you give people choice they do better right now notice that this hypothesis might well be true despite the fact that the theory could be wrong there could be other reasons why this is true without the theory really being behind it that's why if we confirm this hypothesis that's evidence for the theory but it can't definitively prove it

13:23 we usually have to narrow this down to make it to make it possible but that's a gloss over this the the tricky bit is how do we test this if we know that all the scientific experiment can do is review The Thesis we would like to prove the thesis prove this hypothesis we would like to show that this is true not not refute it right so we need a trick

13:52 all right first of all uh how do you set this up so that you can isolate any other influence uh and that's that's another tricky bit so that's uh that's about designing experiments if we take our hypothesis again then the the gold standard is to do a controlled experiment which we work as follows in this in this easy case you have you have a group of students and the important what I should have highlighted here is random you randomly assign them to two groups one that is testing uh your claim by having the choice and the control group which should otherwise be entirely the same ideally identical to the experimental group but is not getting the treatment in this case the choice so uh the experimental group writes about one or two topics say to keep it easy they choose and the control group gets randomly assigned topics now this you already may see problems arising what if all of the experimental group pick topic a because you made a mistake topic is much easier then the experiment might be flawed so there's a lot of details that are um where where this the outcome is sensitive to the details you should have topics you would hopefully want that the experimental group chooses roughly both

14:09 Controlled Experiment

15:20 topics uh equal number of times and when you assign the topics you would try to mimic the number of topics that were picked by the other group there's ways to kind of alleviate that a little bit but it can be quite tough to set this up that's what I'm getting it okay both groups write the essay and you mark them uh of course without uh so ideally double blind which means um can't be double-blind if the the people choose right it's you can mark it without looking at which group the kids are in so that you don't influence it subconsciously but it can't be double-blind in the sense that the kids don't know which group they're in because they get to choose or not get to choose

16:04 all right uh then once you've done this experiment you would compare the results um and now you have to quantify really what doing better means how much better is necessary so that it's not just random noise because kids are are variable in their performance and it's a group of people with variable um with variable uh quality and essay writing say um so this is uh this is worth statistics again comes in because we need a way to control for the expected random variation

16:52 okay uh that still really hasn't shown us how to prove the hypothesis um that's the problem my site did we want to prove something but we can only refute with the methodology so the trick is we just negate what we want to prove okay that's why people uh talk about a null hypothesis that's the assumption that the effect you're trying to prove exists does not exist so um to make it a little easier here it's easier to talk about has some influence versus has no influence as opposed to has a positive influence that just makes the the statistics and the math easier uh hopefully you will be able in your analysis of the experiment to see that the influence if it was significant that it was positive not negative so it's it's often not such a restriction but there are there are more sophisticated statistical methods that can deal with the other hypothesis let's keep it simple though so let's let's say you weaken your hypothesis too there's some change if you let people choose the topic it somehow affects their mark if you negate that that means it doesn't affect it and um what we can formally say here is a statement about the average mark so we always expect that some students will do better than others and others

16:53 Prove the Hypothesis

17:20 Null Hypothesis

18:27 will do worse but we have two groups and we can take the population average the average Mark in the two groups so mu 1 would be the experimental group average and mu zero would be the control group average then our null hypothesis is the two are the same there could be a random noise but really it has no effect and so we expect the average to be the same and we can try to collect data that makes it implausible that this null hypothesis is true right if we see if we see in our experiment that the average is significantly different and now I'm using significant in the colloquial sense shouldn't do it uh they vastly differ then it's unlikely that the two groups really come from the same random distribution you you randomly assign students so there's some built-in Randomness it's a randomized algorithm uh it's unlikely that they really are uh unaffected if the difference is so big but uh how to quantify that is statistics

19:41 nah well okay in principle one way to refute your hypothesis would be to get the true average over all people then you would know for sure then you wouldn't need statistics but of course that's not very useful the other option is uh well sometimes you can actually do this fair enough um normally you can't so you take a random sample and in the design of your experiment I kind of didn't say where these students come from so that's another thing you have to pay attention to how do you re recruit the people in your experiment if you do this with often your only option the people you're teaching anyways you have to put in into the context so this is uh what that type of school and it's a high ability class or whatever whatever it is you have to give the context because that might have an influence on the validity of your hypothesis but then within the group you still randomly assign them to one of the two groups so it's still a sound experiment

20:49 I have really one slide just on the mathematics uh behind this and just exemplary so how can we how can we say that the data rejects our null hypothesis with a reasonable confidence how can we get the numbers in this so we have our two hypotheses again null hypothesis in our case no no change in marks uh the converse of that there is some change then we have the observations from the control in the experimental group uh I think capital x here would just be a list of marks or a set of marks that the kids achieved or multi-set

21:35 and then we compute this this mystical quantity the p-value which is um the the likelihood of seeing the outcomes we did get in our random assignment to groups assuming that the null hypothesis is true so you assume that the two groups really don't differ now you see some data and uh making some some assumptions how things are distributed you can compute How likely is it to see exactly these values uh and you see so uh there's assumptions on distributions that's um you where are you you have different tests so uh depending on what exactly you assume about the world what's your simplification you have lots of different tests and that's all these all these things with the cryptic names uh we'll just have a look look at the student t-test but there's there's many others um there's a reason why uh Natural Sciences and Humanities courses usually have a full term long course on statistics if you want to apply this you better know the assumptions of these tests and use the right one uh for us I think we're just consumers of this uh whenever we look at the paper that is published in a reputable journal the reviewers know statistics very well and this is one of the first things that they usually check other statistical

23:08 methods sound and reasonable for this for this application so we can in a way uh chicken out of this and say we just we just take what people in the experimental design did as as reasonable

23:23 and um well if we come back to this pre-session task later I can comment more on this it's not usually an inappropriate test that people chose that leads to this problem of the reproducibility crisis it's not usually the statistics that's at fault it's not that people assume something like a normal distribution and it wasn't too it was too far away from that the the problem is usually somewhere else and so we can pretend the world is okay here the basic principle for all the tests is you have your observed data and you take say the mean so you have the the average Mark that you observed then you subtract the expected average Mark assuming your null hypothesis now that sounds complicated but it's actually not that hard because the null hypothesis just assumes that it doesn't affect the kids how they perform so you could just take your class and somehow determine in an uh in another essay task before that what's the average Mark that the kids achieve in writing essays yeah if uh if they just get some topic so you can try to to get at this uh it's often not not that hard and then you have to have to have to divide by what's the expected variation you're you're expecting to see given

24:45 essentially how large the classes and how variable their marks are again you can do this uh prior test where you have them write some essay and then you don't just get an average mark from that for the class you also get a variation you get the standard deviation say of of the marks why do we divide by this well uh we're trying to refute h0 which says these two things should really be the same right if h0 is true and we don't have absurd noise we should get zero in the numerator now if we don't get zero that can have different reasons and noise could be one thing so we divide by the typical noise and so the result of this is if we have a large value coming out of this called the test statistic if we have a large value for the test statistic that means we have a small p-value because then it's very unlikely that this data is due to random noise that the difference you see from what you expect is due to random noise if the difference is big it's more likely that there's something else going on some true difference all right

25:56 Example for the Student T-Test

25:59 and here's one example for the student t-test um so the Assumption there is that both groups have a normal distribution you don't know the mean but you know the standard deviation uh it's one of the simplest tests it's also one where I don't know uh you don't often really know the standard deviation very well but in in our case you could try by as I said running an essay before and measuring it uh

26:33 so here uh we could we could take directly the the difference in marks as our observed data and the expected data would be zero because we assume that the means are the same you can compute that this is the standard deviation of the the difference between the two groups and then uh well that's where the test gets its name from there's a specific distribution and you can figure out that the test statistic has this distribution so you can get the p-value from a lookup table okay have you seen student t-test somewhere how many have not seen it

27:11 how many have seen it half half yeah there's there's many more there's uh many more assumptions that you can try to get get away from for the data again for us um let's let's ignore that but on this level of abstraction the tests are usually all the same

27:37 okay so we have a measure now we have a means to refute hypotheses by collecting data and seeing how unlikely is it that this arises under the null hypothesis just from random noise as opposed to being some other true reason so why why is this sometimes problematic several points now in in principle this works right this is how all our scientific knowledge uh is is collected and there's some fields that are not really affected by the reproducibility crisis that much for example particle physics is doing fairly well because they are stricter in their experimental setup so it's not like this method in in uh in itself is flawed in any way but there's a few things that are tough one thing I told you about this gold standard of controlled experiments well it turns out sometimes that's really hard to do I don't know if you know what this story is uh I learned it's an urban legend so the Forbidden experiment is to keep race uh children and not speak with them at all and so there's several uh pseudo historical accounts of of uh Kings who have tried that out and came to various basically the story is always the kids just die uh but I think it's very very shaky evidence of probably no one has

28:31 Gold Standard of Controlled Experiments

29:03 ever done it it's not clear what would happen uh but it's clearly unethical to do this experiment it would be interesting to find out what happens if you raise children without talking to them at all would they develop their own language so in in the past people oh maybe they would come up with uh the the true language of God uh maybe they would just invent something on the spot or maybe they would they would die who knows uh I I certainly don't think we want to find out the experimental way

29:37 um and other experiments it's just very expensive to do them uh this is often I don't know if you would like to find out uh okay causing cancer is probably on the unethical side as well but uh something less controversial I don't know does drinking coffee eventually make you more or less healthy now if you want if you want to focus on the eventually you might want to have Health Data 10 years into the future it's tough to recruit people to stay committed to an experiment for 10 years these are expensive it's not impossible to do but often people try to get something cheaper and there's a way to get at least some information that's from an observational study so there you don't randomly assign people to groups and you don't have a strict control group you just collect data and see what happens naturally so you collect data and see if people you ask people do you drink coffee and you ask them are you healthy I mean you want to formalize that a bit more but you can collect such data and try to find patterns okay now obviously there's a problem suppose everyone who drank coffee also did well let's let's spin this positively they also did a lot of sports and a lot of other things and otherwise

31:03 lead a super healthy life so that you might find that coffee goes together with super healthy but it might just not be the coffee that causes it we've seen something like this with the polar data set for for widening participation uh we can't really do an experiment with kids saying um we we randomly assign you a postcode and see how well you do in higher education right somewhere between those two but we have the data that happens to come from from how people behave so it's exactly this this kind of thing it's very very uh widely spread so bad cautionary sign observational studies can be helpful sometimes the only thing we have but it can only ever ever observe correlations Things That Vary together you can spot patterns but you can't easily say this causes that uh strictly speaking with an observational study you can never ever do that there's just no it's not possible from the setup of the experiment uh there's there's many many examples of this like here's one uh the typical thing that happens as for the coffee thing there's a confounding variable I should have put that on the slides as a as a term but uh maybe if you've probably heard this before so a confounding variable is something else

31:47 Observational Studies

32:18 Confounding Variable

32:29 that is the true cause but it happens to co-vary with what you study so confounding variable would be living healthily you want to measure how coffee affects Health but there's this third thing that happens to be connected with coffee by chance and so you can't really separate the two things there's a fun website where you can randomly they have lots of data sets and they just randomly pick out some highly statistically significant correlations between two things and often it's complete bogus uh well you can you can have a look at that if you want

33:11 before I come to the reproducibility crisis I thought it's useful to bring this back to the essays and the articles that you you're reading now this is this is not something super well uh vetted um this is something that I'm from from reading a bunch of these articles came up with in uh well taxonomy is is pitching it high there are different levels of what things do and then publish as an article in educational venues it could be just that they they've done something and they feel they've learned something so they want to share that knowledge it's totally legit and so people publish such a experience personal experience reports again these these terms I I made those up so uh that's nothing that um is is very well agreed upon I think it's it hopefully is useful so have a look at your articles and see whether you can classify them so this one might just share uh experience so you don't make the exact same mistake twice say but it's quite limited because you don't know there's there's a lot of things that are unclear it's usually without a control group so someone did something and it worked well awesome um maybe it would have worked well without him doing anything different from what's the standard right

34:34 a little better can be to ask the participants uh what they felt this is often qualitative so that's another orthogonal axis uh that I'll come to so you don't just rely on what the person who or who ran some activity says but also what the participants say it's still fairly weak because you don't really have something to compare it to right it worked well compared to what right and maybe the participants just were nice in the survey observational study we talked about this before you can collect data uh say an educational setting you collect how many schools have access to Apple devices and what's the average marks at GCSE level maybe there's an interesting correlation maybe not I'm not even sure if it's positive or negative um and even so it's a correlation so it's not clear how useful that is but you can try to learn something like this I would still say it's um this is not necessarily A ranking but it could be a little more useful than the other two because it's just less local it's a bit more comprehensive if the data includes more uh more settings whereas it could just have been you know this was uh super selective private school with all well-off kits with private tutors and

35:06 Observational Study

35:57 your teaching method works well there that's great but maybe it doesn't help someone teaching in another setting hopefully observational studies are more comprehensive and of course the gold standard is some controlled experiment I've seen this often in advertisement marketing or software people call it a b testing and it's really the same thing so for for software you could say a website you have two designs for a website and you want to find out what works better what keeps people say uh the standard metric what keeps people looking at your page for longer so you randomly roll out the page in the two designs depending on when people visit right just flip a flip a dial flip a coin uh but you record what they do that's a controlled experiment in the sense that you randomly assigned them to one of the two groups and you could have a control group if you want if you have a status quo page so that's um that's just a different name for it

36:07 Gold Standard

37:08 I don't want to say that okay so you can have either qualitative results or quantitative metrics I don't think one is strictly better than the other for the following reason uh so qualitative experiences the weakness of courses is sometimes not clear it worked well but is that enough better than something else to bother with the effort it's hard to to judge based on that quantitative metrics you can have the problem that you measure one thing and it's just a little a surrogate of the real Target um so the the trick there is to find a good metric that reflects really what you want to see uh in in that sense we often have test scores which is all right because a lot in life depends on test scores so if you want you could say that's the real thing we care about it's not really the real thing we care about though right we would maybe focus on on the skills and mindsets or things like that or maybe a different metric could be uh income universities sometimes do that right they show if you go through our studies with some percentage of some probability you make so and so much money after five years or so uh whatever you pick as the metric it

38:25 shifts the picture so that's another danger so people often try to mix these two a bit so have a look at the essay topics that you've picked out and the Articles you found and try to see what they uh what they do again this is I'm not saying that one is not useful or the other is is the only thing to go for but if you put a weight on the evidence how well does this support uh some argument you're making it's useful to keep in mind that some things are inherently Limited or Limited in scope

39:12 so last bit uh what's this video let's let's switch to the slider answers it was a bit more open I think um does that work yep okay what's the p-value uh talked about this but it was 20 minutes ago um that's not wrong it's a measure of statistical significance I haven't actually talked about statistically significance yet right uh

39:44 yeah significance that's that's my definition How likely is the data assuming the null hypothesis good specifically the p-value is not that will be on the slide as well is not the probability of the hypothesis you're trying to prove H1 that's not the same thing unfortunately uh

40:10 and the second bit is uh what can we try to do to improve the reproducibility crisis maybe I should summarize that first so what with the video so okay uh the veritasian videos are often um more neutral in the video whereas the title and the the covers the screen is always a bit uh click-baity in itself so he points out that there are some false positives that you inherently expect from the scientific method that's kind of inevitable just because we we have noisy data and random noise but we see a few of these results too much to be really happy with and there's a number of reasons uh it's incentives on the science side um more a bit more subtle is this publication bias it's just not as exciting to publish that you did something and it didn't work there's also problems with the statistical significance of how sure are you it didn't work because uh it's actually wrong or could you just not show that it is true with your method the letter is the only thing you can really say it would be interesting if you could say oh the I had this hypothesis it turned out it's not true and now I know it's not true but your statement is just I tried to

40:41 False Positives

41:09 Publication Bias

41:40 prove this hypothesis and I failed that's just a harder cell um I specifically and feels like psychology people joke that I it's all it's the psychology of undergraduate psychology students because that's the the easiest to recruit group or experiments um yeah some sometimes you have influence like big companies funding research and then it's it's not clear how independent it can be

42:23 Focus um to reproduce research even if we already know the outcome it's a good thing uh I think everyone agrees we want to do that is one of these Everyone likes change but not to change

42:47 yeah I hope hopefully you don't do that uh we'll talk about the p-value hacking yeah replication studies repeat the whole thing several times that's the the best attack we have yep

43:06 and maybe accept that uh there's value in just repeating something or disproving a certain way to prove something the fail doesn't need to be your personal fail it could be this is not the way to prove it whether or not it's true okay

43:24 Statistical Significance

43:25 I want to comment on this on this statistical significance bit um did I skip this on the slide or was this only at the very beginning let me go back to the where it was um defined here so yeah I hadn't introduced p-value at the time but now you know what it is it's the likelihood of the data assuming the null hypothesis and so what is statistically significant well that depends on your field in Psychology or and other Humanities often people defined an effect is statistically significant if it has a p-value of five percent or better or smaller right so if the probability that the data you get the likelihood of the data is at most five percent assuming the null hypothesis then you're happy to accept the counter hypothesis and that's what uh what the video picked up five percent well that's one in twenty right so one in 20 we expect to uh produce a false positive a statement that I have proven the hypothesis but it really was just random noise and if you go to something smaller like particle physics I think is even putting squeezing more zeros in there by collecting more data you can decrease that likelihood so statistical significance is often seen as a binary thing that's what I um

45:01 add on this light down there so people report various effects and then they say yeah hear the difference between the groups was statistically significant and here we it was not and that's maybe unfortunate for two reasons first of all we have these false positives and so the yes or no is not really useful at least they should always report the p-values which is what people mostly do but still the interpretation then relies on this binary thing and the second problem is you can have a highly statistically significant difference between your two control groups the essay marks but the difference is 0.001 marks so it can be highly significant but super small in in absolute value and it's very easy for people to confuse the two things because common parlance significance is just important without much more uh clarification

46:06 okay that's that's a big No-No uh if you do a study you have to say what you did in all details so that people can check if you made methodological mistakes okay and to hammer down this one one last time the p-value is the likelihood assuming the null hypothesis it's not the likelihood that the null hypothesis is true that's the other way around right you can't really talk about this uh in the in these terms so you can't say uh if your p-value comes out at four percent that okay it's only four percent likely that I'm I'm wrong it's not not quite how it works uh yeah

46:51 I think that's all I wanted to say uh have a look at your articles and try to look at them through this lens just be skeptical um I mean appreciate what people share with you but be skeptical to how much that means uh for for your claims how much can you rely on on their statements look at the evidence don't don't look at what the summary sentence says