Results Are In From Psychology's Largest Reproducibility Test: 39/100 Reproduced 174
An anonymous reader writes: A crowd-sourced effort to replicate 100 psychology studies has successfully reproduced findings from 39 of them. Some psychologists say this shows the field has a replicability problem. Others say the results are "not bad at all". The results are nuanced: 24 non-replications had findings at least "moderately similar" to the original paper but which didn't quite reach statistical significance. From the article: "The results should convince everyone that psychology has a replicability problem, says Hal Pashler, a cognitive psychologist at the University of California, San Diego, and an author of one of the papers whose findings were successfully repeated. 'A lot of working scientists assume that if it’s published, it’s right,' he says. 'This makes it hard to dismiss that there are still a lot of false positives in the literature.'”
39/100 is the new passing grade. (Score:5, Insightful)
Is there a valid reason we accept studies that have not been reproduced at least one more time to truly vet them before the community?
Logistics, resources, patents, or a need to just plain bullshit people. I'm sure there's plenty of excuses as to why we don't, but doesn't sound like we have a whole lot of good reasons why not.
And those that are labeling a score of 39/100 "not bad at all" should have their head checked. Enjoy your legal fun from that ball of lies.
Re:39/100 is the new passing grade. (Score:5, Insightful)
Funding.
Assuming for the moment that the reproducers were not particularly more skilled than the original scientists, you can't go from '60% not reproduced' to '60% wrong'.
Assuming there is some actual effect being investigated, one reproduction will not get you to 'good' levels of surety about the effect. To hit '95%' - you're going to need likely over ten reproductions.
Re:39/100 is the new passing grade. (Score:5, Interesting)
Assuming there is some actual effect being investigated, one reproduction will not get you to 'good' levels of surety about the effect. To hit '95%' - you're going to need likely over ten reproductions.
One study != one sample. Each study should have enough cases to make it statistically significant. The problem is related to issues with the sample population or systematic flaws in what you're measuring. To bring it into the realm of physics, if we do a high school gravity experiment and ignore air resistance we can make as many tests as we like, check for measurement uncertainty in our clocks and whatnot and put up some confidence intervals that are still horribly wrong. It's very hard to isolate and experiment with one tiny aspect of the human psyche and most of the problem is the result is nothing but either a statistical fluke or quirk with the people tested that doesn't generalize to the general population.
Re: (Score:2)
Just because a result is statistically significant doesn't mean it's correct, although I can't search for the appropriate xkcd here at work,so google for "xkcd jelly beans" or "xkcd significant" to find it.
Statistical significance means that the results you got would be gotten by random chance by a certain probability, typically 5%. 5% is not some magic number, and there's no theoretical significance, but it seemed like a reasonable value.
Therefore, if you take a batch of statistically significant res
Re: (Score:1)
The way it's supposed to work is that your peer-reviewed paper get's published (peer-review having established that it is at least basically methodologically sound), and then people can read your paper, and if it's interesting and relevant enough then they might decide to try to replicate your results. If you can't publish first, then how are you going to find people to replicate your work?
Re:39/100 is the new passing grade. (Score:5, Insightful)
Gah. I have mod points but want to add to this conversation.
The point of publishing is to share results of an experiment or study. Basically, a scientific publication tells the audience what the scientist was studying, how they did the experiment, what they found, and what they learned from it. The point of peer review is to review the work to make sure appropriate methods were followed and that the general results agree with the data. Peer review is not meant to verify or reproduce the results, but rather just make sure that the methods were sound.
Scientific papers are _incremental_ and meant to add to the body of knowledge. It's important to know that papers are never the last word on a subject and the results may not be reproducible. It's up to the community to determine which results are important enough to warrant reproduction. It's also up to the community to read papers in the context of newly acquired knowledge. An active researcher in any field can quickly scan old papers and know which ones are likely no-longer relevant.
That said, there is a popular belief that once something is published, it is irrefutable truth. That's a problem with how society interacts with science. No practicing scientist believes any individual paper is the gospel truth on a topic.
The main problem in science that this study highlights is not that papers are difficult to reproduce (that's expected by how science works), but that some (most?) fields currently allow large areas of research to move forward fairly unchecked. In the rush to publish novel results and cover a broad area, no one goes back to make sure the previous results hold up. Thus, we end up with situations where there are a lot of topics that should be explored more deeply but aren't due to the pursuit of novelty.
If journals encouraged more follow-up and incremental papers, this problem would resolve itself. Once a paper is published, there's almost always follow-up work to see how well the results really hold up. But, publishing that work is more difficult and doesn't help advance a career, especially if the original paper was not yours, so the follow-up work rarely gets done.
tl;dr: for the general public, it's important to understand that the point of publishing is to share work, peer review just makes sure the work was done properly and makes no claims on correctness, and science is fluid. For scientists, yeah, there are some issues with the constant quest for novel publications vs. incremental work.
-Chris
Re: (Score:2)
That's a really good summary of the situation.
I do think that there's one more important factor. The flip side of reproducibility is utility. The whole reason that we care about reproducibility is that it means that we can put things to use. We demand falsifiability because if it can't be put to the test, then it's not so much "wrong" as "worthless", i.e. Not Even Wrong. If it can be reproduced but never is, what did it matter in the first place?
That's not the same kind of epistemological issue that falsifi
Re:39/100 is the new passing grade. (Score:5, Insightful)
Please remember that this applies to Psychology, a field that is rife with lots of historical issues (and it is getting better).
In the past, they've practiced thinly veiled religion (Freud, if you don't believe me it's because my unverifiable model explains it, and you are in denial)
They've overstated their findings, to the tune of "and because of this experiment, we can extrapolate that EVERYONE is just the same!" (Stanford experiment, one of the ones with reproduciblity problems too, coincidentally).
They've put up roadblocks to proper scientific evidence gathering (and so this experiment was done before we adopted an ethical code that made verification of its outstanding results, that is reproduciblity, possible, but we are going to believe the conclusions anyway).
It was always called a "soft" Science because that way they could dodge the bullet coming from the real Sciences. However, when you read some of their works (especially their older works) you begin to see a pattern. They come from a history that probably poised them to have a long and hard road to understanding much of what we really do. In their defense they were attempting to build a model of the "mind" which is something that they assumed existed in a particular way, but didn't really test (it took a long time for Turing to come around). Finally, they are burdened with a lot of thinking that doesn't meet Philosophical rigor, because they shore it up with testing that doesn't meet Scientific rigor.
I'm glad to see the new wave of Psychology coming through. The now base a lot of their findings on biochemical analysis and stronger testing (including better attention to controls and double blind testing, which to their credit, they invented). It's just disheartening that the field lacks respect in other ways because in every intro to Psychology class they keep pushing the sensational "Dogma" experiments as facts when in reality they often fail to reproduce the results.
Re: (Score:1)
Re: (Score:2)
Sounds an awful lot like that other "soft science", Economics. Some researchers are finally getting away from the 'revealed wisdom' of Friedman and company, and hopefully Chicago School Economics will soon end up in the dust bin of history with phrenology and vital humors.
Re: (Score:2)
Keep hoping. Also remember there are no Keynesians. Keynes would run a surplus in good times. 'Keynesians' are just 'print and spenders' with 'intellectual' cover circle jerking.
Re: (Score:2)
Lets show who ran the house on the same graph!
There are no Keynesians!
Re: (Score:2)
The key difference between psychology and economics is that there's money in getting psychology correct (because it'll help you sell crap), while there's money in "proving" whatever economical hypothesis benefits the current elite.
Liars study psychology and practice economics.
Re: (Score:2)
I'd argue the opposite -- it's the "Physics Envy" i.e. basing psychology on the classic scientific model of objective measurement that makes it worse, and now it gets 39/100 score. And we don't even know how important those 39% that passed are -- it may be some stupid stuff few people care about while the big ones failed.
The reason is in psychology you simply can't measure reliably -- often times your measurement is asking people what they think or feel, or you observe some behaviors that depend on a thousa
Re: (Score:2)
Re: (Score:2)
Must suck for the few that are actually trying to do science. Not so much for the 99% pushing a agenda and faking it.
It's hard to do social science right and it has a danger of upsetting your world view. So nobody does.
Re:39/100 is the new passing grade. (Score:5, Insightful)
I think that there is a bit of a disconnect between scientific methods and things like psychology. I think it stems from the fact that it's easy to set up a thermometer, or an EKG, or whatever else and get those discrete data points, but it's difficult to measure things like "I feel better" or "I don't think about that the same way." I have a few friends who would swear up and down that EMDR therapy helped them out tremendously, but I don't know if there's a single way to gather data that would actually quantify what happened to them. I see the results not so much as psychology being flawed, but more about the difficulty of simply gathering the type of quantified data that a scientific study would require.
Re: (Score:3, Funny)
Re:39/100 is the new passing grade. (Score:4, Interesting)
Psychology, sociology and other social sciences have always been given special treatment precisely because its difficult in some cases to get two independent groups together to rerun an experiment in the first place - and if you try and reproduce an experiment done in the 1950s today, are the results due to poor scientific method in the original experiment, or because the evidence gathered was misinterpreted, or because society has changed which means the results have changed?
Re:39/100 is the new passing grade. (Score:4, Insightful)
Many sciences, not just the ones you listed, have at least some problem with reproducibility. Verification isn't nearly as sexy as coming up with a new idea.
During my academic days, all the focus was on new work and literature reviews, but only one professor seemed to (defeatedly) care about verifying the results of other researchers. That doesn't get the funding.
Re: (Score:2)
This always surprises me. I know how contentious the physical sciences are, and from what I understand the soft sciences are even worse. I'd expect lots of people with differing opinions would be out to refute studies contradicting their assumptions. Maybe the flow of income makes this less possible?
Re: (Score:2)
It's amusing when two social scientist, neither of who understand basic statistics, get into a pissing contest.
What's not amusing is they decide who 'won' the argument by polling their coworkers feelings.
Re: (Score:2)
It's not just about trying to prove someone wrong. But few people want to spend their energy on proving someone else to be correct, although that is arguably the best kind of science we can do.
And suppose you do try to prove them wrong, and fail to do so. What's in it for you? Too few papers are submitted or published where the author's hypothesis is shown to be flawed.
Re:39/100 is the new passing grade. (Score:4, Funny)
And those that are labeling a score of 39/100 "not bad at all" should have their head checked.
They did, but only 39/100 of them found anything out of whack.
Re: (Score:2)
Because the methods seem reasonable and no one else has looked. It is the best information available.
Re: (Score:2)
What do you mean by "accept?"
When you publish in a journal what you're really saying is "hey, look what I did! What do you guys think?" The point of publishing your work is to tell people what you found so they can evaluate it and try to reproduce it themselves.
If you mean that people, both scientists and the layman, shouldn't believe something is true until it's reproduced? Absolutely that would be a good idea. Even better, wait until a good meta-analysis is performed.
Incidentally, the FDA generally re
Re: (Score:2)
I had a friend who worked for the FDA in animal feeds. She said that all she wanted to recommend approval was three honest studies significant at the 5% level, and was really annoyed at companies who couldn't understand that. (Control group? What's that?)
Re: (Score:2)
Are you talking about "Accept" as in accept for publication in a peer-reviewed journal? Because that's the entire point for a PRJ, getting other qualified scientists to read and possibly call bullshit on your findings if they can't replicate it. Engineering papers (from China!) get debunked all the time. The difference is, very few of the many stupid news papers don't c
Re: (Score:2)
Is there a valid reason we accept studies that have not been reproduced at least one more time to truly vet them before the community?
I don't think it really works like this. In practice, most studies are totally uninteresting and their only purpose in life is either to not ever be read or to seed more uninteresting studies. Nothing of value is lost if they're wrong (and probably they often are). The studies that do matter are replicated because they're interesting enough that other people try to use them. So if a study discovers an interesting new effect or develops an interesting new tool then other (good) researchers jump on to the ba
Re: (Score:2)
Is there a valid reason we accept studies that have not been reproduced at least one more time to truly vet them before the community?
Well, you shouldn't. It's usually the press that blows a single study way out of proportion, because they have no understanding of how science works. Science *always* generates contradictory results early on as it gets the kinks worked out of a hypothesis. This is not some kind of failure of science, it's the way science is supposed to work. A critical follow-up attempt to check on some study's results is *of course* much less likely to reach significance, because of researcher bias either way.
The gold
Re: (Score:2)
And those that are labeling a score of 39/100 "not bad at all" should have their head checked.
At least they should have their studies replicated.
The comment that several of the failed replications were "broadly similar" but failed to reach statistical significance leads me to wonder if there has been any data cherry-picking in some of the original studies.
Re: (Score:2)
Will the first issue is that about the only thing we would have got out of the Large Hadron Collider is confirmation of the Top quark.
Various space probes would also be rather limited. For example MESSANGER would be limited to stuff that had previously been found from earth or Mariner 11.
In fields like chemistry the overwhelming majority of studies produce results that are pretty much what you would expect so there is little reason not to accept them (other than the yield always assume the yield is inflated
Re: (Score:2)
Is there a valid reason we accept studies that have not been reproduced at least one more time to truly vet them before the community?
The point of papers [in real science] is to say, "we did this, here's what we found". It's not to announce a beacon of new Revealed Truth. That's largely the fault of science "reporters" looking to sell advertising space.
The papers are themselves the invitations to replicate.
The problem is the government science-funding model is largely based on fame and popularity, and doi
Re: (Score:2)
There is. Not to *accept* the study but to be on the lookout whether the pattern that the study claims to exist really exists, if it useful for you, so you can validate it first hand.
Example (not a great one but will do) -- suppose a new study that seems reasonably well done claims that drivers of black cars are significantly more prone to road rage. You drive a car, so if this pattern holds, it's relevant to you. Then from time to time when you see a black car on the road you give it a little extra attenti
Re: (Score:2)
The bible was last edited by the council of Nicea in about 250AD. It gets re-translated regularly. Look at the various translations of 1st Timothy 2:12 for how politics can affect that. Who decided to remove 'she should be quite'?
By your standard, that makes the bible about 1/10 as reliable as the Hindu books about flying elephants having battles.
Re: (Score:2)
It was a full tilt edit. How did the poor old lady giving a single copper story show up in John? It was never there before the edits. There are multiple fragments that cover the verses affected. It wasn't there before Nicea. They just paraphrased it from Mark.
They fully consolidated the bible. Synchronizing 'good' gospels while burning others etc.
false positives (Score:5, Insightful)
'This makes it hard to dismiss that there are still a lot of false positives in the literature.'
An even more widespread problem is that there are a lot of true negatives that aren't in the literature.
Of course, this is a problem in most scientific fields, not just the "soft sciences" like psychology. I'm occasionally impressed by a researcher who publishes descriptions of things studied and found to be not significant, but this doesn't happen very often.
Re: (Score:2)
Re: (Score:3)
False positives are why we have to insist on doing research to verify past studies, and they are inevitable for various reasons.
Accepting that, and funding verification studies, is how a science goes from soft to hard.
Re:false positives (Score:5, Informative)
You missed the GP's point: the problem is not that true negatives were found; the problem is that they were not published. Because they were not published, future researchers might waste more effort re-discovering them.
Re: (Score:2)
You missed the GP's point: the problem is not that true negatives were found; the problem is that they were not published. Because they were not published, future researchers might waste more effort re-discovering them.
Indeed, though there is often a more insidious effect. Suppose there's a claim that treatment T is effective for medical condition C, but it's actually a "placebo effect". If one study showed a (perhaps small) effect, and other studies showing no effect aren't published, a lot of money can be made selling T to customers. If the true negative results are published, the makers (and prescribers) of T lose that income.
This is one of many reasons for the low level of publishing "negative" results. Another
Re:false positives (Score:5, Informative)
More to the point, this is a problem of funding in all fields.
No one wants to pay for basic research, even if it yields other useful ideas for further research. Unless it hyped to high heavens, the possibility of getting dollars is nil.
Gent I know was a decent researcher who got demoted to teaching community college. After a year of not being able to produce the "right" (read: able to secure further funding), he was canned, and another researcher who was more accommodating to fudging results got the position.
It's not like the experiments were going to be reproduced anyway. Just fodder for additional grants because you produce "results".
Re: (Score:2)
You shouldn't be impressed by people who publish insignificant results. Insignificant doesn't mean "not true." It means inconclusive. You should be impressed by people who go the extra mile to turn their not significant results into meaningful limits on parameter estimates (setting limits on how big an effect could be). That's done a lot in physics but only occasionally in other fields.
The only reason for publishing inconclusive results is to allow somebody to incorporate them into a later meta-analysis
Obg. XKCD (Score:5, Insightful)
I think it could have something to do with this XKCD:
https://xkcd.com/882/ [xkcd.com]
Re: (Score:2)
I think this paper is saying it is worse than this. Only about 5 percent of studies with 5% chance of results by random chance should be wrong. But this is saying that 60% weren't replicated. It's almost as if the researchers feel pressured to publish papers and they have a hard time publishing papers without results.
Re: (Score:2)
Let's say that *no* hypotheses are correct. 1000 studies are conducted. 50 of them find a significant result!
When we do a replication, 2 still pass! They *must* be true.
I know, I'm butchering the statistics, but the main point stands, because we see only the studies with p 0.05, significance doesn't mean what we like to think it means.
Frankly, I'm surprised they reached 39%.
Has Anyone Reproduced Their Results? (Score:4, Funny)
Perspective (Score:5, Interesting)
Re: (Score:2)
You need to put this in perspective. Sure, psychology is wishywashy field filled with pseudo science. But apparently their studies are about as reproducible as a bunch of the hard sciences fields. If there is anything that reproduciblility studies have taught us is that if there is around a 50% chance your result is correct than you are around the norm, in a great many fields. This 39% would make them about on par with what I remember from medical/cancer reproduciblility studies.
Based on the evidence you've presented here, I've come to this simple conclusion; The hard sciences are just as wishywashy and filled with pseudo science as psychology is.
And if those in the "real" science community want to cry bullshit over this conclusion, then fucking prove it with reproducible studies.
Until then, you appear to be either incompetent or corrupt, with the latter not being a far stretch when you consider the potential revenue involved in the medical community as a whole.
The scary part is
Re: (Score:2)
Welcome to science! In 10 years, a lot of what we believe today will be somehow invalid bullshit!
Say it with me: current theory suggests....
I'm an engineer. When the entire field of cognitive science rolled over on its back for K. Anders Ericsson, I said, alright, the research is funny, but the conclusions are useful. So it's all fucked up, full of bullshit and misunderstood data. Ericsson's hundreds of papers and books all boil down to one thing: experts become experts by a principle he calls "de
Re: (Score:2)
Do all the people that study liver disease have fucked up livers?
If no then they are not the same. Psych majors are _all_ batshit insane.
Re: (Score:2)
If it's published, it must be true? (Score:2)
It wasn't just published on the internet, it was published in a scientific journal!
Silly researchers. You're not supposed to publish science fiction just because a company paid you to write a story that matches their agenda.
Re: (Score:2)
I tend to seriously dislike the kind of comment that attributes malicious intent to researchers. I do not think that this is a problem with collusions. It is a problem that making a sound and reproducible experiement is HARD. It is easy to forget to report a phase of your experiment that you did not think about but that turn out to be important. It is also easy to have an implicit biais you did not recognize: an obvious one could be you did your experiment on sunday, so you excluded all the church goers.
Re: (Score:2)
At the same time, just like in any group, there are "bad" researchers and "good" researchers, and the ones performing due diligence and trying to be upstanding can still have a bad day.
The primary focus was to point out that even a scientific journal is not without fault, and there are even some that have been known to accept and promote complete drivel that was created intentionally to point out how flawed the scientific journal system can be in some places.
The general idea is not to make the researchers s
The bar for passing research is low in several fie (Score:2)
I'm trying to imagine the results for a similar review of computer science and computer architecture research from most universities.
Since there is nearly zero reproduction of results, limited validation, generally poor test content and few incentives to improve research quality I doubt the results would instill much confidence.
Quick plug for JASNH (Score:5, Informative)
Just taking this quick opportunity to post a link to my favorite journal, the Journal of Articles in Support of the Null Hypothesis: http://www.jasnh.com/ [jasnh.com] .
JASNH is one of the few places where you can submit a paper that says "we tested for X effect on Y and found no evidence that X affects Y". Generally this research is unpublishable and people will tweak parameters to get something career-advancing out of their research; I like JASNH because of the reminder that "falsifiability" can really happen.
Re: (Score:1)
Don't forget, those studies can be just as wrong.
Which gives a global p value of... (Score:3)
Great! With p=2E-65, studies in psychology aren't totally random.
Sure, replicability is the problem... (Score:1)
I'd say, credibility is the problem plaguing the field.
The experimenters really wanted each experiment to predict their theory... The less precise a science, the worse a problem this is.
In Mathematics, where absolute proofs are possible, and proponents of this or that public policy fiddle not, things are fine. But if a psychologist or, dare I say it, a climate scientist thought, that all odd numbers are prime, for example, they would'v
Re: Sure, replicability is the problem... (Score:2)
So, are all you skeptics paid shills, or just really passionate about your cause?
Re: (Score:2)
I, for one, am rather passionate about some odd numbers not being primes.
That there is an infinite amount of them strengthens my position, but having only a few exceptions is enough to invalidate the theory I cited as an example.
A floor sweeper contributes more to society (Score:1)
Re: (Score:2)
What amazes me is that someone with no scientific background can make such absurd statements about a field, of which they have absolutely no understanding, with such conviction.
If 39% is "good science" (Score:1)
Re: (Score:2)
There's no grant money available from government for that. It's only available (in huge quantities) for supporting the paradigm. Strange but true.
I rather thought that energy companies had a few extra dollars running around. Exxon's 2014 revenue was over 400 billion. Surely they could fund a few studies all by themselves.
Baseball and Psychology (Score:2)
Where batting .390 is considered a good thing.
What? - Question Solved. (Score:1)
Re: (Score:2)
Actually 39% is not bad at all. I am sure it is not better in computer science. As a reviewer I typically need to fight with the authors for them to give enough details to be able to even attempt reproduction. Most CS papers lacks basic information on :
-how the code is written (language, major data structure),
-how it is executed (complied, interpreted, which level of optimization),
-where it is executed (which machine, complete spec, operating system, idle load, is parallelisation used),
-what dataset are us
Re: (Score:2)
bitter much, aren't you? :)
For sure, there is a bunch a second grade science in there. But there is a lot of science going on in CS. I suggest you read about it instead of passing for an idiot.
Re: (Score:2)
I don't get the argument. You are saying that becasue most trained computer scientist are engineers then computer science is not a science?
Then I guess medicine is not a science either since most medical doctors are saving lifes and treating illnesses and not researching their root cause.
I also assume that mechanics is not a science since most professionals are build bridges and designing component and not solving Navier-Strokes equation.
There is a science called computer science that gets published in scie
Re: (Score:2)
well...I beg to disagree.
Computer science is not mostly discrete mathematics, it use to be true in 1980, but that was last century. Also we have "reproducibility" issues in proofs as well. Many proofs in the field are not correctly written. And it causes many of them to be incorrect. For instance there was a critical flaw in the proof of TimSort which caused problem recently.
But Computer Science is much more than that nowadays. Algorithm get tested in practice because they are proved on models of computers.
Re: (Score:2)
Okay, 39/100 is an absolute, total and complete failure in all possible regards. Legitimate scientific fields don't get recognized for being able to backup 39% of there research.
Yes, that's why [nature.com] we abandoned the pseudo-science of medicine [plos.org] ages ago. Oh, wait...
Given the little data we have, psychology is 'average'. We won't know if they're doing exceptionally well, or exceptionally poorly, until more studies are done not only on reproducibility in psychology, but in other fields as well.
Reproducibility problems aren't often investigated, and very few fields are actively studying the issue. I suspect that we'll find serious problems in virtually all branches of science as these stu
Social Sciences are hard to constrain (Score:2)
Which makes it very very hard to do research on. In addition effects may change over time, e.g. as a person's individual psychology changes as they age or as culture changes people over time. Doing research in a natural system, like the Social Sciences, is very hard. It is much harder than Physics, Chemistry, or other such Sciences. It is much harder than putting together a cutesy mobi app. And longitude studies are even harder to do well.
God bless him... (Score:2, Insightful)
I had despaired that psychology could ever pull its head out of his own ass. But if they start actually doing real science again then the field might actually be saved.
It had gotten so bad that I just assumed that the neurologists would have to deal with all this stuff from the other side. Answer the psychology questions with neurology science.
Psychology has become something of a joke lately and there is no way to fix it short of subjecting it to cold empirical science.
Re: (Score:2)
The shift to various neurosciences (e.g. neuro-economics) is inevitable, as was the shift from philosophy to philosophy of nature and then the various natural sciences. While philosophy still exists, it deals with issues nobody truly gives much of a fuck anymore.
Re: (Score:2)
Psychology can be a real science though. It just has to preform experiments and limit its theorizing to the data.
Furthermore, statistical data is vastly over used and really should be reduced radically in its application. Especially data that is mined from other studies and re-appropriated for other purposes. The mishandling of statistics is something I think we've all seen and I'm sick of it.
Re: (Score:2)
I had a friend that quit psychology because he couldn't perform experiments. A lot of it is ethics considerations.
Imagine a chemist having to worry about the ethics of the chemicals he was studying? The problem with psychology is that to do a proper experiment, you might need to break someone's mind to learn what is going on. But that is unethical and so you can't really do serious study into the human mind in most cases.
Consider medicine if they were unable to ever harm a human body and if the body couldn'
Re: (Score:2)
That was an interesting post.
Re: (Score:2)
While philosophy still exists, it deals with issues nobody truly gives much of a fuck anymore.
Like epistemology. Oh, wait. That's absolutely essential to modern science. My bad. I didn't mean to disturb your delusion.
Re: (Score:2)
If you need someone to hold your hand that is one thing. However, that isn't why I go to see a doctor.
Societal effects (Score:1)
Re: (Score:2)
I liked Soros' point that a reason for having an "open" society is that you can never be sure if the fantasy you're operating under, is actually going to do more benefit than harm, so one is always trying to remain not too sure of oneself, hence the "openness". For example, don't have a death penalty, because you can never be sure you're not making a big mistake (you might still jail the person, but basically try to be as available to correcting mistakes as possible). Try to be softer, more open, which I gu
Not just soft sciences (Score:2)
A lot of people claim the soft sciences are not 'really science' due to the intangibility of their results - and this plays directly into that bias.
However, it's very much not just the softer sciences that have this issue. There's a growing realization that it's pervasive across many hard science disciplines:
http://www.wsj.com/articles/SB... [wsj.com] : 64% of pharma trials couldn't be reproduced.
http://retractionwatch.com/201... [retractionwatch.com] - half of researchers couldn't reproduce published findings.
We're inundated with data t
Re: (Score:2)
A lot of people claim the soft sciences are not 'really science' due to the intangibility of their results - and this plays directly into that bias.
However, it's very much not just the softer sciences that have this issue. There's a growing realization that it's pervasive across many hard science disciplines:
http://www.wsj.com/articles/SB... [wsj.com] : 64% of pharma trials couldn't be reproduced.
http://retractionwatch.com/201... [retractionwatch.com] - half of researchers couldn't reproduce published findings.
We're inundated with data that, due to the specificity of the field or detail of the results, has to come from 'experts' and doesn't lend itself to a sort of common-sense vetting that we can use to filter bullshit in the usual course of our lives. Whether it's from ignorance of statistical methods, poor experimental technique, motivated mendacity (for whatever reason), or simply experimental results that represent only an unusual end of a bell-curve, there are many, many reasons that scientific data has to be taken with a serious grain of salt. It can't be assumed to be conclusive until we've reproduced it in whatever context we're trying to apply it.
With the exception of climate science. It's settled and you're a heretic if you suggest any uncertainty exists in any part of that field...
Ah psychology (Score:2)
The "soft science" in the sense that a stick of butter is a "soft structural material"
Re: (Score:2)
You ether don't know any psychology majors or are one yourself.
Of course the average nut job doesn't go to medical school, then specialize in psychiatry.
Which isn't to say that psychiatrists aren't also loons.
Re: (Score:2)
You also appear to want to group neurologists (scientists) in with psychologists (not scientists).
Beyond non-reproducibility (Score:2)
I'd suggest a study going farther than just checking reproducibility.
I bet for many studies you could produce opposite or contradictory outcomes.
That ought to get someone's PhD published.
Social science is mutable (Score:3)
When a physicist announces a new particle, nothing changes. All the particles keep behaving how they were behaving before the announcement, and they don't care how society changes. The findings should be reproducible 100 years from now.
Many other comments have correctly pointed out that studies in general often focus on the new and shiny and statistically significant rather than reproducing prior results or reporting null findings, but the issue of settling on "truth" is made that much more difficult in the social sciences due to the existence of moving targets.
The answer is here (Score:2)
If you use p=0.05 to suggest that you have made a discovery, you will be wrong at least 30% of the time. If, as is often the case, experiments are underpowered, you will be wrong most of the time.
And given the low power of most psychology experiments I am not surprised by this result.
Pure sciences vs. not pure sciences? (Score:2)
I've always thought it had something to do with this. Yes, another xkcd post:
https://xkcd.com/435/ [xkcd.com]
I can see how messy proving things are in sociology and psychology, and how absolute mathematical proofs are. It's always disturbed me how uncertain we can be with the sciences as we move to the left, though I really don't know at what point we can call something 'pure.'
Because it is a SOFT science (Score:3)
You can measure how many parts per million of some matter is in teh air.
You can measure how many bacteria of a certain type is in your blood stream.
How do you measure if someone is in a good or bad mood?
The tester's bedside (or couch-side) manners can be enough to tilt the result one way or the other.
And if the researcher has an idea of what he is looking to find, he can (even subcounsciously) manipulate the patient into reacting one way or the other, tainting the measurement.
What do we measure, how do we measure it? The subject could be lying. They subject could be be imagining something. The tester has no way to verify.
Reproducibility is NOT the problem.
Even research that was reproduced can be wrong, for same reasons as above.
The NATURE of the field is the problem, not the lack of reproduciblility.
Lack of reproducibility is merely the proof that there are fundamental problems with measurements and conclusions.
But I agree that the conclusion we can draw, is that there are a lot of false positives.
Sounds quite reasonable (Score:2)
'A lot of working scientists assume that if it's published, it's right,' he says. 'This makes it hard to dismiss that there are still a lot of false positives in the literature.'
Ummm... they do? Like, who? Not a single one I know.
If a result is published, I assume (as do most other scientists) that means very little until it's been reproduced, and even then I remain quite skeptical until it's stood the test of time. I assume many published results will turn out to be wrong. That's just the nature of science. Every paper is a work in progress, a snapshot of someone's research at one moment. And that's fine.
So 39% were successfully reproduced, and another 24% came close? I'd
Invisible Hand (Score:3)
As dismal as Psychology's record is as a science, it's still way more rigorous and evidence-based than Economics.
Re: (Score:1)
Right there. Psychology is as much a science as Astrology and Tarot cards. It's all based on what the observer thinks is normal.
Re: (Score:2)
Sounds about right.
I trust the judgement of peers before I trust the judgement of a shrink. I've heard to many things from and about shrinks to believe that they know shit from shinola.
Re: (Score:2)
I grew up a few blocks from a large mental institution, and many of my classmates' parents worked there. An awful lot of the people who worked there ended up loonier than their patients.
Re: (Score:2)
You mean they said they worked there.
Re:What They Don't Say (Score:4, Funny)
Think of it this way... psychology is the 1960-70's equivalent of today's MBA, and have many similarities:
* neither has an objective means of measuring success or failure, in spite of claiming to have a wide array of methods by which to do so.
* neither the psychologist or the MBA is held accountable for incompetence or non-criminal malice.
* sometimes either one can take on the semblance of religion, minus a deity.
* the big 'do-nothing-but-are-promised-great-riches' degree of the 60's-80's was psychology, as hordes of students took that class thinking just that. In the 90's through today it's the MBA program.
* both can stretch logic and credulity in their work to attempt things that would get an engineer either incarcerated or killed.
(...add your own here...)
(Trigger warning for the MBAs and Psych majors: this is what is known as a joke.)