Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
Science

Psychology's Replication Battle 172

Posted by Soulskill from the study-of-american-undergrads dept.
An anonymous reader sends this excerpt from Slate: Psychologists are up in arms over, of all things, the editorial process that led to the recent publication of a special issue of the journal Social Psychology. This may seem like a classic case of ivory tower navel gazing, but its impact extends far beyond academia. ... Those who oppose funding for behavioral science make a fundamental mistake: They assume that valuable science is limited to the "hard sciences." Social science can be just as valuable, but it's difficult to demonstrate that an experiment is valuable when you can't even demonstrate that it's replicable. ...Given the stakes involved and its centrality to the scientific method, it may seem perplexing that replication is the exception rather than the rule. The reasons why are varied, but most come down to the perverse incentives driving research. Scientific journals typically view "positive" findings that announce a novel relationship or support a theoretical claim as more interesting than "negative" findings that say that things are unrelated or that a theory is not supported. The more surprising the positive finding, the better, even though surprising findings are statistically less likely to be accurate."
This discussion has been archived. No new comments can be posted.

Psychology's Replication Battle More

Psychology's Replication Battle

Comments Filter:
  • good luck with that.

    • Re:Freud's problem too (Score:5, Insightful)

      by Anonymous Coward on Sunday August 03, 2014 @05:16AM (#47592665)

      When psychologists stop producing so many studies with obvious bias, subjective terminology, subjective conclusions, and stop arbitrarily coming to conclusions based on data flawed for those reasons, maybe it could be taken seriously. Obviously, replication is needed, too.

      But so many people are fooled by it. Want a study that says video games cause people to be aggressive? There's a psychology study for you, but there's also one for your opponents. And all of them are bad science.

      • Re:Freud's problem too (Score:5, Insightful)

        by sjwt ( 161428 ) on Sunday August 03, 2014 @06:06AM (#47592761)

        Yup, like the recent one about men not being able to 'be alone with their own thouhgs' [washingtonpost.com]..

        That same data can also read 'Men, more willing to put up with pain' or 'Men, more curious and want to know what they may experience'

        • Re: (Score:1, Offtopic)

          by AthanasiusKircher ( 1333179 )

          Yup, like the recent one about men not being able to 'be alone with their own thouhgs' [washingtonpost.com]..

          Yeah, note it was already discussed here too [slashdot.org].

          That same data can also read 'Men, more willing to put up with pain' or 'Men, more curious and want to know what they may experience'

          Perhaps the one thing more common than flawed social science experiments is Slashdot commenters who think they can find flaws but haven't actually read the paper or thought about it.

          This "same data" really CAN'T be "read" that way: the researchers specifically asked the subjects to experience the shock FIRST (so we can't assume they were just curious). And that stage of the study specifically excluded those who weren't seriously offended by the shock (they only

          • Of course, that selection bias could also be read as those who are willing to pay to avoid something unpleasant have less patience than those who are not. Or that those who have a lower tolerance for pain are also less likely to value quiet and solitude - e.g. they are more likely to be extroverted - than those who have a higher tolerance. The data is clear: based on the chosen subset of the male population, there is a correlation between the subset and the dislike of and/or inability to endure solitude. Th

  • Easy to measure versus important (Score:4, Insightful)

    by Tablizer ( 95088 ) on Sunday August 03, 2014 @04:33AM (#47592585) Journal

    Psychologists are up in arms

    Perhaps they need some therapy :-)

    a fundamental mistake: They assume that valuable science is limited to the "hard sciences."

    Software engineering has a similar problem. Things that are objective to measure, such as code volume (lines of code) are often only part of the picture. The psychology of developers (perception, etc.), especially during maintenance, plays a big role, but is difficult and expensive to objectively measure.

    Thus, arguments break out about whether to focus on parsimony or on "grokkability". Some will also argue that if your developers can't read parsimony-friendly code, they should be fired and replaced with those who can. This gets into tricky staffing issues as sometimes a developer is valued for their people skills or domain (industry) knowledge even if they are not so adept at "clever" code.

    Thus, the "my code style can beat up your style" fights involve both easy-to-measure "solid" metrics and very difficult-to-measure factors about staffing, side knowledge, people skills, corporate politics, economics, etc.

    • Completely different situation. In programming discussions are how to optimise the processes involved, the problem with psychology its that they aren't sure if they're working on computers or breakfast cereal boxes with a few rectangles drawn on them. The main value that psychologist bring to the table today is to fulfill the role of that good friend who isn't afraid to lay out a few home truths. Of course if you already have such a friend, the need to attend a psychologist is naturally obviated...

      So, I'm j [arachnoid.com]

      • Re: (Score:2)

        by tomhath ( 637240 )

        Completely different situation. In programming discussions are how to optimise the processes involved

        But the discussion is based on nonsense. "Separation of Concerns" is a Good Thing, right? Who says? Gang of Four patterns are the proper approach, right? Why?

      • The main value that psychologist bring to the table today is to fulfill the role of that good friend who isn't afraid to lay out a few home truths.

        Actually, I don't think so. In my limited experience, there seems a large methodological bias towards non directed therapy. Let the patient talk. Maybe ask leading questions, but no trace of "lay out a few home truths", at least about the patient himself.

        • Re: (Score:2)

          by HiThere ( 15173 )

          As you said, "limited experience". That is one (or a few related) schools of psychology. Others, are much more directional.

          OTOH, nobody goes around "laying out a few home truths", because that is counter-productive. (Some psychologists don't seem to do better than random, but they all avoid known bad choices...like "laying out a few home truths".)

    • the fascinating thing to me is that sometimes programmers with drastically different coding styles (say, a Lisp macro/functional style compared to an object-oriented small-objects-everywhere style), who would argue vehemently about how the other side is wrong, can still both write incredibly good code. That is, the code will get the job done, be readable, and be flexible.

      Because drastically different styles can end up with good code, I see that as a sign that we as programmers haven't figured out the elem
      • There's a basic foundation that's roughly agreed upon, delineated by rules and best practices. Once those are mastered, then coding becomes an art form. And as art it can be subjective, defy description and all apparent rules of logic, and yet work incredibly well. If there's one thing I've learned on /. over the years from reading all the arguments between coders, it is that there is more than one way to become a master of one's craft (where coding is considered) and that coding becomes Art. Which I person

        • There's a basic foundation that's roughly agreed upon, delineated by rules and best practices.

          What rules and best practices are those? As far as I can tell, everyone has their own set.

          • There's a very basic level of hygienic measures that are are taught to first graders and nobody disagrees with. Things like don't overuse global variables, don't build one-mile-long procedures, avoid spaghetti code by banning goto, declare the type of your parameters in C.

            For other rules of style, yes, every house has their own rulebook.

            • Well, even those are rather vague, LISP programmers like global variables (and everyone else calls global variables 'singletons'), Linux programmers will flame you for trying to ban goto. Some people really like mile long procedures, although I confess I don't understand those people.

  • because it's wishful thinking (Score:1)

    by Anonymous Coward

    Scientific journals typically view "positive" findings that announce a novel relationship or support a theoretical claim as more interesting than "negative" findings that say that things are unrelated or that a theory is not supported. The more surprising the positive finding, the better, even though surprising findings are statistically less likely to be accurate.

    Because it's always wishful thinking and the 'findings' are always BS. About time it's called out for the non-science nonsense that it is.

  • "less likely to be accurate" (Score:4, Funny)

    by Vinegar Joe ( 998110 ) on Sunday August 03, 2014 @04:44AM (#47592599)

    That's a surprise.

  • WTF? (Score:5, Insightful)

    by Oidhche ( 1244906 ) on Sunday August 03, 2014 @04:51AM (#47592619)

    it's difficult to demonstrate that an experiment is valuable when you can't even demonstrate that it's replicable

    Duh. That's because an experiment that is not replicable has *no* value.

    • Re: (Score:3, Interesting)

      by thesandtiger ( 819476 )

      There's different levels of replication.

      In physics, you can generally replicate an experiment vary precisely if you've got a handle on the factors that went into that experiment - control the environment, etc. You can have an almost perfect replication. Yay, science!

      In social psychology research you can't ever even approach that same level of control over the environment the experiment takes place in. The subject will be different - even if it's the same subject used in the first experiment, because people

      • In social psychology research you can't ever even approach that same level of control over the environment the experiment takes place in.

        Exactly. This alone negates any claim as to whatever result is "found". That basic replication you refer to later is like saying "I used copper wiring in all my experiments" and never giving the length or gauge.

        a lack of easy replication under mostly similar circumstances indicates that that factor probably isn't as strong as hypothesized, and it cuts o

        • Actually, hell let me throw a challenge at you:

          Please explain the value in trying to understand gravity in a way that is general enough to also apply to numerous other fields that are deemed to "have value" but that excludes trying to understand human behavior.

          If you can do so in a way that is meaningful and isn't intellectually dishonest I'll be surprised.

    • Re: (Score:2)

      by fermion ( 181285 )
      We also have to look at how repeatability works. One reads a paper, does one best to follow the work, perhaps calls one of the researchers to get clarification, combine this with known methods, and at the end of the day maybe get a similar result. If, as in the case of cold fusion, the result is not similar, then there is at least some carelessness if not fraud in the original result. Which is fine because it is just one result, and no one should thinks one result is conclusive.

      In social sciences repro

  • Not Just Psychology (Score:4, Insightful)

    by jamesl ( 106902 ) on Sunday August 03, 2014 @05:13AM (#47592655)

    The reasons why are varied, but most come down to the perverse incentives driving research. Scientific journals typically view "positive" findings that announce a novel relationship or support a theoretical claim as more interesting than "negative" findings ...

    This applies to all science, not just psychology.

  • Once is an anomaly
    Twice is a coincidence
    Three times is a pattern

    • "Three times is a pattern"

      I sampled a random bit sequence just the other day. I can now assure you that a random bit stream is all ones! all friggin' ones I tell you!

      • As Goldfinger was written in 1959 that quote is a paraphrase of a much older saying [barrypopik.com].

        It’s unclear that the saying’s origin is from Chicago; Fleming was probably thinking of Chicago’s gangster years of the 1920s-1930s. “Once is an accident, twice is a coincidence, three times is a habit” has been cited in print since at least 1921. “Once is nothing, twice is coincidence, three times is a moral certainty” has been cited in print since 1923.

  • So, it's not a science, it's a religion (Score:3, Insightful)

    by Anonymous Coward on Sunday August 03, 2014 @05:17AM (#47592667)

    Falling into the 'cult' category

  • I think that too many "studies" set out to prove a hypothesis instead of test a hypothesis. The drive to prove something puts bias into the study and skews the outcome. No one wants to be proven wrong. This is especially important when the measurements are subjective as in many psychology studies.

    • The other problem is sample size. Psychology sample sizes are *way* too small. In a world of 8 billion people today, anything you find out in a psychological experiment that involves at most a few hundred subjects, often less, cannot have anything universal to say. The samples are just too small.

      Here's an analogy. You plant a dozen tulips in your garden, and observe how well they grow when you do X. Now you claim all plants will grow like that when you do X. The claim is way too broad. Even if you had a d

      • The other problem is sample size. Psychology sample sizes are *way* too small. In a world of 8 billion people today, anything you find out in a psychological experiment that involves at most a few hundred subjects, often less, cannot have anything universal to say. The samples are just too small.

        Sample size is independent of population size, and sample sizes far less than "a few hundred" can be significant. In the social sciences, errors are far more likely to be caused by sample bias than size. Most psychology experiments conducted on people use university undergraduates as subjects, which are more likely to be politically liberal, altruistic, trusting of others, etc. Increasing the sample size isn't going to fix that.

        • On the contrary, increasing the sample size to big data sizes of say 2 billion subjects would definitely fix that bias problem. [of course this is unrealistic].

          • Re: (Score:3)

            by khallow ( 566160 )

            On the contrary, increasing the sample size to big data sizes of say 2 billion subjects would definitely fix that bias problem.

            Not at all. For example, try extrapolating behavior from 2 billion young men to older women. You can have huge sample sizes and yet still have sample bias simply because you've excluded an important category (such as the people you actually wanted to study).

            • Re: (Score:2)

              by mpe ( 36238 )
              For example, try extrapolating behavior from 2 billion young men to older women. You can have huge sample sizes and yet still have sample bias simply because you've excluded an important category (such as the people you actually wanted to study).

              Even if you try hard for a "representative sample" you can still have a problem where you lack a "box" to "tick" for something which turns out to be important.

        • Let alone the cultural environment. Behavioral psychology often attempts to extrapolate its findings on the whole Earth population, without taking into account that the cultural background of its subjects is (virtually) identical for each subject. The cultural background _most definitely_ influences behavior. Do the same study on Western Europeans, Arabs and Japanese, and you'll likely get huge differences per group.

      • Fixed typo.

        Agreed on study size, which is why social scientists look at meta-studies of hundreds of studies performed over as much as a decade, to eliminate the noise and other transient junk.

        What they really need to do, though, is examine more hypotheses. You need 7-10 additional hypotheses, not including the null hypothesis, that are orthogonal to each other and to the hypothesis being tested. This would allow you to binary subdivide the problem space, not only showing what something isn't but also showin

    • my psychological disorder compels me to point out the misspelling of "premise"

    • Re: (Score:2)

      by mpe ( 36238 )
      I think that too many "studies" set out to prove a hypothesis instead of test a hypothesis. The drive to prove something puts bias into the study and skews the outcome. No one wants to be proven wrong. This is especially important when the measurements are subjective as in many psychology studies.

      But hardly confined to "psychology". Possibly even not confined to "soft" sciences. Since attempts at falsification can easily turn out to be very politically incorrect.

  • "Social science can be just as valuable" (Score:2, Insightful)

    by Anonymous Coward

    No, and it shouldn't carry the same "science" label to start with. Make it "social studies" or whatever. To call it science, one tries to put it on the same level as real science, where the processes are completely different on numerous levels. It's an insult to real science. For example, when a scientist builds a collider to find a particle, and he finds one, he puts up the results so they can be verified by peers, and if the collective brainpower finds an error and puts it down, the process is considered

    • Re: (Score:1)

      by Anonymous Coward

      I once read a study that claimed that porn makes people have a callous attitude towards women. To 'prove' this, they asked college students how long rapists should be sent to prison. Then, they showed those students some porn videos. Afterwards, they asked the same question, and some of them supported reduced sentences for rapists. The arbitrary, subjective conclusion they came to in the face of the subjective data they gathered using biased methods was that porn makes people callous towards women. If you w

      • Re: (Score:2)

        by mpe ( 36238 )
        I once read a study that claimed that porn makes people have a callous attitude towards women. To 'prove' this, they asked college students how long rapists should be sent to prison. Then, they showed those students some porn videos. Afterwards, they asked the same question, and some of them supported reduced sentences for rapists. The arbitrary, subjective conclusion they came to in the face of the subjective data they gathered using biased methods was that porn makes people callous towards women.

        Did the

    • As I read it the initial flap was over the people and journal involved in the replication lying to get the cooperation of the original researcher.

      They promised to give an opportunity to review and publish a comment on their own results. They secured her cooperation, getting detailed descriptions of the methodology - far beyond what was in the publication - copies of the original film, and the like. Then, when they got differing results, they denied her the percieved-as-promised opportunity to examine thei

  • Who writes this crap (Score:5, Insightful)

    by awol ( 98751 ) on Sunday August 03, 2014 @05:49AM (#47592705) Journal

    "Those who oppose funding for behavioral science make a fundamental mistake: They assume that valuable science is limited to the "hard sciences." Social science can be just as valuable, but it's difficult to demonstrate that an experiment is valuable when you can't even demonstrate that it's replicable."

    No, those of us that oppose the funding of this crap recognise that if you cannot replicate your "study" then it is not an experiment. If what you are doing cannot be proved (one way or the other) by experiment then IT IS NOT SCIENCE. I don't really care what it gets called and some of it may even be valuable for some values of valuable however the amount of dross that is produce by social researchers that try and call themselves scientists is truly extraordinary and a plague on our world.

    • Yes, but it is still useful to achieve positive outcomes. There are a lot of people in society with mistaken ideas and if science like this can be used to push their repugnant ideas out of the mainstream then all the better. We all need to support these scientists and not take such a narrow view.

      • Re:Who writes this crap (Score:5, Insightful)

        by Intrepid imaginaut ( 1970940 ) on Sunday August 03, 2014 @07:22AM (#47592959)

        The above comment is precisely why these "social sciences" need to be delegitimised and rubber-roomed until they can figure out the meaning of the phrase "scientific method". Grant them no authority in deciding government policy, massively defund them in academia, get them out of the courtrooms, and generally pillory them for the witchdoctors they are.

        If you have to ask why, you're part of the problem.

        • Re: (Score:2, Insightful)

          by Anonymous Coward

          Here's my challenge to individuals such as yourself who denigrate psychological science:

          How would *you* study behavior?

          It's very easy to dismiss behavioral sciences when you're not trying to study behavior. It's a very complex, difficult topic. E.g., how do you define depression? How do you define psychosis? How do you determine whether or not early childhood interventions actually have an effect on adult outcomes?

          Maybe you would argue that behavior shouldn't be approached scientifically, but that's a cop-o

          • You can start here -> http://www.arachnoid.com/psych... [arachnoid.com]

          • How would *you* study behavior?

            The absence of a good way to study behavior does not make psychology good science. If you don't have a good way to do so, then you don't have a good way to do so; the end.

            I'm sick of ignorant arm-chair narcissists denigrating psychology when they don't have the balls to admit they have no clue how to approach the subject because it's too hard for them to understand.

            And I'm sick of people who come up with absolutely illogical defenses of the indefensible.

      • This from a guy whose primary interests are networking and BDSM.*

        * See his Slashdot alias, then laugh. It's funny.

        • This from a guy whose primary interests are networking and BDSM.*

          Must be mixed carefully. It's OK to use network cables for bondage, just don't put them back in the network afterwards.

      • There are a lot of people in society with mistaken ideas and if science like this can be used to push their repugnant ideas out of the mainstream then all the better.

        If their ideas are truly repugnant, then science can do the job of showing why they are mistaken. You don't need to use fake science. Fake science used the way you describe is more succinctly known as lying.

  • Replicating scientific results (or failing to) is a good thing.

    Being rude about it, as was apparently the case here, is plain old asshattery.

    • Re: (Score:3)

      by awol ( 98751 )

      No the asshat is not saying that if you cannot get the same results it's not science (in fact the exact opposite), but rather that if you cannot demonstrate that the experiment itself is replicable then it is not science. The contention in the article that in social sciences this lack of replication of experiment may just be a reality up with which we must put IS the reason why whatever you want to call it, it is not science.

  • ... then it ain't science. End of story.

    • Define 'replicate'.

      • To replicate an experiment, you take the description of the conditions, tasks, environment, fixed independent and dependent variables, analytical method and results provided by the original experimenter in the (peer-reviewed) paper they published.
        If you can show the same results, with the same statistical significance, then it's reasonable to assume that the experiment shows a valid scientific phenomenon.

        If you can't then one of the two experiments got it wrong and more work is needed.

        The basic proble

        • Re: (Score:2)

          by mpe ( 36238 )
          To replicate an experiment, you take the description of the conditions, tasks, environment, fixed independent and dependent variables, analytical method and results provided by the original experimenter in the (peer-reviewed) paper they published. If you can show the same results, with the same statistical significance, then it's reasonable to assume that the experiment shows a valid scientific phenomenon.
          If you can't then one of the two experiments got it wrong and more work is needed.

          Actually it woul

  • Simple solution (Score:3)

    by jd ( 1658 ) <imipakNO@SPAMyahoo.com> on Sunday August 03, 2014 @08:37AM (#47593195) Homepage Journal

    Have a journal, call it Debunker's Weekly if you want, that is divided evenly between papers on replication and papers showing negative correlation at the start. Pay authors a nominal amount, according to the thoroughness of the work as judged by referees. Provide the journal free to University libraries. Submit summaries of major stories to Slashdot, The Guardian, various Skeptical societies and other places likely to raise the extreme ire of dodgy researchers. In fact, the more ire, the better.

    The journal doesn't have to last long. Just long enough to force bad researchers to improve or quit, force regular journals to publish a wider range of findings to avoid humiliation, and to correct dangerously erroneous beliefs. Since there must be a stockpile of unpublished papers of this sort, you should probably be able to get six or seven bumper editions out before anyone notices the dates, and maybe another two before the journal is sued into oblivion for defamation.

    That would be plenty to make some major course corrections and to "out" a few frauds.

    • Let's review:
      "Pay authors" ... "Provide journal free ... "

      The journal doesn't have to last long

      Don't worry, it won't. I'd reckon on one edition.

      Of course, what this whole field of study needs is a rich uncle (or sugar daddy) to provide funding for specific, basic, pieces of research. You'd think that for all the money they've made from social media, some of the FB/Twitter/others founders or major beneficiaries could put their hands in their pocket.

      Or maybe they are the *last* people who want to make this subject rigourous and scientific?

      • Re: (Score:2)

        by jd ( 1658 )

        It needs to be funded the same way as the British BBC, by license fee, or the same was as for public utilities, tax.

        It needs to have a charter guaranteeing payment in advance for the requested service and guaranteeing immunity for any actions provided within the terms of the charter. (If it's not chartered, you'll have every drug company and its brother suing you for publishing the suppressed papers Ben Goldacre keeps talking about.)

        If it's not free, it won't have readers. Negative results aren't as desirab

  • If you think Psychology has a replication problem, get a load of Economics.

    When it comes to "hard" sciences, Economics is basically remote viewing with a political agenda.

    • Nonsense, economics has quantities and flows that can be measured. Psychology has none of that.

      • It's not the data that's the problem with Economics, it's the postulates that are formed from whole cloth, and "laws" that are similarly . In fact, even the data in a lot of Economics is just hokum, based upon opinions more than anything measurable. MMT is a good example.

        I would suggest that Psychology's reputation has caused researchers to become a lot more rigorous. The opposite has happened in Economics.

        • MMT is "hokkum" because you say so? Some economists find it useful model, others don't. That proves exactly nothing about economics having hard data to deal with, whereas psychology has nothing.

  • Without replication, science doesn't build on previous results. It just thrashes around. Psychology (and theology) are like that. They change, but don't improve much.

    There's a practical problem. Without repeatable scientific results, a technology cannot be built based on the science. "Science is prediction, not explanation." - Fred Hoyle.

  • The problem with soft science experiments (Score:4, Interesting)

    by PPalmgren ( 1009823 ) on Monday August 04, 2014 @07:54AM (#47598805)

    There are plenty of good psychology experiments/case studies that produce a lot of really useful information and are repeatable (albeit over a very long period of time). The problem is there are also a lot of complete and utter ass psychology experiments. It is really really hard to produce a good study that provides useful results in soft sciences, and in cases of psychology, they take a very long time and sometimes a lot of money to complete. Yes, they have to account for a lot of variables and exclude them via statistical analysis, but the ones that do it right do it exceptionally well.

    I used to think negatively on those types of studies until I actually took the time to read one while helping my girlfriend with a paper. I was amazed at the level of detail and the amount of effort they took to isolate the results into meaningful data.

Slashdot Top Deals

Every nonzero finite dimensional inner product space has an orthonormal basis. It makes sense, when you don't think about it.

Close