Tech Firms Hire 'Red Teams.' Scientists Should, Too (wired.com) 97
The recent retraction of a research paper which claimed to find no link between police killings and the race of the victims was a story tailor-made for today's fights over cancel culture. From a report: First, the authors asked for the paper to be withdrawn, both because they'd been "careless when describing the inferences that could be made from our data" and because of how others had interpreted the work. (In particular they pointed to recent op-ed in The Wall Street Journal with the headline, "The Myth of Systemic Police Racism.") Then, after two days of predictable blowback from those decrying what they saw as left-wing censorship, the authors tried to clarify: "People were incorrectly concluding that we retracted due to either political pressure or the political views of those citing the paper," they wrote in an amended statement. No, the authors said, the real reason they retracted the paper was because it contained a serious mistake. In fact, that mistake -- a misstatement of its central finding -- had been caught soon after the paper's initial publication in the Proceedings of the National Academy of Sciences in July 2019, and was formally corrected in April of this year. At that point, the authors acknowledged their error -- sort of -- while insisting that their main conclusions held. That the eventual retraction came only after the paper became a flashpoint in the debate over race and policing in the wake of George Floyd's murder ... well, let's agree that the retraction happened.
[...] As we and others have written many times, peer review -- the way journals ask researchers to perform it, anyway -- is not designed to catch fraud. It's also vulnerable to rigging and doesn't go so well when done in haste. Editors and publishers tend to admit these problems only under duress -- i.e., when a well-publicized retraction happens -- and then hope that we believe their claims that such colossal blunders are somehow "the system is working the way it should." But their protestations only serve as an acknowledgement that the standard system doesn't work, and that we must instead rely upon the more informal sort of peer review that happens to a paper after it gets published. The internet has enabled such post-publication peer review, as it is known, to happen with more speed, on sites like PubPeer.com. In some cases, though -- as with the PNAS paper described above -- the resolution of this after-the-fact assessment comes much too late, after a mistaken claim has already made the rounds. So how might journals do things better? As Daniel Lakens, of Eindhoven University of Technology in the Netherlands, and his colleagues have argued, researchers should embrace a "Red Team challenge" approach to peer review. Just as software companies hire hackers to probe their products for potential gaps in the security, a journal might recruit a team of scientific devil's advocates: subject-matter specialists and methodologists who will look for "holes and errors in ongoing work and ... challenge dominant assumptions, with the goal of improving project quality," Lakens wrote in Nature recently. After all, he added, science is only as robust as the strongest critique it can handle.
[...] As we and others have written many times, peer review -- the way journals ask researchers to perform it, anyway -- is not designed to catch fraud. It's also vulnerable to rigging and doesn't go so well when done in haste. Editors and publishers tend to admit these problems only under duress -- i.e., when a well-publicized retraction happens -- and then hope that we believe their claims that such colossal blunders are somehow "the system is working the way it should." But their protestations only serve as an acknowledgement that the standard system doesn't work, and that we must instead rely upon the more informal sort of peer review that happens to a paper after it gets published. The internet has enabled such post-publication peer review, as it is known, to happen with more speed, on sites like PubPeer.com. In some cases, though -- as with the PNAS paper described above -- the resolution of this after-the-fact assessment comes much too late, after a mistaken claim has already made the rounds. So how might journals do things better? As Daniel Lakens, of Eindhoven University of Technology in the Netherlands, and his colleagues have argued, researchers should embrace a "Red Team challenge" approach to peer review. Just as software companies hire hackers to probe their products for potential gaps in the security, a journal might recruit a team of scientific devil's advocates: subject-matter specialists and methodologists who will look for "holes and errors in ongoing work and ... challenge dominant assumptions, with the goal of improving project quality," Lakens wrote in Nature recently. After all, he added, science is only as robust as the strongest critique it can handle.
Peer review? (Score:2)
Isn't the whole point of peer review to find flaws in methods, hypotheses, and conclusions?
Re:Peer review? (Score:5, Informative)
Yes. They've described peer review in tech-spy terms.
If you think there's a difference, try not paying your penetration testing team, requiring them to do their testing on a short deadline while also doing other work, and not giving them any credit whatsoever, and see what kind of results you get.
The problem is not peer review, it's a misunderstanding of what a journal publication is. A publication is a Slashdot post that got modded up. A couple people thought it wasn't utter shite so it passed a threshold where it now gets seen by a lot more people.
Scientific publication is supposed to distribute reports of experiments. Other scientists are then supposed to read those reports and replicate those experiments. *That's* the stage where the fraud, subtle mistakes, etc. get caught.
Re: (Score:2)
A publication is a Slashdot post that got modded up.
Incorrect. They are slashdot posts that havent gotten modded down.
Re: (Score:2)
Never been a journal editor hey?
Published articles are like browsing a Slashdot story a few hours after it's been posted. Most people browse at +something, and most of the stuff up there will stay. Occasionally something will get modded to oblivion (retraction). But if you browse at -1 you see all the dross that didn't even make it that far.
Re: (Score:2)
I think you've missed the joke.
Re: (Score:2)
It would be wonderful if this was actually what peer review is for. I'm a professional scientist, I've published and reviewed a lot of papers, I know and have worked with a lot of journal editors.
Peer review is simply about evaluating the suitability of a paper for publication in a specific magazine.
That's a simple statement that carries a lot of meaning, but peer review is first and foremost about the business of selling magazine subscriptions.
The reason we as scientists agree to peer review is that it giv
Re: (Score:2)
It would be wonderful if this was actually what peer review is for. I'm a professional scientist, I've published and reviewed a lot of papers, I know and have worked with a lot of journal editors.
It must vary by field. In high energy physics, we're brutal on our collaboration's papers long before they get to the journal. Why? Because better we beat the crap out of the paper before it gets loose than have the referees do it.
Re: (Score:2)
Re: (Score:2)
Yes but only a bit.
90% of everything is junk.
The point of peer review is to act as a first filter on the 90% of junk that crosses the journal's or conference's (for peer reviewed conferences) desk. It'll get rid of the obvious stuff, where the authors have completely misunderstood a technique, or massively over interpret experiments, or maybe have failed to account for clear systematic biases in the data, that sort of thing.
Peer review is not a stamp of approval saying "this is definitely right", it's a st
Re: (Score:2)
Not in the slightest. Peer review is not peer review, peer review is "arbitrary personal injunction". Any one reviewer can arbitrarily declare an injunction on the publication of a paper for even the shittiest reasons, and since the (very recent in fact) establishment of the peer review system the general quality of scholarship has declined precipitously to the point we routinely have things like the Sokal Hoax, Conceptual Penis hoax, and Grievance Studies whistleblowing happening.
Paid peer review then? (Score:2)
The difference seems to be that they want to get paid for peer review, and maybe drop anonymity as well. I think both sounds reasonable, but you do not really need to give a new name for it and others have been talking about it for awhile - for some wierd reason, the journals do not like the idea much (the paid peer review part). Some part of it also sounds like typical science, i.e. challenge dominant assumptions IS science as usual.
Re: (Score:1)
Re: (Score:2)
You would need to also require the reviewers to repay their money if there is a future retraction. Otherwise you just recreate the USPO problem where the faster they rubber-stamp the more money they make.
Incase you don't want to dig (Score:1)
Re: (Score:1)
Sounds like a non-retraction retraction then.
Re: (Score:2)
Damn. Somebody explained prevalence / prior probabilities to them.
To be fair, most physicians famously get that wrong too.
Cops who shoot aren't more likely to be white (Score:4, Insightful)
> Because every observation in the studyâ(TM)s data involved the use of lethal force, the study cannot possibly reveal whether white and nonwhite officers are differentially likely to shoot minority civilians.
Let me see if I can phrase this in a clear and simple way.
All of the data points involve person getting shot and that person is a minority. That's the constant. The variable is the race of the officer. So the question answered is:
Given a suspect is shot, is it disproportionately likely that the cop is white? The data says no. Putting that in more simple terms:
Cops who shoot aren't more likely to be white.
As far as I can tell, that's the statement that accurately describes the data. Anyone disagree? (Not disagreeing with the idea because CNN focused on whatever, but disagree that statement accurately reflects the data in the study).
An interesting number I saw a while ago is that about 85% of cops never have shot their guns at all, in their entire career, other than at the training range.
Re: (Score:2)
Legions of SJWs disagree because they just, "know"
Re: (Score:2)
Just like the person you answered who says:
An interesting number I saw a while ago ...
with no citation?
Re: (Score:2)
I didn't expect that fact to be controversial at all.
If you are perhaps wanting a different number, why trust my source? There are several, only a five-second Google away. Loking at the first few results in Google, the numbers range from 5% who have ever do their weapon (Washington Post) to 27% (Pew Research Center).
Pick your favorite number in that range, if you want, I guess.
You can also sanity check it yourself. If you take a few seconds to Google anything related to this, you'll undoubtedly find an an
Re: (Score:2)
So we can do a little 3rd grade artithmetic:
1,000 suspects killed by 800,000 officers That means 0.125% of officers will kill anyone this year. (99.875% will not). If a typical cop has a 25-year career, her odds of ever killing anyone are 25 X 0.125% = 3%.
To be pedantic, the correct calculation is 1 - (0.99875 to the 25th power), but the result is the same: 3%.
Re: (Score:2)
That's right, of course, and it's an instance of what I call "the hardest easy math problem in the world". I spent a few days in grade school trying to figure out how to compute that, without looking it up. There are calculations which seem intuitively to be the right way to figure it out, but all lead to absurd conclusions when you scale the numbers and down.
As you said, in this case the intuitive, simple method gives essentially the right answer, without interrupting the train of thought with a formula
Re: (Score:2)
I remember how to do these kinds of problems by reversing the question: "What are the chances of not shooting someone in the first year?" The answer, as you posted, is 0.99875. Thus, the chances of not shooting someone in 25 years is 0.99875 times itself 25 times, and the reverse of that, the chance of shooting someone in 25 years, is 1 minus that.
Re: (Score:2)
Yeah that's what makes it very hard, yet easy.
You pretty much CAN'T figure out the chances of ___ any ___. (A or B or C). As long you try to compute it in that direction, you're headed for frustration.
The moment you realize you can calculate the odds of not A and not B and not C, then just subtract from 1, you're golden. The way I present it for fun, driving smart people mad, is:
If you enter ten contests and you have a 1/100 chance of winning each contest, what is your total chancs of winning a prize?
It's
Re: (Score:2)
Another example is the Birthday Problem.
Re: (Score:2)
Yeah the birthday problem (important in my field of security) is this squared* - any can match any, rather than one matching any.
* Not mathematically squared. :)
Re: (Score:2)
This article about a study based on a survey says 73% hadn't (yet) fired their weapon on duty.
https://www.pewresearch.org/fa... [pewresearch.org]
Re: (Score:2)
That's a good source thanks.
Taking the numbers used by anti-police activists and doing a little simple arithmetic, we can see the odds of an average cop ever killing anyone must be less than 3%:
https://slashdot.org/comments.... [slashdot.org]
Re: (Score:2)
Let me rephrase that. Assuming that the numbers cited by anti-police activists are true, it must be that less than 3% of cops ever kill anyone.
Re: (Score:2)
So the question is: is that a large number? Unequivocally yes. We have an average number of police officers, but almost four times [wikipedia.org] as many police killings as the next highest first-world country.
Re: (Score:2)
If it were evenly distributed, if Omaha was as dangerous as Detroit, the number would be 0.030785. That's The maximum possible number of officers. (If you multiplied, that's a shortcut that always over-estimates, but it's close for small percentages).
> assuming there aren't a bunch of kill-happy officers skewing the curve.
Detroit IS more dangerous than Omaha. Chicago is more dangerous than Dallas. Washington DC is more dangerous than Plano.(1) The curve is very skewed.
Lod Angeles SWAT officers have to
Re: (Score:2)
I don't know
Re: (Score:2)
I don't know what you think you can infer from that map, but here's the data. I'm sure you know how to use Excel.
https://ucr.fbi.gov/crime-in-t... [fbi.gov]
Re: (Score:2)
Re: (Score:2)
You may have been redirected to a more general page, but if you click the categories, near the top left you'll see "download Excel".
Re: (Score:2)
Alternatively we can do it by state: average of state murder rates for red states is 5.1. Average of state murder rates for blue states is 3.7. No surprise that it'd be a little lower than averaging over the whole population, but it doesn't change anything really.
In other words: your claim is almost sort of accurate, you just mixed up red and blu
Re: (Score:2)
I got 47% higher in blue states, which matches up what I've seen from others who did the calculation. [Shrug]
Re: (Score:2)
Re: (Score:2)
> That's not a shrug, that's a huge difference. One of us is drastically incorrect.
https://xkcd.com/386/ [xkcd.com]
Thanks for sharing the link. I might look at it later on my computer if I'm bored.
Re: (Score:2)
Btw if it's really critical for you to know if I was wrong this last week and if you were wrong, here's the answer:
Yes.
Yep, I was wrong a few times last week. Mostly I was wrong when I was a dick to a vendor - with one of our HR people cc'ed on thread. That was a dumb move. I also discovered this week that I was wrong 120 days ago when I thought excepting the test user from the single-use policy wouldn't have any side effects, in a 2FA system I designed and coded. 120 days later, that took down the VPN.
Re: (Score:2)
Feel better knowing that? :)
Not at all, arguing on the internet is not always without purpose. There's a lot of speculation about why so much fake news is targeted at right-wingers, and I certainly don't know which reasons are correct, but there was a story a while back about some eastern European fake-news pushers and why they targeted the right and not the left. The reason the guy in the story gave was simply that when he published something targeted at the left, someone would quickly debunk it in the comments and the story would di
Ps - of course you aren't :) (Score:2)
> ) I'm not going to look up numbers for the small number of people on specific response teams.
Yeah your argument requires that being a campus cop at BYU is just as dangerous as being on the Chicago SWAT team. Of course you don't want to see the actual data. Lol
Re: (Score:2)
"...not sufficient to support claims about differential officer behavior without knowing how many times officers encountered racial minorities to begin with."
It also seems to ignore differential behavior by the person shot, i.e. is an encounter more likely to become violent when the suspect is a different race than the officer? I suspect so, but don't have any data to back it up.
Re: Incase you don't want to dig (Score:2)
Thanks!
Welcome to 2003 (Score:3)
Michael Crichton, at Caltech, January 17, 2003.
From Aliens Cause Global Warming [s8int.com]
Re: (Score:2)
Crichton's got his problems but that's a pretty damn good idea. Do all research in duplicate with an A team and B team who don't have any contact or knowledge of each other, or at minimum assign a red team to try and replicate or disprove everything.
Social science peer review is a joke (Score:1)
I'm a bit salty since I spent all morning peer reviewing a paper. I don't know why; I get no recognition, credit, or pay for it. I do it mostly out of duty to try to help people get better results.
But this approach won't work. The main reason is that data are proprietary. I have to trust that they did their analyses correctly. I could "request the data from the author", and then spend days trying to decipher their coding and then try to replicate their analyses. That is if they give the data. In many cases,
Native Americans (Score:2, Offtopic)
It's interesting how one cannot say "white" and "black" list, but it's still ok to bring in the "red" team.
Don't play perfect as the enemy of good (Score:2)
Peer review does the job it's intended to do... most of the time. But there's certainly always room for improvement, as with most processes.
The only issue I see with this proposal is - is this really practical? While it's been a number of years, I did work in research for some time. Depending on the area of study, it can be hard enough to garner enough willing and subject-matter-competent peer reviewers under the current system - I don't see where these additional "red team" members are going to come form.
That's not a flaw in the study, it's a parameter! (Score:2)
I suspect this would lead, not necessarily to improvement of the experiments themselves, but improvement of the descriptions thereof. I.e. rather than fixing the bug, document it so it's expected behavior.
Which is still an improvement!
there is no cancel culture (Score:1)
Nobody has actually been canceled. We only have a crybaby culture that panics whenever someone calls them out for their bullshit. The heat goes away when we move on to the next spectacle.
The "canceled" gets a new job, maybe deleted their Twitter account, and learns the less they complain the less affected they are by the quickly evaporating "social justice".
Re: (Score:2)
Urban Myth (Score:2)
These are peoples lives you are talking about. Loosing a job when you have a family to feed is everything.
Louis CK is still working, despite the full weight of the media against him. He suffered less than a local dentists does with a few bad Yelp reviewed. This cancelling is not real, it doesn't really happen. Or rather it never has the intended or proportional consequence.
People lose their jobs over this when they're small like you or me, usually for arguing with their boss or HR department. With a little luck and a marketable skill, they also have a pretty good chance of getting a new job. Us little people ar
Re: (Score:1)
Re: (Score:2)
Powered by GOYA Beans. Which thankfully have not been cancelled.
Re: Urban Myth (Score:2)
Are you trying to gas light everyone or do you really actually believe all that crap?
Why do you think it's ok for someone to get fired because they didn't wear a mask at the mall or said "all lives matter" on their private twitter account?
Do you think kids should report their parents to their authorities for wrong think, too?
You can't be canceled, no matter how mean you are (Score:2)
Saying ” All Lives Matter” Can Get You Fired From Your Job or Killed !! [thejewishvoice.com]
Except Dr Leslie Neal-Boylan still works UW Oshkosh (since March 2015). This is despite being "fired" in June 2020 from University of Massachusetts Lowell.
Most(all?) of the bullshit you read about how people are being affected by cancel culture:
* didn't happen
* applies to high-profile public-relations related job roles
I've looked pretty hard and I can't find the supposed unemployed (potentially homeless?) people who were "can
Re: You can't be canceled, no matter how mean you (Score:2)
Did you skip the Cisco article on the front page where people got fired for saying exactly that?
Why is this ok?
Is there a guide book of forbidden words and phrases that will get you fired? There's definitely a hidden list of phrases.
Maybe like the Hispanic guy at SDGE who got fired for playing with his fingers allegedly making the "OK" sign as he drove down the street because 4chan trolled the left into thinking that was a symbol of white supremacy?
Are you going to tell that guy it's no big deal he lost hi
Re: (Score:2)
Did you skip the Cisco article on the front page where people got fired for saying exactly that?
I didn't think it was relevant to this thread [bloomberg.com]. But you think it is relevant?
Why is this ok?
Political correctness has been, for better or worse, made a part of modern business culture. To the degree that it is often part of an employee handbook that HR requires employees to sign. Fail to follow your employer's rules in a right to work state, you don't get severance or warning or really any appeal when then happens. You're just handed your last paycheck and shown the door.
"Cancel culture", in all it's poorly defined flavors
Re: (Score:2)
Re: (Score:1)
Re: (Score:2)
Why do you think it's ok for someone to get fired because they didn't wear a mask at the mall
We're in the middle of a pandemic. Wear a fucking mask. Stop acting like an antisocial asshole and putting others at risk.
Re: (Score:2)
The social justice movement is pretty much one of the single most exclusively rich and white groups in the entire US, you may as well be speaking greek when you try to get concepts like running out of money or not having a trust fund to fall back on to them.
Re: (Score:2)
LOL. The way people cry about it they must be taking that meme literally.
Re: (Score:2)
This weakness is why they engage in the so-called "cancel culture".
The right-wing media controls the narrative here.
No cancelling by the right wing. Nope. Not them. "Let's get back to the mid 90s where we could cancel gays!" :-/
Re: (Score:2)
You'd have done really well in the Red Guard
Re: (Score:2)
Dude, it's a lot older than communism. For thousands of years, anyone gay got canceled from their job, if not jailed or killed. Don't like Jesus? G'bye.
The problem is the "live and let live" lesson was not learned. Only the vile, vile master has changed.
Re: (Score:1)
Re: (Score:2)
Calling it a crybaby culture, is this the kind of bullshit your talking about?
Seems rather like everyone has got something to complain about including yourself.
What's a "Red Team?" (Score:2)
Not defined anywhere in OP, and not everyone here inhabits the same exact habitat.
Re: (Score:2)
It's a team made of up Native Americans. The Blue Team is made up of people suffering from hypothermia.
Re: What's a "Red Team?" (Score:2)
No, the blue team are usually Smurfs
Re: (Score:2)
It's just another trigger word waiting to be used by white social warriors to fake their empathy of the victims.
Re: (Score:1)
A red team is a group that takes an opposing view when critiquing. Too often peer review is an echo chamber. It is often little more than a proof-read. Raw data isn’t reanalyzed. Red teams bring diversity of thought to ideas and often help sharpen the ideas.
Re: (Score:1)
Floyd was allegedly murdered. A finding in a court of law is needed to determine if he was actually murdered.
With reports of the body cam footage leaking out, it's looking more and more like a drug overdose.
Here's what a medical doctor has to say about it - https://www.youtube.com/watch?... [youtube.com]
He goes over all of this stuff.
That's why I don't bother with journals (Score:2)
Suppose, there is "systemic racism"... (Score:1)
Ok, suppose such systemic racism really exists — why is it blamed on Republicans in general and Trump in particular?
George Floyd died in a city ruled by a Democratic mayor, in a Democrat-governed State — why were the protesters attacking the White House 5 days later [cnn.com]?
An attack so strong, that — with Washington's Democrat-commanded police useless — Secret Service saw fit to move their charges into safe bunker?
It was even more ridiculous in Baltimore, which rioted in 2015 over the death
Re: Suppose, there is "systemic racism"... (Score:2)
Because Orange Man Bad. Systemic racism. Historical injustice and oppression. And kill whitey.
You had to ask?
Re: (Score:1)
In other words black activists aren't smart enough to realize they're helping their slave masters perpetuate their own problem.
White people have been duped into believing bullshit from black activists sometimes to the point of hating themselves. With they'd just off themself instead of trying to kill other white people.
So Orange man good. Yank all support from BLM, antifa and democrats.
Clearly Very Uninformed Advice (Score:2)
The peer reviewers assigned by a journal are tasked with being the devil's advocate.. And they do tend to be brutal. Of course, they cannot catch everything but that's why we have other studies replicate them. At the very heart of what science is, is try and survive intentional attempts at falsifying (disproving them in any way possible). Only one successful falsification renders a hypothesis or theory invalid. No number of failed falsifications proves a hypothesis/theory true.. only increases our conf
Paying for Red Teams is a bad idea (Score:2)
The budget for science is not that big and is shrinking. It's much more efficient to use that budget to pay people to do science, than it is to pay them to be on a "Red Team." Even if they do it for free, their time is better spent elsewhere.
Save the "Red Team" situations where the stakes are life-or-death.
Re: (Score:1)
I'm a scientist.
I think you must be looking at different budgets than the rest of us.
There's a ton of noise (i.e. news) about science funding, but the actual data shows that it is exceedingly rare for the funding to actually decrease.
The value to the rest of society of that science spending does decrease over time (something called "research efficiency"). Maybe we should worry more about the fact that we're getting worse at science in an absolute sense and in an output/dollar sense and less about absolute b
Last time I checked (Score:2)
Re: (Score:2)
Yea, that long time ago when facts mattered, and principles like "innocent and proven guilty". In today's society you are guilty as soon as you are accused, and facts are just hindrances to the narrative of whoever happens to be the the loudest vocal minority at the time. Who needs facts, it's the loudest voice that matters, right?
Re: (Score:2)
Yea, that long time ago when facts and principles like "innocent until proven guilty" actually mattered. In today's society you are guilty as soon as you are accused, and facts are just hindrances to the narrative of whoever happens to be the the loudest vocal minority at the time. Who needs facts, it's the loudest voice that matters, right?
Journals need to be policed (Score:2)
It would be simpler if journals employed a commissar to make sure only the right things are published.
Re: (Score:1)
and the correlation stops (Score:1)
just
"black"
right?
a mockery