Statistical Accuracy of Internet Weather Forecasts

markmcb writes "Brandon Hansen considers the statistical accuracy of popular on-line weather forecast sources and shows who's on target, and on who you probably shouldn't rely. Motivated by a trip to a water park that was spoiled with hail despite a 'clear sky' forecast, he does a nice job of depicting deviations, averages, and overall accuracy in a manner that stats junkies are sure to love."

The more the merrier?

[Reverse Gear]

What a nice piece of work.

I can't help but smile a bit that MSN weather in this test turns out to be the absolutely worst when it comes to accuracy in almost all categories.

I would think there is a lot of difference on how the forecasts are made in the different channels, some of them probably do get a lot of their information from meteorologist working on their own stations. I wouldn't wonder if MSN doesn't have a meteorologist (or maybe only one) working to provide their forecasts.
Computers and simulations play a big role in predicting the weather today, but human eyes are worth a lot still.

I don't myself live in the USA, so my primary use of these are to check on when there is severe weather in areas where I know someone.
I have gotten used to check on weather underground for this information, I haven't checked on many other weather channel, but I feel quite well capable of following what is going on in the USA with tornadoes and such here from Denmark.

For a long time we only had one weather forecast service here in Denmark, a national institute. Since a primarily private TV station (TV2) a few years ago started their own weather forecast service, I really feel the national institute have been pulling themselves together and have provided many services that they didn't provide until now. So even though some of the services provides terrible accuracy they might still serve a good purpose in giving the other services competition and thereby forcing them to improve also.

When I am really dependent on the forecast I tend to study the information behind the simple prediction of the given weather, that way I am also much better prepared for possible scenarios, knowing which front move where and can better "read the skies".

NOAA/NWS

[zoward]

I use the NWS website, mostly because I hate all the annoying flash ads on most of the other sites. I was also under the impression that most of these sites get their information from the NWS and pass it along to you (along with a bucket of ads). There was a lot of complaining amongst the popular weather sites when the NWS opened its own web site.

Re:The more the merrier?

[Gabrill]

It seams odd to me that he started the project because of rain, and then completely ignored rain in his observations. Otherwise, the study was very cool.

Reliable forcasting method...

[AmIAnAi]

I remember some years ago a radio presenter saying that you could achieve greater accuracy than supposed weather forcasters simply by using the assertion: today's weather will be the same as yesterday. Have we moved on from this position?

rain?

[Shooter6947]

What about predictions for precipitation? The author claimed that he was motivated by a washed-out waterpark visit. Variations in the high temp by 3 degrees don't really matter a lick -- what matters is if it predicts sun and you get rain, or vice versa. How accurate are those "20% chance of rain" predictions really? Inquiring slashdot readers want to know!

I want one of these for stock analysts.

[Anonymous Coward]

Anyone know who tracks the accuracy of individual stock analysts?

Re:rain?

[Wordplay]

You'd need a lot of data for that one, if you want to establish the accuracy of the probabilities. Unless I'm missing a more mathematically clever way to do it, I'd assume you'd require lots of 20% days to determine whether rain happened on anything near 20% of them. Similarly, 10, 30, 40, 50, etc would require their own groups of lots of days.

It'd be even trickier in, say, the SF Bay Area, where it only rains for two or three months a year, and then almost every day. Your 0% and 80-100% groups would be well-stocked, but not so much the other ones.

Interpretation of the models is everything

[Aliks]

I am not a meteorologist, but I have worked with them a few times.

Generally the competing weather models will show a range of possible outcomes with various probabilities. You can average across all scenarios and come up with a 60% probability of rain, but the more days out you go more the scenarios diverge, so the less useful a single average will be.

Most people would not find it useful to hear that "there will be probably be thunder on Wednesday if it remains hot enough, but if it cools down on Tuesday then the thunderstorm will be off to the north somewhere"

Additionally, a lot of weather conditions are influenced by thin layers of cloud high up, so thin that precise measurements are critical so precise forecasts in one location more than 3 days out are difficult.

My experience with the BBC weather forecast...

[OneSmartFellow]

for my area is that they are usually accurate down to a period of about 3 hours. As an example.

http://www.bbc.co.uk/weather/24hr.shtml?world=41 79 predicted that it would be snowing yesterday morning by 0600. Sure enough, I woke at six, and it was snowing. I awoke earlier in the night (about 0400) and it hadn't yet started.
Similarly, that site predicted that the snow would drop off by noon, and turn to sleet or rain by 1600. Again, this prediction came true, within an hour of the predicted time.

Generally speaking I find the BBC weather site to be accurate significantly more often than not (guesstimate 80% accuracy) with the 24 hour forecast being almost universally correct, and the 5 day forecast being the least reliable. (as expected)

This is a FAR cry from the weather predictions when I was a lad. Then the weather forecast on TV was simply a way to poke fun at the meteorologist, who clearly was doing the best he could, but invariably got it wrong.

Re:What is your source?

[Anonymous Coward]

The fun part.... My personal weather station is only lacking radar, I do have old slow scan satellite imaging from a ham radio receiver. and I can predict weather at my location far more accurate than the NWS and the local stations.

By looking at the rate of barometric pressure drop and the rise in humidity and wind changes I can tell you within a 1/2 hour when it will rain or storm and typically how badly.

What blows my mind is why does the National weather service not install a crap load of cellular connected mini weather stations across the country in the military grid coordinates? it would give them a ton more data to parse to increase the accuracy of the predictions.

The fun part is when I go to some of the local media functions and start talking to local weather experts with their PHD in weather science I talk about my observations and what I notice in trends and they crap their pants when I tell them MY pc is running software I wrote to do most ofthe work for me and no I do not have a degree in weather science.

Re:NOAA/NWS

[smokin_juan]

Absolutely! Even if the graph is "wrong" you still get an approximate idea of what's supposed to happen. Hope you like the daylight indicating shading... it was my idea ;). Too bad they didn't implement the suggestion to change "48 hour period" to a drop menu that'd let you show 48, 72, 96 hours or the whole 5 days.

Anyway, I'm surprised this is the first time I've seen (would've seen if the site weren't 'dotted) forecast accuracy data. With everyone claiming their weather is the most accurate, you'd think someone would've tried to prove it before now.

Re:NOAA/NWS

[knightri]

In the US today, two major organizations release weather forecasts, the National Weather Service and AccuWeather which is based in central PA. However, both of these services get their information from the same source, the satellites, and base their forecasts on the runs of the various computer models. The major computers are the GFS, the NAM, the UKMET and the ECMWF. There are many more though and each is fed the satellite data/ground observations etc to help make a forecast. Some computers, like the NAM only go out 96 hours, while others like the GFS go out 15 days. I live in NJ, and currently, most of the major computers have a serious snow storm headed our way for next week. One area where I find the computers lacking is that they don't know how to respond to new snow falling on the ground. For example if a major snowstorm were to cover a large portion of the country with snow, the temperature in those areas would be 15 degrees lower than expected just because of the snow. The computers don't always account for this. I could talk forever about this so I'll stop here. Most of the time I find going outside and looking is the best way to see whats going on. Just a few of my cents. -Alex

ForecastWatch and ForecastAdvisor

[AceGopher]

I was quite curious about weather forecast accuracy as well. So three years ago I started collecting weather forecasts from the primary providers (Accuweather, Weather Channel, NWS, CustomWeather, Intellicast, etc.) and comparing them to actual observations. It's tougher than you might imagine, and there are a lot of factors that need to go into creating usable verification statistics.

I have a public site with some statistics for about 800 locations in the US available at ForecastAdvisor.com [forecastadvisor.com]. There is also a blog with more in depth analysis (like how do temperature forecasts fare relative to how deviant the actual temperature is [forecastadvisor.com]...in other words how well do forecasts do the further away from normal the actual is, and how to forecasts fare the further out they forecast for [forecastadvisor.com], and how does forecast accuracy compare over time [forecastadvisor.com].).

ForecastWatch.com [forecastwatch.com] is used by meteorologists and professionals. Accuweather, The Weather Channel, and several private meteorological companies use this system to help them understand and improve their weather forecasts.

And a geek note: ForecastWatch.com [forecastwatch.com] runs on Quixote (a Python web framework), while ForecastAdvisor.com [forecastadvisor.com] runs on Ruby on Rails. The back-end forecast and actual collection, and calculations are Python with a MySQL database. Both sites are close to migrating to Django, a new Python web framework and ORM.

-Ace

Why trust the intertubes?

[Asklepius M.D.]

When I did SAR in Alaska, a good weather forecast was essential. It didn't take long to realize that every source for information analyzed their data differently and came up with slightly different conclusions. The solution? Learn to forcast yourself. Learn to read an isobar map, learn to predict wind/temp/precip for your local area and you'll get a far more accurate feel for what's going to happen than any regional analysis could give you. Practice locally - you'll learn what worked and what didn't, then apply it to other locations. When I went to a friends wedding in Minnesota, I was able to predict when the rain would start to within 5 minutes - just enough time for the outdoor ceremony to be performed.