'The Sun' case: Translating accurate forecasts into unambiguous icons - and how to read them
MeteoGroup is good at forecasting. Especially storms. But last week’s UK media storm, we couldn't have predicted that.
The English newspaper The Sun wrote an article stating that: “Because of inaccurate forecasts on the BBC weather app, tourism chiefs claim business is hurt. They say many visitors were put off by storm predictions over a recent Bank Holiday weekend and stayed away. Yet many places apparently stayed dry." Jon Weaver, head of resort marketing and events in Bournemouth, added: “The BBC predicted rain all morning for the Saturday. Their picture for the day was pretty awful. In the end we had blue skies and sunshine. It’s so frustrating. It’s having a negative impact on businesses.” The discussion was then picked up by The Daily Mail and on social media.
In its official reply the BBC said: “We know how important it is for our audience to get accurate and up to date forecasts and we now use even more weather data from multiple sources through advanced statistical techniques. MeteoGroup combine these sources to create a superior forecast and on average, the statistics confirm that our users can rely on very accurate forecasts.”
There are two stories here:
- The Sun’s article, it’s conclusions and examples. And why it just wasn’t correct.
- The lack of clarify as to how our forecasts are made, how they are translated into hourly icons and daily icons and how these icons should be read or interpreted. How is it possible that here we have rain, yet one kilometre down the road it is dry? And how hard is it to predict that?
Article 'The Sun' incorrect
The story that has appeared in the Sun is simply incorrect and the data they have displayed in their mock-up graphic - comparing the forecast and what actually happened - is inaccurate.
Sun Mock-up app picture:
Take the London forecast. No measurable rain (0.0mm) was recorded between the hours of 12pm – 3pm. From 1.30pm there were sunny spells for the rest of the day with no showers. Forecast temperature was 18C and the actual temperature was 18C. We believe our forecast to be accurate.
Below is the total rain in Bournemouth from radar for the afternoon/evening of the 26th May. On the left is the 6hr rain up to 6pm and the evening (1800-0000) on the right, when there were frequent heavy showers and thunderstorms.
Dennis Schulze, Chief Meteorology Officer of MeteoGroup: “In the millions of forecasts we produce every day, people will always find places and times when someone else scored better. We have to ensure that we do better on average. That's why we have the Service Level Agreements on forecast accuracy.” “It’s simple: you can't just take three instances and then derive overall conclusions. And secondly, these three examples the Sun chose in its mock-up are not even correct, nor are the conclusions.”
Our senior Meteorologist Nikki Berry also explains this on Radio5 Live:
And her points are further supported by Dr Andrew Charlton-Perez, Associate Professor of Meteorology at the University of Reading, on BBC Radio4:
Accuracy in General
MeteoGroup’s forecast accuracy is superior to what many other weather services can supply. This was also thoroughly tested during the procurement process for the BBC contract. MeteoGroup and the BBC continuously work on further improving both forecast accuracy and presentation to the audience. Dennis Schulze: “The BBC weather app still uses weather models of the Met Office combined with MeteoGroup, incorporating model output from other national and international meteorological services (like ECMWF, NOAA, KNMI, DWD). Those are then considered in our statistical optimization to select the best combination of model data that fits a particular location and parameter. Our experienced forecasters eventually have the final control on the forecast.”
Meteorological Researcher Wim van den Berg wrote an excellent Blogpost explaining how hard it is to forecast exactly when, where and how much rain will fall: “Compared to a rain belt, a shower is short-lived (sometimes lasting only 30 minutes) and local (it may remain dry just 3km away). Showers are also strongly dependent on the daily cycle and local terrain. A shower forecast must therefore be described as “a risk” or “a possibility”. Showers will occur in some areas, however the when and the where are both uncertain. Forecasted radar is no longer as reliable as it was, as shown in the case of the rain belt. Due to the daily cycle, the radar cannot detect daytime showers before they develop and may forecast showers from the afternoon through to the evening, while in reality they fade.”
Forecasts and Icons
The BBC Weather app and website provide the percentage chance of precipitation in their hourly forecasts. Each breakdown gives the percentage chance of it raining where you are at at each hour mark, but what does a 40% chance of rain at 14:00 mean? Is it a 40% chance it will be raining at that exact time, that it will rain for that percentage of the hour from 14:00 to 15:00, that there's a 40% chance it will rain continuously for the whole hour, or that that amount of your local area will experience rain?
As counter-intuitive as it might seem, hourly predictions of the chance of rain actually refer to the 60 minutes preceding - not following - the time mentioned.
"We have to verify our forecasts," says Nikki Berry, senior meteorologist at MeteoGroup, in an interview for BBC Radio4. "So, if you imagine putting a measuring jug out in your garden at 13:00, and you go and check it at 14:00, the 40% chance of rain at 14:00 is the chance of you having some rain in your jug at 14:00. So, If you're heading out at 14:00, looking ahead to the forecast at 15:00 will give you some guidance for how the weather might play out."
How do forecasters choose which symbol to use?
"The probability of rain is one element," says Nikki Berry, "but we're also trying to forecast the amount of rain - is it a high chance of a little bit of rain, or a low chance of a heavy downpour? When we're creating the weather symbol that you see above the percentage chance of rain, that's not just linked to the percentage chance of rain - that's linked to the humidity, to the temperature, the wind, and whether the rain is convective - showery - or whether its large-scaled with a band coming in."
Chief Meteorology Officer Dennis Schulze: “The standard MeteoGroup method to create the day symbol is a balanced approach, to show the character of the weather of the whole day. One hour of rain will not ruin the day so it will not show up in the day symbol. For a number of customers we have created some customer specific day symbols, where we select the most impactful weather for the day summary symbol. Usually that tends to be somewhat biased towards bad weather. From time to time this can give the wrong impression of the weather forecast and we are working with our customers to review this approach in the future.”
It takes Nikki Berry just 7 minutes to explain. Listen to the clip below:
Or read this blogpost of Wim van den Berg.
It’s really quite simple. You can't just take three random instances and then derive overall conclusions. The story that appeared in the Sun was simply incorrect and the data they displayed in their mock-up graphic - comparing the forecast and what actually happened - is inaccurate.