How it's made: How Data Provisioning Adds Value to Weather Data
We’re rounding up Category Four: Quality Control & Data Management in the How It’s Made series, with this article that reveals how data management contributes to creating your weather forecast. How It’s Made is a 20-part series, which gives you a look under the hood of weather forecasting, sharing the knowledge and experience of the weather experts.
Missed any of the earlier posts in the series? Check out the How It’s Made series here.
Make sure you don’t miss out on the rest of the series. Sign up for the weekly How It’s Made Update. Stay informed and get weekly updates, delivered straight to your inbox.
Your organization is one-of-a-kind. The data you’re using, tracking and storing is likely to be unique to you and your needs. But over time, data integrity can be compromised, which is bad news for business. Poor quality data costs $3.1 trillion a year in the US alone.
The good news for you is that, when it comes to your weather data, you don’t need to worry - so long as the experts manage it.
Data scientist enjoy solving complex puzzles. If something doesn’t work as expected, all kinds of cogs start spinning in their heads to try and solve the problem. They dig into systems and also work to achieve a robust future-proof solution, ensuring the weather data is as accurate as possible. Let us show you how they do it.
“The Data Provisioning team is taking care of processing weather data to the highest standards, to give the weather experts the most accurate and reliable information they need."
Dr. Marco Radke-Fretz,
Data Manager (Data Provisioning)
How the experts manage weather data effectively
The Data Provisioning (DP) division is responsible for ingesting, processing and storing all incoming data; both sourced and generated. If something goes wrong in that processing, if parameters are missing or lacking, then DP will look for the leak and get it sorted as soon as possible.
However, the senior software engineers do much more than just solve problems. They streamline data processes, construct the transition of all data processes into the cloud, and extract all kinds of derived weather data from the incoming information, ensuring that it’s ready for the weather room.
The data is processed by the team to the highest standards, to give the weather experts the most accurate and reliable information they need. The data is shared with internal departments, to form the backbone of each and every forecast.
The Two Data Provisioning Disciplines
Data provisioning for weather forecasting covers two distinct areas:
1. Observation data, which deals primarily with observation, lightning and radar data
2. Model data, which focuses on model/grid and ensemble.
Although there are some exceptions, the data provisioning division primarily provides data to other internal teams, rather than directly to the customer.
How Data Management Adds Value to Weather Data
This technology continues to evolve and innovate at a rapid pace, but there are four fundamentals, where data management adds real value:
● Trust: 99.9% data availability, migration to the AWS cloud, and continuous monitoring of servers and services by the Operations Center all help instill trust in the team
● Availability: data is delivered in industry-standard formats including SI (Système international) and WMO-approved units (e.g. Celsius)
● Speed: Over 90% of the data is available within a few minutes of it being received from a third party
● Visualization: usability of forecasting data is vastly improved thanks to the ability to data and prepare it for online and on-screen presentation
Ensuring modern data management
Modern weather forecasting data arrives in large quantities and from many different sources. Without the best DP teams and technology, the forecasts simply wouldn’t be as powerful or reliable as modern customer expect it to be.
But there’s still more to be done. Historically, data was processed via local servers. This meant data processing had to be done step-by-step because server capacity was limited. Essentially, when data came in, the first step in the process could start, and only when that was finished could the next step begin.
Now, investment in modern cloud computing enables DP to run tasks in parallel, meaning more can be done at the same time. Plus DP can automatically scale up or down processing capacity, depending on their needs at the time. There’s still more work to be done to make all stages cloud-based, but doing ensure data processing is future-ready.
Sign up for the weekly How It’s Made Update. Stay informed and get weekly updates, delivered straight to your inbox.
The Data Provisioning team is taking care of processing weather data to the highest standards, to give the weather experts the most accurate and reliable information they need.