IRELAND 19 February 2018
Value for Money Examination 35. Met Eireann Summary
Met ?reann is a division of the Department of Public Enterprise but operates relatively independently within the terms of an administrative budget agreement.
The mission of Met ?reann is to meet the national requirement for high quality weather forecasts and associated services, with optimum efficiency and value for money.
This examination was concerned with the techniques used to determine the accuracyof weather forecasts and with the arrangements between Met ?reann and the Department for the management of the performance of the service.
Accuracy of Weather Forecasting
The accuracy and timeliness of weather forecasts are key performance criteria for Met ?reann. Accuracy is considered for three different kinds of service - the underlying computerised weather prediction model, general weather forecasts andspecialised forecasts of particular aspects of weather such as wave height and severe weather events. A weather forecast is based on predictions of a number of weather elements including pressure, temperature, humidity and wind speed.
Data reviewed in the course of the examination indicates a significant improvement in weather predictions since the introduction of the current prediction model in 1994.
Over 120 general weather forecasts are issued each day through radio, television and the press, and through a telephone and fax service known as Weatherdial. The examination findings are
- General forecasts (for example, the weather forecast broadcast at 7:55 a.m. onRT?Radio 1) are typically given in non-numeric terms which limits the scope for assessing their accuracy.
- During this examination, a special exercise was performed to measure theaccuracy of minimum and maximum temperature forecasts given over a three-month period on the Weatherdial service. The results indicated that almost half of the temperature forecasts were within one degree of the temperature observed while three quarters were within two degrees
- A comparison of the accuracy of actual weather forecasts with persistenceforecasts (i.e. forecasts based solely on weather conditions in the previous 24 hours) generally confirms that the skill exercised by forecasters significantly improves the level of accuracy.
Specialised weather forecasts are provided for aviation services (e.g. wind directionand speed, visibility and height of cloud cover), for coastal shipping (sea area and wave height forecasts) and for local authorities (road ice warnings).
There is no standard method for measuring the accuracy of aviation forecasts. Forthis reason, international comparisons of accuracy are difficult. However, Met ?reann monitors the accuracy of its own aviation forecasts and has estimated that, in 1998, the internationally agreed standards of accuracy were achieved more than 90% of the time.
Data is collected on wave height from six buoys located around the Irish coastline. For the purpose of this examination, a comparison between actual recorded wave height and wave forecasts for a six month period in 1999 was made. This indicated that the forecasts were reasonably accurate with a slight tendency to under estimate the wave height.
For ice warnings and severe weather alerts, there are three possible outcomes whichcan be monitored
- a hit: where the forecasted weather actually occurs
- a miss: where the forecaster fails to predict an exceptional weather event
- a false alarm: where adverse weather is forecast but fails to materialise.
Met ?reann is developing an accuracy verification scheme based on the analysis of hit, miss and false alarm rates. Data for the winter of 1998/99 for forecasting road ice indicates a hit rate of 71% and a false alarm rate of 16%.
A special exercise to consider the accuracy of severe weather alerts (fog, wind, snow and rain) in the period November 1998 to January 1999 indicated a hit rate of 92%. However, this analysis includes warnings which were issued later than the targeted preferable to include in the accuracy analysis only warnings issued within a predetermined lead time.
Met ?reann provides a range of non-forecasting services which are mainly concerned with the provision of data to international agencies. Some external reviews are made which give reasonable confidence of the accuracy of the data supplied. The continuing acceptance by the international agencies of the data is also considered to provide some reflection of its accuracy and acceptability.
In recent years, Met ?reann has operated through an administrative budget agreement with the Department and has published customer service plans, statements of strategy and business plans. While these documents are very useful, there are a number of limitations which restrict the proper assessment of performance These include
- The high level objectives focus more on how the mission will be achieved rather than on the outputs and impact which is sought from its activities.
- No measurable benchmarks are in place for the range of outputs (e.g.forecasts), the volume of each output or the accuracy of the output.
- There is a lack of clarity in separating services which are to be provided on a commercial basis (like aviation forecasts) and those which form part of the publicly funded service.
- There is implicit prioritisation of services to meet the competing demands of clients. The strategic statements and business plans should make the prioritisation more explicit.
Met ?reann has had a general mandate to develop its commercial services since the mid 1980s. These activities should be underpinned by financial targets for turnover and profit and by a management accounting system. However, such a system is not in place. The accounting function is performed by the Department and is limited to the normal expenditure recording system for producing the Departmental appropriation account. The absence of a more developed accounting system meansthat the pricing of commercial services is based on estimates of the percentage of time spent by each division on the relevant services and an ad-hoc apportionment of overheads. Met ?reann is aware of these accounting limitations and is in the process of obtaining advice on the type of accounting system which would best serve its needs.
Some performance information has been included in Met ?reann?s annual reports since 1995 but the information provides a selective picture of efficiency and effectiveness as different services have been highlighted in each year. It is intended from the 1999 report onwards to present accuracy scores on data produced by theprediction model, the aviation forecasts, temperature forecasts for Dublin and Cork
and road ice predictions. It is desirable that performance reporting should be further developed to cover all services. Other aspects of performance such as cost and timeliness indicators and customer satisfaction measures should also be considered.