IRELAND 24 November 2017
Value for Money Examination 30: Performance Measurement in Teagasc Summary
The effectiveness of an organisation depends on the extent to which it achieves its objectives. Determining effectiveness requires a clear definition and assessment of the impact of the activities of the organisation. This is a complex area, as it requires understanding of the significance of the outputs of activities and their contribution to impacts. Many outputs and impacts are difficult to measure and in these cases some alternative means must be found to determine impact. This can be done through qualitative techniques such as comparison with similar activities elsewhere or through surveys of the client population served.
This report examines the systems and procedures employed by Teagasc to evaluate its own effectiveness. Teagasc was formed to provide services for the agriculture and food sectors in three distinct services areas ? research, advisory services and training. Teagasc 2000 is a comprehensive plan for the future delivery of services which was published in the middle of 1998. Management systems including those associated with the determination of effectiveness are being developed in light of proposals in Teagasc 2000.
Evaluating Programme Effectiveness
Teagasc?s three service areas are divided into five programmes and 24 sub-programmes. The stated objectives for most of the sub-programmes are relatively clearly defined and susceptible to measurement, so the extent to which the desired impacts at that level are achieved can generally be evaluated. Programme objectives set are generally less clear and closer to organisational level gaol statements.
While many of the elements required for a system to determine effectiveness are in place and much good work has been done, Teagasc does not have a comprehensive approach to evaluation of programme impacts. The principal limitations noted are discussed in the following paragraphs.
The agricultural production research programme is intended to benefit Irish agriculture but the objectives set lack precision and could usefully be set in context. Specific impact targets were not set in all cases, although some good examples were noted such as research to achieve the production of milk at 9 pence per litre and of mid-season lamb at 136 pence per kilogram. Where specific impact targets were set, the absence of a suitable benchmark and of a time frame for achievement of the target limits the extent of evaluation which might be performed.
A distinction is made between pre-commercial and commercial research in the food production research programme. The quality and quantity of commercial research which follows on from the pre-commercial phase is one impact indicator of pre-commercial research. The relevance of the information gained from the pre-commercial research to other programmes goals is also an impact indicator but this can be hard to isolate. For commercial research, proxy measurements such as the participation of clients in the research and the direct improvement in processes would provide an indication of impact. Teagasc has not set impact targets for food production research and does not use techniques like cost-benefit analysis to evaluate the potential or expected contribution of the research to commercial economic returns.
Evaluating the impact of the four sub-programmes which make up the rural economy research programmes is particularly difficult due to the nature of the research. Teagasc has specified performance indicators which identify and track research outputs. No formal arrangements have been made to evaluate the impact of the research or the quality of the research outputs.
The advisory services programme provides a good opportunity for implementing an impact evaluation system. The services provided are tailored to either full time of part time farmers. Many of the services are provided on a fee paying basis and Teagasc regard the willingness of clients to pay fees as a measure of successful impact. Impact targets have been identified in the programme activity plans but are not adopted formally as programme targets in Teagasc 2000.
The impact of the training programme is measurable in terms of the direct impact of the training provided on the knowledge and competence of the workforce in the agriculture and food production sectors. Some measures have been adopted such as the number of participants on courses and the number gaining employment and implementing the techniques learned. However, better impact measures are available such as the proportion of entrants to farming who hold the Certificate in Farming. The means to gather information about the impact of training is available through existing demographic and labour force analysis for the agricultural sector.
Evaluating Organisational Effectiveness
The system to evaluate organisational effectiveness relies on the same basic ingredients of clearly defined impact objectives and systems of measurement and evaluation as are applied at programme level. Evaluation at the corporate level is more complex due to the potential conflict between competing organisational objectives, the extent to which impacts may not be measurable and the need to consider not only the contribution of programmes but also the linkages between them. New methods of analysis, such as the balanced scorecard approach, would help with the evaluation of the achievements of multiple impact targets.
Teagasc has applied a strategic approach to the conduct of its activities since it was founded. The latest statement of strategy, in Teagasc 2000, sets out clear organisational objectives which were developed with appropriate consultation with stakeholders. Most of these objectives are capable of being measured.
The documentation of the evaluation of research project outputs has been strengthened since the introduction in 1998 of end-of-project reports. A special publication called Teagasc Achievements was introduced in 1996 to highlight particular achievements. While these initiatives are steps in the right direction, there is a need to distinguish the evaluation of impact from routine management reports of achievements and end-of-project reports of outputs.
There are a number of limitations to current evaluation practices. The strategic documentation does not provide for a corporate approach to evaluating organisational impact. Specific measure or indicators to measure organisational impact have not been identified for corporate goals. The approach to evaluation of effectiveness where desired impacts are difficult to measure (such as the rural economy research) has not been worked out. There is a risk that evaluation exercises will be selective rather that comprehensive.
A good starting point for setting up a system to evaluate impact would be for Teagasc to produce a plan for the comprehensive evaluation of corporate goal achievement. The plan might harness the work of external evaluators of Teagasc such as the Department of Agriculture and Food and EU monitoring committees. The internal structures for facilitating these independent evaluations could provide a framework on which to organise the evaluation system. The level of resources required for evaluation should be defined but the development of evaluation systems should not become a rigid management procedure or an expensive data gathering exercise. The aim should be to promote a culture of impact evaluation as part of the existing strategic process.