In recent years, the idea that the participation of the actors in research is a key element in research to transform and generate impact in society has been insisted on repeatedly. For those who are still a little unsure of this statement, studies in the assessment of the impact of research demonstrate the idea. They emphasise the participation of professionals, patients and the remainder of the actors of the system throughout the entire process of research during which the effectiveness of research in reaching society is improved.
Hence, the AQuAS, in collaboration with the Fundación Bancaria “a Caixa”, considered it necessary for the systems of assessment to be involved too if we want research to have a greater impact. With this objective, and a great contribution from Maite Solans and Paula Adam, the first monographic on Responsible Research of the SARIS (Assessment System of Research in Health) was presented. As a result of reviewing the existing international literature, 47 indicators were identified that measure the activities and participation of the actors in research institutions.
The indicators identified cover a large part of the research ‘process’, and they have been grouped in six aspects – training, governance, resources, activity, dissemination, and primary results. This allows us to see how the different actors in the system can be involved during the entire process of research (planning, execution and translation).
Figure 1. Aspects in the process of research
However, we need to be cautious! On the one hand, the proposed indicators can be useful to stimulate and open up the reflection on how to monitor the activities carried out in research institutions in relation to the participation of the actors of the system. On the other hand, it must be pointed out that an indicator must be used responsibly by taking into consideration their context, incentives and the distortions that a measure can provoke. Experts in measures and indicators in science know very well that the research fabric and the research system change their behaviour according to what is being measured, ‘you get what you measure’, said Cyril Frank, chairman of the panel on the impact of research in health of the Canadian Academy of Health Sciences (CAHS).
Referring to participation in research, the question of measure is even more complex because it is still in its infancy. As Derek Stewart said, director of Patient Involvement of Nottingham Biomedical Research Centre and speaker at the SARIS session where this monographic was presented:
“One of the greatest challenges of measures is that ‘inputs’ are often measured – the number of people that participate, the number of groups that have been called or who have been talked to or the number of studies in which patients participate. It is about carrying out actions that add value, that help provide knowledge for solutions for patients”
Derek Stewart’s reflections during the preparation of his conference can be found on his blog ‘Making a difference’.
Indicators should help to answer specific questions based on each situation and need, and this is why it is essential to assess their relevance and feasibility within their context. The concept of relevance refers to the proximity of an indicator to a need and specific question that we pose; for example, what is the real value of the participation which an indicator measures? The concept of feasibility refers to how easily available and affordable data is when constructing an indicator.
As can be seen in Figure 1, among the indicators found in the literature there is a large number that are relevant, which was to be expected because they are recommended by very prestigious entities. Nevertheless, their feasibility in our environment is low in the majority of cases which means that although we have identified several indicators with limitations, none are fully recommendable.
Figure 3. Feasibility vs Relevance of the 47 indicators
Therefore, bearing in mind the context of research in Catalonia, at AQuAS we have catalogued the indicators as follows:
- 27 indicators as not recommendable for use
- 20 indicators with limitations
- No indicator as being fully recommendable
Our findings allowed us to learn a valuable lesson: before defining indicators or measuring the participation in research, we must first explore what is done in our environment and so by first analysing what is feasible we can then decide what the most relevant actions are.
The success of assessment systems depends on the acceptance of their processes and results among those in charge of taking decisions and the research communities which are being assessed. The challenge, thus, is to understand what is done in practice in our environment based on the guide which the indicators in the international literature have provided us with.
Let us continue!
Post written by Ion Arrizabalaga.
(Post published jointly in the AQuAS blog and the blog of CaixaCiència)