Suzanne Lazaroo
21 May 2026: When philanthropic giants Movember and The Distinguished Gentleman’s Ride (DGR) funded 15 organisations to run 23 mental health initiatives for veterans and first responders, they turned to University of Canberra researchers to assess the impact of the diverse, multi-country projects – drawing on their ground-breaking Global Impact Analytics Framework (GIAF) developed to support better impact assessment, project design, organisational learning and effective direction of funding.
“We had to evaluate projects that were completely different in their design, context and implementation – the only thing many had in common was the funding source,” said GIAF co-lead Professor Luis Salvador-Carulla, from the University’s Mental Health Policy Unit, at the Faculty of Health’s Health Research Institute (HRI).
“The GIAF enabled this, because it can be used to run comparisons across heterogenous projects and contexts, and so allows us to conduct multi-project program evaluation. We expect it to support researchers, policymakers, and health service planners, by enhancing their capacity to evaluate complex interventions, compare outcomes across jurisdictions, and inform evidence-based decision-making.”
A comprehensive suite of tools, the GIAF includes a taxonomy or knowledge map – showing the throughput components in the process of implementation, and how they are related – a glossary to get everyone speaking a common language, checklists for data gathering and ladders or scales for measuring.
These ladders enable qualitative data – what Professor Salvador-Carulla calls “narrative data” – to be translated into numeric data, and then combined with quantitative information.
The framework – an accessible, open-source suite of tools – is the first of its kind, and available for use under a Creative Commons license. The team is also running courses to help researchers use it.
“Our initial scoping review revealed 71 assessment frameworks in existence, but none with tested tools, or that took into account the ‘black box’ of the research on implementation process – the part between the inputs and the outcomes, which reveals how a result comes about,” said GIAF co-lead Associate Professor Sue Lukersmith.
Led by Associate Professor Lukersmith and Professor Salvador-Carulla, a team of 31 experts in implementation science from Australia and Europe worked to bring the framework to life.
The GIAF is the culmination of over 13 years of research, testing and refinement – the idea sparked when Professor Salvador-Carulla and Dr Lukersmith first crossed paths in 2012.
“We both worked in industry-based research, but separately – Luis in Spain, me in Australia,” said Dr Lukersmith. “We’ve always had a keen interest in the evaluation of research impact in the real world, rather than in a controlled environment.”
No controls in the real world
“Sue approached it from a best practice service and provider perspective, while I had been working in public health agencies and therefore seeing the problem from a government point of view – from both perspectives, it was clear that there was a huge gap in the way implementation was evaluated,” Professor Salvador-Carulla said.
This was, in part, because the traditional research paradigm adopts an evidence-based approach built on randomised control trials – and there are no such controls in the real world.
Because the GIAF doesn’t require the controlled environment of a lab or clinic, it can be easily deployed to test the mettle of community service projects, education initiatives, and more, in everyday settings.
“This also makes it ideal for use in complex healthcare environments, which have many variable components – when you analyse these components separately and then put them back together, it tends to give you a different picture every time,” said Professor Salvador-Carulla.
“Context and time – among other factors – will differentiate each analysis from the previous.
“For instance: We assessed the EMPOWER project in Europe, a digital platform for improving wellbeing and mental health in the workplace – it was very interesting because it was at the time of the COVID-19 pandemic. And then came the war in Ukraine, which also had a huge impact. Traditional methods were simply not fit for purpose to analyse the project in the context of all those real-world complexities.”
The project leads have taken what Associate Professor Lukersmith calls an ‘ecosystem approach’ – rather than just assessing the elements of a project, the GIAF expands its analysis to the context around the project.
“This is why the using the GIAF will allow you to meaningfully compare projects in Canberra with projects in Sydney, Dubbo or Dublin, all of which will have very different contexts and settings,” said Associate Professor Lukersmith.
“And it’s why we were able to use it to evaluate the 25 Movember and DGR-funded projects, which were run across seven countries.”
The Movember projects were funded for three and a half years.
“There were over 13,000 participants in total – veterans and first responders, including police officers, paramedics and firefighters,” Associate Professor Lukersmith said.
It was a very heterogenous group of participants, and the projects were equally diverse – some aimed at helping the families of first responders, others aimed at just one group of first responders. Some of the organisations running the programs were also the employers of the first responders, others were universities or not-for-profit organisations.
“We needed to be able to compare such different projects and groups in a meaningful way, recognising what worked and why it worked – and the GIAF was able to take us beyond comparing apples to apples, and look at comparing three oranges to a pear – in a way that is fair,” Associate Professor Lukersmith said.
The evaluation ‘black box’
Another problem the GIAF looked to address was that traditional evaluation methods generally look only at input and output, rather than considering what actually led to outcomes.
For instance, in assessing a multi-faceted program to reduce smoking, researchers look for outcomes such as a reduction in cancer diagnoses or cardiovascular disorders, theoretically related to the implemented program – but how do you know which intervention actually made a difference?
“It could be a new policy, strategy or tool that leads to a result – but it’s important to understand which exactly,” said Professor Salvador-Carulla.
“In order to assess the outcomes of your intervention, you need to understand the process between implementation and the outcome you are looking at. This is the ‘black box’ – and previously, no one has tried to apply a quantitative aspect to it, to really grasp this connection.”
One of the team’s first analyses was of the Ed-LinQ program from the Queensland Mental Health Commission, aimed at improving the relationship between schools and mental health centres for children and adolescents in the state.
“When the program was introduced, the initial assessment noted a huge increase in the number of visits of children and adolescents to the community mental health centres, so they thought that the program was very successful,” Professor Salvador-Carulla said.
“Then they asked us and a company called Connectica to run an assessment with the GIAF, to get a better picture of what was happening. We found that just one per cent of the schools in question had really adopted the Ed-LinQ strategy. So the desired outcome was there – an increase in visits and referrals to the community mental health system – but it wasn’t related to the program. This then impacted how the next phases of the program were designed.”
The importance of a common understanding `
“In our research we found that people often use different words to refer to the same thing, which can mean a big difference in definition,” said Associate Professor Lukersmith.
“For instance, ‘uptake’, ‘spread’ and ‘reach’ – these three words are often used interchangeably. This is why we created a glossary as part of the GIAF, to get people speaking the same language.”
The GIAF glossary contains 189 terms defining all the domains and the subdomains in the taxonomy, to promote a common understanding.
“We also did a lot of work with community of practice platforms for knowledge sharing,” Professor Salvador-Carulla said.
“This enhances the capacity of cooperation and sharing across different projects. In some ways, we provide a common language that allows these projects to communicate, and it creates a new environment in which to understand research in the field of implementation.”
Assessing the impact … of the impact assessor
In the case of the Movember and DGR projects, the impact of the GIAF assessment has been clear. Some projects have expanded; others have used the assessment to advocate for more funding.
“It has also driven some design changes, because organisational learning is a cornerstone of the GIAF philosophy,” said Associate Professor Lukersmith. “In the course of the evaluation, we mapped and tracked what was actually going on with each project, and that allowed stakeholders to learn from each other."
The team has found the GIAF can also be a valuable development tool, invoked at the start of a project.
“When running projects through the GIAF, we have spotted critical issues in design that were not taken into account by traditional paradigms,” said Professor Salvador-Carulla. “This has sometimes meant they are simply not ready for implementation, but in other cases, they have simply been aimed at the wrong target audience.”
To complement the GIAF – especially when using it for multinational, cross-cultural comparisons – the researchers have also come up with a companion tool, which they call the Barriers and Enablers Register.
“These barriers or enablers could be cultural, governmental or policy-related that are local to a particular area,” said Associate Professor Lukersmith. “This tool also uses narrative data, translates it into a numerical form and helps to group those into categories. In this way, you can, for instance, compare policies in different countries and consider if they function as barriers or enablers.”
The GIAF is an accessible, open-source suite of tools which has so far been adopted in 16 studies across 10 countries.
“Our general strategy at HRI is to develop tools applicable in the real world, which can be used as extensively as possible. We allow for Creative Commons usage by public agencies, universities, researchers, and not-for-profits,” Professor Salvador-Carulla said. Organisations can hire the team to conduct assessments for them.
“We also run courses on using the GIAF, with another coming up in October at UC’s Sydney Hills campus. While you don’t need to attend a course to use it, it is recommended to attend a course, as it does help to understand the bigger picture and get the most out of the tools.
“Another advantage of attending the course: you can bring your own case study for discussion.”
Future plans include creating online training modules, but this is dependent on funding.
Ultimately, the GIAF is set to bring a whole new meaning to implementation science the world over.
“The GIAF could be applied in any sector or context,” Associate Professor Lukersmith said. “It's project neutral. So it doesn't matter what sector or industry you’d like to apply it to – whether it's implementing a work health and safety initiative in a workplace, or a policy in schools for an education program for at-risk children – the GIAF will reveal actual real-world impact and drive better design and evidence-based funding decisions.”
Find more information on the framework at The Global Impact Analytics Framework - University of Canberra.
To sign up for the GIAF course in October or join the mailing list, email Sue.Lukersmith@canberra.edu.au or Sally.Fitzpatrick@canberra.edu.au.
