I just got back from a fabulous three days of learning at the Australasian Evaluation Society conference in Canberra. I tweeted my little heart out and before I rest my weary fingers, I thought I would jot down some of my top of mind reflections from the conference.
I think there’s a sense that we are living in dark times. Trump, Duterte, the global treatment of refugees received attention from more than one of our keynote speakers. We were sitting in the capital, while in Melbourne, the Federal Government argued for the right to hold a postal vote on the rights of a section of our population.
Evaluation has long posited itself as a public good, a means of overcoming authoritarian, arbitrary decision-making. But Sandra Mathison’s opening plenary challenged us to question our assumptions.
“If we talk about evaluation making the world better, we have to own that claim.” Sandra Mathison
She was highly pessimistic in her judgement, but in the path she laid forward I saw hope. She argued strongly for the empowerment of the powerless through evaluation, and if there was one theme I saw more strongly than anything else in this conference, it was a renewed excitement in the democratisation of evaluation.
Co-design has become a buzzword in recent years. I do a lot of work in the mental health sector, especially in youth mental health, where a lot of great work is happening to include the beneficiaries of evaluation in the design and delivery of the approach, not just as research ‘subjects’. I’ve always been a fan of models of evaluation and policy design that I think are co-designed and participatory. They overlap with my core values of rights, democracy and empowerment.
But I think there’s a huge risk with these approaches in a policy environment that tends to authoritarianism. Tokenistic co-design flourishes here. As Yvette Clarke put it “empowerment is an outcome, not just a process.” I think there is a risk that we have viewed participatory evaluation approaches as a means to an end, not a democratising evaluation process in and of itself.
“We think evaluation is about the return we get from evidence; that’s not enough. We can do far more with evaluation.” Dugan Fraser
So the call to arms at the conference couldn’t have come at a better time, and I was heartened to see that the rally cry came from across the evaluation spectrum – commissioners, consultants, academics. We all want to be doing this work.
I chaired a session on LGBTI inclusion in evaluation by Jeffery Adams, and the discussions that ensued were some of my favourite of the conference. We can’t assume that we as evaluators know what is best for people or even how they identify. Just as visibility through evaluation findings leads to action, invisibility leads to inaction. It’s important to allow people to find their own voice to ensure they are seen and heard, but also so they are seen, heard and interpreted as they want to be.
Understanding identity is inherently tied to the discourse of power and control, as Sandra Mathison, Dugan Fraser and Richard Weston emphasised in their keynotes. Richard Weston put this perfectly when he said that government needs to shift from being the ‘fixers’ to the ‘enablers’ of community needs.
“Governments don’t own the stories of Aboriginal people.” Bronwyn Rossingh
I think this will take more than the traditional top-down evaluation and partnership approaches that much have been to date. I still haven’t quite worked out how we get commissioners, evaluators and ethics committees comfortable with acknowledging the value of Indigenous knowledge systems and ways of evaluating – answers on a postcard please – but supporting communities and Indigenous evaluators to take the lead can only be the start.
As non-Indigenous evaluators working with Indigenous communities to design, implement and evaluate, we have to be explicit in the promises we make to stakeholders through the co-design process and the expectations we establish, as Matt Healy outlined. One of the ways to do this by asking a series of facilitated questions with your stakeholders at the outset to understand what everyone’s values are, what they aim to achieve through the intervention and how they would define success. These are some of the issues Caroline Tomiczek and I discussed in our presentation today.
I’m an evaluator, but, as a person with a lived experience of mental ill-health, I am also a beneficiary of programs who, at times, has been at risk of losing her voice. I think this was what made the conference so exciting for me. Governments all over the world might be falling apart, evaluators might get caught up too much in their discipline sometimes, but ultimately, the kinds of evaluations the sector wants to be doing, are the kinds of evaluations, I want to be doing, as both an evaluator and a beneficiary, and that gives me hope.
