Leadership lessons from school principals

source: pixabay.com by @ThinkTanks

Part of my role in overseeing professional learning at my school is building a variety of ways to develop the capacities of leaders. Our termly leadership forum, a new initiative this year, provides a place and space for all of our leaders – from coaches and pastoral leaders, to heads of faculty, senior leadership, and the Executive team. We meet each term for an evening of wine, cheese, provocation, and connection. In Term 1 I ran a session with the Director of Strategy on thinking about leadership in terms of research, organisation, team, and self. In Term 2 we welcomed Professor Dylan Wiliam to our forum.

This term we welcomed a panel of three independent school principals to present to our school’s leaders. These three panellists represented more than three decades of principalship between them. They had some clear messages about leadership for leaders at all levels, including the following.

……

Embrace opportunities

The panel encouraged everyone to embrace and pursue available opportunities, to take on challenges and pursue work and service that energise, inspire, and motivate us, and that align with our framework of personal beliefs and values. For me this is about aiming to do good work, without a clear vision of where this might take me.

Be yourself

All three principals said something that resonates with my own philosophy: in order to lead effectively, we need to be authentic. That is, rather than trying to perform the identity we think others are hoping for, each of us can be ourselves. Being ourselves means knowing ourselves. To be authentic leaders, we each need a clear sense of our own core values and beliefs, and a willingness to be transparent in our thinking.

Back yourself

The stories of these three principals showed that we need to be ‘in it to win it’; that is, to put our hat in the ring even when we might not be the obvious choice for a leadership position. Backing ourselves means having the courage and confidence to put our hands up to take on responsibility, and having the self-awareness to know what we bring (and don’t bring) to the work and leading we do. Part of this also means to be unafraid to challenge others or to call out injustice, and to have the capacity to be decisive even when faced with challenging issues.

Receive and give encouragement

All three principals had at some point received a ‘shoulder tap’ where a colleague or more senior leader had suggested they apply for a leadership position they had not considered. I have also had these experiences where someone has recognised for me an opportunity that I didn’t recognise for myself. These are moments that can help us to reimagine of what we are capable, and where our paths might take us. I am grateful to those who have taken the time or opportunity to challenge me on the limitations I have sometimes set for myself.

We can each listen to advice from others and be open to opportunities we may not have considered for ourselves. Each of us can also find opportunities to recognise, acknowledge, and encourage those around us; to let others know when we see leadership potential in them; and to pull others up with us, championing their work and helping them and others to see their possibilities.

……

The panel also had plenty to say about being a principal. Principalship is leadership as service that can have very real impacts on those in the role. As the results of the Australian Principal Health and Wellbeing survey show, Australian principals score lower than the general population on positive measures of wellbeing, quality of life, and mental health; but higher on negative measures such as stress, depression and sleeping trouble. Our panel discussed their own self-care strategies and the ways in which they look after themselves as they navigate what is complex, unrelenting, ethically-challenging, and often isolated work.

Our panel also noted that ‘principal’ is a leadership position that can be reached via a range of pathways. This encouragement comes at a time when Australia has a shortage of those aspiring to principalship, with a looming shortage as the majority of Australian principals reach or near retirement age.

The message from our panel was that being a principal is doable. Their stories brought a human side to the role and one panellist noted that the principalship is not a special place for an elite few but something to which many can aspire, and in which many can find success. The caveat here was that aspirant principals needed to be those with a strong values framework who is clearly aligned with the core values and mission of the school they are leading, and an ability to make decisions under pressure.

The lessons from this panel of principals are relevant for those aspiring to leadership and those already leading. Whether we have a leadership title, or are seeking opportunities to positively influence the world around us, we can be authentic and true to ourselves. We can be motivated by what energises us and by our desire to make a difference in the world. We can be courageous in our action and communication, make deliberate ethical decisions, and enact well-considered actions that are based on a solid foundation of self-awareness, self-efficacy, and self-belief.

Advertisements

What does it mean to be a leader?

leadership according to the internet

One thing that drives me mad in my social media feeds are the images that accompany articles on leadership. Infographics about leaders often feature male suited figures. An Google image search for ‘leader’ results in swarms of male figures in front of a group or standing atop a mountain. This presents a very limited notion of what a leader is or to what leaders should aspire. The men photographed or illustrated for these images of leadership tend to be white and photogenic, and wearing suits or capes. Leader as man. Leader as hero. Leader as at the apex. Leader as forging ahead.

Some of the academic writing I’ve been doing around leadership, in the form of journal articles and book chapters, has me revisiting my thinking around leadership. I’ve written before about challenging traditional notions of what a leader is and what they do. I wonder how my own approach and journey might play a part in offering alternative narratives of leadership. How does my story allow others to imagine a leader who may not be out in front, or on top, or male, or in a suit, or wearing a cape? How might leaders or aspirant leaders give themselves permission to lead differently, or to aspire to images of leadership that are different: softer, more collaborative, less visible, more joyful?

This isn’t about being a woman or a man, but about everyone being able to access a continuum of ways of being and leading. Or perhaps it isn’t a continuum but a web of possibilities, connected but divergent.

I have always lived the educational cliché – doing my very best, striving for high achievement, immersing myself in lifelong learning. Many of the leaders in my PhD study said the same: not only had they drunk the Kool-Aid of education, but they also felt its essence down to their bones. Leading, teaching, and learning aren’t add-ons or aspirations, but ways of being based on deeply held beliefs.

I have been a school leader since my first principal took a chance on me by promoting me to a Head of Faculty position in my second year of teaching. I was 22 years of age. I was tasked with leading teachers who had been teaching for more years than I had been living. My approach then, similarly to my approach now, was around building trust and relationships as the foundation stones of leadership. As Bryk and Schneider (2002) assert, relational trust is the connective tissue that binds together individuals with the common mission of advancing the education and welfare of students.

Now, my leadership style is based in an understanding of leadership literature, valuing of relationships, belief in the capacities of those I lead, and willingness to listen equally to enthusiastic perspectives and dissenting voices. My PhD and current role mean that I am a practitioner committed to I research-informed and data-rich practice. I also, however, place great value in practitioner experience, the wisdom of professional practice, and the capacity of those with whom I work, to grow, improve, and serve their students and communities.

My approach to school and cultural change is ‘go slow to go fast’. Deliberate, collaborative change coaxes buy-in and ownership from stakeholders. It involves creating a shared need, designing a shared vision, and then energising, mobilising, and building the capacities and motivations of others to propel change. This kind of leadership isn’t about me, but about how to fire holonomy (Costa & Garmston, 2015): the nuanced interactions between ‘me’ and ‘we’, individual and organisation, cog and machine. As Andy Hargreaves and Michael Fullan (2012) point out, the group is more powerful than the individual in school and system change.

The reason that I continue to blog, to edit and contribute to books, to act as a peer reviewer for journal articles, to engage at conferences and online, is because I want to be part of shaping narratives of education and leadership. It is my hope that through sharing my voice I can be part of offering alternatives and providing solutions.

I have had two children along the way, and have navigated my way through the decision-making that comes with finding ways to be a good parent, a good spouse, and to do work that I think makes a difference in the world. As a leader I am mindful of the example I set for others in the decisions I make around work, family, and wellbeing.

As a leader, I don’t aspire to embody the hero, perform the all-knowing problem-fixer, or forge ahead with innovation at a rate of knots. I aim to be my authentic self and work to empower and elevate others in what Andy Hargreaves, Alan Boyle, and Alma Harris (2014) call ‘uplifting leadership’. Sometimes leading means holding the line or being calm in the eye of a storm. It often means giving others what they need based simultaneously on a balcony view of the macro picture, and an intimate understanding of the individual.

References

Bryk, A., & Schneider, B. (2002). Trust in schools: A core resource for improvement. Russell Sage Foundation.

Costa, A. L., & Garmston, R. J. (2015). Cognitive Coaching: Developing self-directed leaders and learners. Hawker Brownlow Education.

Hargreaves, A., Boyle, A., & Harris, A. (2014). Uplifting leadership: How organisations, teams, and communities raise performance. John Wiley & Sons.

Hargreaves, A., & Fullan, M. (2012). Professional capital: Transforming teaching in every school. Teachers College Press.

Semantic space: ‘How we talk around here’

I’ve been thinking about talking and talking about talking. Last month I published this piece on my school’s blog about why and how we approach professional conversations as an organisation. The pictured infographic is one I designed to distil our school approach to professional conversations. While deceptively simple, it is the result of a lot of research, practice, and writing, over time. In that blog post, I talk about why we use Cognitive Coaching as a coaching model for developing a collaborative professional learning culture, but also when we might deliberately use consulting, collaborating, or evaluating as ways of talking. Rather than adopting deficit models of conversation aimed to fix or tell teachers, we base our professional conversations on a belief in the capacity of everyone in our community to grow and improve.

At the Australian National Coaching Conference in Melbourne last month, I was immersed in talking about coaching, and talking with coaches. I was delighted to be on a conference panel with Christian van Niewerburgh, Rachel Lofthouse, and Alex Guedes, discussing coaching in education research. One of the points I made was around the use of terminology within a community like the one in the conference room. 400 educators and coaches were all talking about coaching at the conference, but not necessarily with the same understanding of what ‘coaching’ means.

As Scott Millman points out in his blog, many of the conference attendees had what Christian van Niewerburgh calls a ‘coaching way of being’. A conversation with them is a coaching conversation. These individuals actively and intensely listen, paraphrase, pause, and ask thoughtful questions designed more for the benefit of the talker than the listener. These aren’t conversations where the other person is waiting for their turn to say their piece or pushing a personal agenda; they are ones in which the listener serves the talker via thoughtful and deliberate ways of talking and ways of being in conversation.

At the conference, Rachel Lofthouse talked about Kemmis and Heikkenen’s (2012) notion of a semantic space. I enjoyed this way of thinking about an organisation. Semantics is about linguistic meaning; the logic of language. In organisations I imagine a semantic space is about ‘how we talk around here’, the meanings of words, the way communication happens. Lofthouse and Elaine Hall (2014) define semantic space as one of professional dialogue, constituting tone, choice of words, routines of dialogue, and balance of participation in conversation. Semantic space interacts with organisional structures, physical spaces, and relationships.

Harvard academics and developmental psychologists Robert Kegan and Lisa Lahey (2001) say that our places of work are places in which certain forms of speech are promoted or encouraged, and places where other ways of talking are discouraged or made impossible. What kinds of talk are promoted in our schools? Which are limited or suppressed?

Can it ever be ‘just’ semantics? No. The words we use, the way we talk, and the way we interpret language are vital to our work, especially in education. Members of high-functioning teams, for instance, respectfully challenge one another in order to find the best ways forward. Something Rachel Lofthouse said in her conference keynote stuck with me: “Don’t talk less and work more. The talk is the work. So talk well, talk productively, talk to learn.” The way we talk can influence the way we think and the way we behave. In any organisation it’s important to figure out and work on ‘how we talk around here’ as well as why we talk, when we talk, what we talk about, and how we want to talk.

References

Kegan, R., & Lahey, L. L. (2001). How the way we talk can change the way we work: Seven languages for transformation. John Wiley & Sons.

Kemmis, S. and Heikkinen, H.L.T. (2012). Practice architectures and teacher induction. In: H.L.T. Heikkinen, H. Jokinen, and P. Tynjälä, eds. Peer-group mentoring (PGM): peer group mentoring for teachers’ professional development. London: Routledge, 144–170.

Lofthouse, R., & Hall, E. (2014). Developing practices in teachers’ professional dialogue in england: Using coaching dimensions as an epistemic tool. Professional Development in Education, 40(5), 758-778.

Reflecting on the school leader

The bad leader is he who the people despise; the good leader is he who the people praise; the great leader is he who the people say, “We did it ourselves”. ~ Peter Senge, The Fifth Discipline

Part of my PhD literature review encompassed what makes effective school leadership, and effective leadership of change or reform in schools. If you don’t fancy savouring all 300 odd pages of my dissertation, the summary of my literature search determined that effective school leaders:

  • Develop shared vision;
  • Have high expectations and clear accountabilities;
  • Develop an environment of trust;
  • Empower others and allow them autonomy, space, and support to lead;
  • Solve complex problems;
  • Engage with the wider community; act as storyteller and sense-maker; and
  • Balance instructional and transformational leadership.

Many of these points are reflected in the Australian Professional Standards for Principals, which break school leadership down into the following components:

  • Leading teaching and learning;
  • Developing self and others;
  • Leading improvement, innovation, and change;
  • Leading the management of the school; and
  • Engaging and working with the community.

Both of these lists cross over one another, and each seems simple in its short-list nature (5 dot points! How hard can it be?), but looking closely at many of these aspects of school leadership quickly reveals the complexity of the mandate. On top of that, school leadership teams are under pressure from constant measures of their performance. Leadership itself becomes a quantified, evaluated performance. Meanwhile, on a daily basis leaders constantly code-switch as they move from the classroom, to the boardroom, to the parents’ committee, to the community event, to the performance management conversation, to the staff member or student who needs support.

My PhD study found that school leaders are constantly navigating internal, relational, and organisational identities. These complex and sometimes competing identities affect leaders’ experiences and decision making. The leaders in my study were moving, often deliberately and relentlessly, between leadership modes that were directive and empowering, hero and servant, visible and invisible.

Leading is a constant state of becoming and of identity work. Peter Gronn, in his 2003 book The new work of educational leaders: Changing leadership practice in an era of school reform, reminds us that leaders’ senses of who they are, and who they aspire to be, play a pivotal role in their engagement with their work. Having multiple leadership roles in my current school has meant that it is not only me who has had to shift my self-perceptions or identity enactments, but also my colleagues who have had to see me in new ways across my time at the school. Additionally, I have multiple, competing identities that exist simultaneously with my school identity; as parent, spouse, sibling, daughter, researcher. Boundary spanner and pracademic. Identities like plates precariously spinning atop spidery poles.

While Gronn suggests that individuals rework their perspectives in relation to their contexts, my PhD found that, while context does shape professional identity, individuals also choose their contexts to fit their own identities. My leader participants indicated that they stayed in schools that resonated with their senses of professional self, and left schools in which they did not feel aligned with organisational purpose and action. That is, school contexts shape leaders, and leaders shape their contexts. Leaders can and do choose schools with which they feel an identity fit, and leave schools in which they feel they do not fit.

Wellbeing is a real issue in school leadership, as reflected in the results of the longitudinal Riley study, which has found that Australian principals score lower than the general population on positive measures of wellbeing, quality of life, and mental health; but higher on negative measures such as stress, depression and sleeping trouble. Leaders need formal and informal support, as well as their own strategies for self-care and renewal. It might be that school leadership can be summarised in a series of dot points, but it is contextual, complex, and lonely. It is challenging and rewarding, exhausting and exhilarating.

Running the PISA race

The education culture of performativity is wrapped up in notions of measurement. How do we measure student success, teacher performance, effective school leadership, and successful education policy? How do we know which school systems are successful and why they are successful? How can we tease out and understand causes of performance, such as the influence of social and economic factors, the system, the school, and the teacher? These are perennial education questions, and ones which continue to become more and more important in a globalised world in which countries, schools, and teachers can be compared, and in which there is an ever-increasing weight of accountability.

Standardised testing is a central issue in this neoliberal education context. Individual schools have their own approaches to measurement, such as the one I describe in this blog post. In Australia we have NAPLAN and WACE. There is currently talk of a national Phonics Check in the early years, such as that used in the UK. Internationally we have TIMSS, PIRLS and PISA.

Steven Lewis and Anna Hogan have shown how oversimplified reports of international testing measures can contribute to oversimplified ‘fast policy’. As Marten Koomen points out, systems should respond to international testing measures but these systems are complex. Stewart Riddle and Bob Lingard wrote that looking at a single country’s PISA ranking is useless; rather we need to carefully disaggregate the data and consider social and economic factors, and differences between states, schools and groups.

This week I received in the mail a copy of new book The global education race: Taking the measure of PISA and international testing, by Sam Sellar, Greg Thompson and David Rutkowski. (Full disclosure: I got a complimentary copy.) It is a pocket rocket at about A5 size and 99 pages. In the Foreword, David Berliner and Pasi Sahlberg question the value often ascribed to PISA, noting that PISA tests are linked to the social conditions as well as school systems and that “when the race to the top gets tougher … curriculum narrows and children suffer” (p.ix).

The book demystifies the workings of PISA, using the extended metaphor of the ‘race’ as a way to make sense of what PISA is, how it works, how it might be used, and how it shouldn’t be used. The authors make their explanations accessible without ignoring the complexities of standardised international testing on a large scale. The book is at once matter of fact and told with a wry sense of humour (as a reader I’m a sucker for references to Monty Python and 1980s arcade games). The book rails against the commodification of simple solutions to solve complex problems. As I explored satirically in this He-Man inspired post (speaking of 80s references) there are plenty of edu salespeople hoping to profit from the pressures of accountability pressing in on schools and teachers. This book, however, is about helping those on the ground to understand the complexities, inner workings and possibilities of PISA.

What sets Sellar, Thompson and Rutkowski’s book apart from other literature is its ability to engage with complexity in an accessible way; to explain clearly without simplifying; and to avoid binaries and polarising divides while acknowledging differing perspectives. They note, for instance, that PISA has been innovative, carefully developed, and judiciously administered, but that such a big project will undoubtably suffer from technical issues and limitations.  They note that PISA is an assessment of select content areas of one sample on one day, but also argue that transparent standardised tests like PISA can be a useful tool for understanding social systems. They explain validity in understandable ways and show how countries can use PISA data responsibly.

The international examples help the book to be relevant to people in OECD countries around the world. For Australian readers like myself there are some gems, such as that PISA performance has become an end in itself, evidenced by the national target of improving Australia’s PISA ranking by 2025 (articulated in the 2013 Education Act). The authors call this move “astounding”. Their tongues are firmly in their cheeks when they state on page 76 that: “The aim of reversing the trend of declining PISA scores seems to be to improve PISA scores through intensifying those policies that have not worked so far. A bold move.” A bold move, indeed, and one that Simon Birmingham might want to reconsider.

The authors also recognise the desire of policy makers, educators and the media to understand PISA and to glean the most important messages it can tell us. They warn, however, about over-attribution of causality, when correlations become causal claims, pointing to the way the media and policy makers often use PISA to point to particular factors as being the cause of high or declining PISA performance. I’ve written about the dangers of policy moves like performance pay for teachers, and the authors have warnings to offer their readers about the negative effects of pairing standardised testing with punitive accountability regimes, and of governments desperately scrambling to ‘win’ against other countries. Run your own race, they argue.

The global education race presents an important challenge to policy makers and educators alike: to develop informed communities willing and able to engage in discussion of how educational measurement, including system-level measurement tools, can be judiciously used to inform policy and practice. The kind of shift they suggest is a challenge. It will require ministers, media and educators to take a non-divisive, sense-making and collaborative approach of seeking to understand, and of deep, thoughtful engagement with data and with one another.

E4L and the value of dissent

I find it ironic that, just after a blog post in which I reflected that blogging often feels like shouting into the void, a recent post on this blog has received a robust and ongoing response, as well as plenty of rich conversation, online and in my immediate context.

I wrote earlier this month about my ponderings and cautions around the Evidence for Learning Toolkit (based heavily on the UK’s Education Endowment Fund Toolkit) currently gaining traction in Australian education circles. I felt compelled to share my thinking as the E4L Toolkit is connected with some Australian heavy hitters. It is powered by Social Ventures Australia and the Commonwealth Bank. It is advised by prominent academics from a range of Australian universities, including John Hattie who is both a champion of meta-analysis and the equivalent of an education household name. Its advisors also include high-level representatives from the Australian Council for Educational Research (ACER), the Australian Council of Education Leaders (ACEL), the Australian Curriculum, Assessment and Reporting Authority (ACARA), various independent schools associations, and various State education departments. This who’s-who is all the more reason, I think, for those in the education community to look carefully at the Toolkit and its adoption across the Australian edu-landscape.

This week, John Bush, Associate Director of Education at Social Ventures Australia and part of the leadership team of Evidence for Learning, wrote a blog post for the E4L website (the first of a series). In it, he responds specifically to some of the comments I made in my post.

John points out that my post was part of “a flurry of public comment across Australia and the UK in the value and risks of meta-analysis and synthesis of meta-analyses in education research.” Although I didn’t realise it at the time, my post did come in on the crest of a wave. Published the same day (UK time) were this article in the Times Education Supplement in which Professors Terry Wrigley and Gert Biesta call the EEF Teaching and Learning Toolkit misleading and unhelpful. There was also this response by the EEF addressing the concerns around and risks of its Toolkit.

A few days later, Rachel Buchanan posted about the need for a critical perspective when we talk about using an evidence base in education. I was also made aware
(thank you, Twitter) of this post from David Didau from last month, which questioned the objectivity of the EEF Toolkit. He has previously cogitated on the limitations of meta-analysis and effect sizes as measures in education.

In my recent post, I linked to a couple of other thoughtful blogs on the limitations of meta-analysis in educational research, one by Jon Andrews and one by Gary Jones. It’s also well worth listening to both Dan Haesler’s comments, and Cameron Malcher’s interview with E4L’s Tanya Vaughan and Janet Clinton, on this recent episode of the Teachers’ Education Review podcast.

So, amid this flurry, I was delighted when John Bush began his own considered response within a complex field: the use of evidence to inform teaching, in order to most positively impact on student learning and achievement.

Despite John’s explanations of the padlock rating system, bearing in mind the many reservations about meta-analysis, I’m not sure I’m confident with E4L’s five-padlock ‘security’ of interventions that are supported by “at least five robust and recent meta-analyses.” I was, however, relieved to read that E4L deliberately uses careful language in the Toolkit in order to facilitate nuanced understandings of the evidence it presents.

John writes that, “We do not envision the Toolkit as a resource that should dictate or direct professional decisions in schools. Instead, we hope school leaders and teachers will use it to start discussions with their peers and to help inform their professional judgement with research evidence.” These are important words, but we in education remain challenged by the tension between a desire for the simple fix and the reality of the complexities of our work. Careful language and a critical approach are important, but the appeal of the misleading media headline, the ranking on a league table, and the easy answer, remain. As a profession we need to push back against these, despite the performative culture in which we operate.

This ongoing public conversation reminded me of a podcast from Brand Newsroom in which the hosts Nic Hayes and Sarah Mitchell interview influential marketing consultant Jay Baer about his book titled Hug Your Haters. In it, Jay discusses the science of complaints. He says that praise is over-rated. Praise feels terrific, he tells us, but teaches us nothing. Negative feedback and criticism, on the other hand, teach us everything; it’s where learning and opportunity come from. Now, I wouldn’t classify myself as a hater, or a complainer, but I would probably be seen by those championing E4L Toolkit as a dissenting voice.

Being publically challenged, even gently, can be difficult, especially when we believe strongly that we are doing good work. I respect John Bush, and the folk at Evidence for Learning, for having the transparency and commitment to engage in some graceful disagreement. In doing so they are responding directly to the concerns of those like me, who, as it turns out, wasn’t blogging into a void after all. Rather, I was offering my perspective to a community of educators who seek to understand one another in order to best serve our students.

While I have moments of high cynicism and outright despair, it is blogversations like the one in which John and I are currently engaging, that can help model publically the kinds of conversations educators can and should be having. I remain cautious and critical about the Evidence for Learning Toolkit, especially in terms of the ways in which it might be adopted by educators in the busyness of their work, but I am open to developing my thinking and continuing the conversation.

Evidence For Learning in Australia

In the UK the Education Endowment Foundation (EEF) is championed by some as a tool for helping teachers, school leaders and schools to make the best decisions for their students, based on what research and evidence shows. Now in Australia, Evidence for Learning (E4L), powered by Social Ventures Australia and the Commonwealth Bank of Australia, is piggybacking on the EEF’s toolkit in order to provide an Australasian equivalent. It is advised by, among others, John Hattie, and is partnering with AITSL and with State education departments to map the toolkit to State education frameworks and the AISTL Professional Standards for Teachers and Principals.

Last year I spoke with John Bush, Associate Director of the Learning Impact Fund, about the toolkit, and this week I attended a breakfast workshop run by Dr Tanya Vaughan, Associate Director for the E4L toolkit and Honorary Fellow at the Melbourne Graduate School of Education (MGSE) at the University of Melbourne. As the Research Lead at my Australian school, I was keen to hear more about how it was progressing and what it is offering Australian schools.

The aims of the E4L Toolkit

Tanya framed the toolkit as as an instrument for helping great practice become common practice. E4L aspires to make accessible, and develop the rigour of, evidence of what works and why in education, in order to make a difference to learners. That is, it aims to build, share and use evidence to support better decision-making in schools, to in turn lead to better outcomes for students.

The E4L toolkit is free and unrestricted in order to provide all schools with access to evidence of what works best in education, regardless of budget or postcode. This, Tanya explained, will help to address the barriers for teachers engaging with research:

  • Shortage of time;
  • Overload of information; and
  • Insufficient contextualized information for practice.

I would add that much educational research is behind a pay wall in journals inaccessible to non-researchers, or in very expensive books that aren’t affordable for many schools. Tanya was adamant that “front line professionals are the heart and soul of evidence-based education practice”, and that E4L endeavoured to improve communication between professionals and researchers, teachers and ‘the evidence’. This connection between educational research and practice is one to which I am especially committed.

What does the E4L Toolkit look like?

The E4L effect size league table’s Top 5 edu-practices

A first glance, the E4L toolkit shows a set of effect-size league tables of teaching practices, each showing – via symbols – the average cost of implementation, the ‘evidence security’ of the claim, and the average month’s worth of learning impact.

Visitors to the toolkit can drill down into the site. Clicking on a single practice such as ‘feedback’ reveals summaries addressing the following questions: What is it?; How effective is it?; How secure is the evidence?; What are the costs?; and, What should I consider? Clicking further into ‘References’ reveals the studies that sit behind this practice, with abstracts. Some practices additionally have an Australasian research summary.

Tanya was clear that the toolkit presents averages. In fact, it presents averages of averages, or more accurately meta-meta-analyses. While Tanya advocated for mixed methods – including talking to leaders, teachers and students – most of what the toolkit presents are syntheses of meta-analyses and randomised control trials (often considered the ‘gold standard’ of educational research).

The lock rating symbols, showing apparent ‘security of evidence’ are based on the number of meta-analyses beneath the meta-meta-analysis. It is the notion of evidence security and the simplification of ‘what works’ to effect size league tables that has me feeling cautious about the toolkit and its potential use. In attempting to address education practitioners’ shortage of time to engage with research and the overload of research information out there, does E4L provide an oversimplified tool likely to be accepted uncritically by busy educators working in our schools?

What is meta-analysis?

Meta-analysis is a statistical analysis using an equation: the experimental mean, minus the control group mean, divided by the population standard deviation. Simpson (2017) gives us this description of what happens:

“Individual studies report quantitative measures of the outcomes of particular interventions; meta-analysts collect studies in a given area, convert outcome measures to a common metric and combine those to report an estimate which they claim represents the impact or influence of interventions in that area. Meta-meta-analysis then takes the results of meta-analyses, collected in broader fields, and combines those estimates to provide a rank ordering of those fields which make the most difference.”

Simpson’s paper, released in January this year, challenges analogies between evidence-based practice in medicine and education. Treatments in medicine, he argues, are often standard and well-specified, with agreed outcomes which are relatively easy to measure. Education is more nuanced, complex and contextual.

Simpson invokes Eysenck’s (1984) notion of comparing apples with oranges, when he points out that meta-analyses often do not compare studies with the same comparisons, measures and ranges of participants. He contends that aggregated effect sizes are more likely to show differences in research design manipulation than in effects on learners. Bloggers such as Jon Andrews, in this post, and Gary Jones, in this one, have teased out the limitations of meta-analysis as method in educational research. Gary insists that “if teachers and school leaders wish to use effect sizes generated by research to help prioritise interventions, then it is necessary to look at the original research”, rather than relying on simplified lists. Educators need to look behind the curtain.

Snook et al. (2009) argue that when averages are sought or large numbers of disparate studies amalgamated, as in meta-analyses, the complexity of education and of classrooms can be overlooked.  They also point out that any meta-analysis that does not exclude poor or inadequate studies is misleading or potentially damaging. Terhart (2011) points out that by focusing on quantifiable measures of student performance, meta-analyses ignore the broader goals of education.

Meta-analysis is singled out by Wiliam (2016) as an unsuitable technique for identifying the relative effectiveness of different approaches to student learning. He states that:

Meta-analysis is simply incapable of yielding meaningful findings that leaders can use to direct the activities of the teachers they lead.”

Wiliam’s PowerPoint presentation from last year’s ResearchED conference in Washington—titled ‘Why teaching isn’t—and probably never will be—a research-based profession (and why that’s a good thing)’—presents the problems with meta-analyses for deciding ‘what works’ in education. In the presentation, Wiliam reminds us that everything works somewhere and nothing works everywhere. He encourages us instead to ask: Under what conditions does this work?

Possibilities and reservations

In her E4L Toolkit presentation this week, Tanya Vaughan advocated for trusting the profession to be thoughtful and intelligent and to engage with the research literature that sits behind the seductive league tables of the E4L toolkit. Her call for mixed methods research—for qualitative and quantitative to “play together”—resonated with me. Many methods of research have something to offer the field, and all are limited.

My hunch is that the E4L toolkit has something to offer educators in Australia (as a starting point rather than an answer sheet), and I can see the significant work that has gone into producing it, as well as the good intentions behind it. Yet I have my reservations. I worry that an uncritical acceptance of the toolkit’s content, alluring in its apparent simplicity, will result in an impoverished understanding of ‘what research says’. We are in danger of giving education research lip service, or wading in shallow pools of evidence. The use of meta-meta-analyses as the basis for the toolkit has the potential to over-synthesise limited quantitative data to the point of distorting original findings, and ignore the limitations, qualities and complexities of the synthesised studies.

Everyone from the profession to the media is likely to translate these effect-size league tables into seemingly authoritative soundbites of ‘what works’ without taking the time to consider what might work where, for whom, and under what conditions. If Australian organisations and schools are to embrace the E4L Toolkit as part of their pursuit of having a positive impact on learners and more systematic bases on which to make decisions, I hope they do so with a cautious step and a critical eye.

References

Eysenck, H. J. (1984). Meta-analysis: An abuse of research integration. The Journal of Special Education 18(1), 41–59.

Simpson, A. (2017). The misdirection of public policy: Comparing and combining standardised effect sizes. Journal of Education Policy, 1-17.

Snook, I., O’Neill, J., Clark, J., O’Neill, A. M., & Openshaw, R. (2009). Invisible learnings? A commentary on John Hattie’s book: Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New Zealand Journal of Educational Studies, 44(1), 93-106.

Terhart, E. (2011). Has John Hattie really found the holy grail of research on teaching? An extended review of Visible Learning. Journal of Curriculum Studies, 43(3), 425-438.

Wiliam, D. (2016). Leadership for teacher learning: Creating a culture where all teachers improve so that all students succeed. Moorabbin, Australia: Hawker Brownlow Education.