Beyond the Algorithm: Safety by Design as an Act of Moral Leadership
This article is adapted from a keynote address I delivered to the Safety by Design Conference, Sydney, 16 June 2026.
I’ve spent almost fifteen years in online content regulation, first as an investigator, then building teams and regulatory operations at a senior level.
I was fortunate to serve with the eSafety Commissioner, Julie Inman Grant, as the early shapes of Safety by Design were being sketched.
What I want to do here is outline what I think a leadership operating model for Safety by Design actually looks like in practice.
Many will be familiar with the eSafety Commissioner’s three Safety by Design principles: service provider responsibility, user empowerment and autonomy, and transparency and accountability.
What I want to do is go behind those principles and ask a question:
What does an organisation have to do — culturally, operationally, at a leadership level — to make them real?
Because without that kind of commitment, Safety by Design risks being so much virtuous marketing hype.
It’s worth establishing why this question is pressing right now.
Last month, the Australian Government released its Digital Duty of Care Issues Paper, which is the most substantive attempt yet to codify platform responsibility into Australian law.
What struck me most about the Issues Paper was that it didn’t just gesture towards Safety by Design as best practice. It placed the three eSafety principles at the centre of the expected regulatory architecture. Safety by Design is becoming the core logic of how Australia will hold platforms legally accountable.
The urgent bit is that we know that abuse of online services across all kinds of domains — social media, banking and finance, utilities, telecommunications, insurance — can be a key indicator of lethal domestic violence.
Coercive control is significantly enabled through abuse of these services.
The operating model I want to outline goes beyond mere compliance, to the heart of the organisation and its fundamental ethos. Accountability through Safety by Design expresses why a service exists and whether it actually serves its users, or exploits them.
The model has three practices: identifying the right problems, surfacing expertise from within the organisation, and leading through values, or what I’ll call moral leadership.
Each builds on the last.
Part One: Identifying the Right Problem
The first practice is about foundations. Before you can design for safety, before you can surface expertise or exercise moral leadership, you have to be working on the right problem.
And the uncomfortable truth is that the data you have, the reports you receive, the patterns you can see are almost never the full picture.
I want to walk through four forms of bias that can derail otherwise well-intentioned Safety by Design efforts: selection bias, survivorship bias, the streetlight effect, and the availability heuristic.
Selection Bias
Selection bias, in this context, is what happens when the filter you’ve chosen to collect data through systematically excludes certain kinds of harm.
For online services, that filter is almost always a reporting flow. This can take the form of an in-app report button, a set of trust and safety policies, a natural language processing algorithm scanning for keywords, a requests portal for law enforcement or regulators.
Whatever the flow, it produces data.
But it only captures harms reported by people who are capable of reporting, who trust the mechanism enough to use it, and whose concerns are actually represented by the coded logic of the system.
The same problem shows up in who you choose to test your systems. Platforms typically recruit ‘power users’, and the feedback they get reflects the skills, education, and technical confidence within that cohort.
That’s a skewed sample.
The people most at risk of harm are often the least well-represented in your testing data.
The implications run deeper than data quality. If your reporting infrastructure is built around users who are confident, technically literate, and trusting of institutional mechanisms, it will render invisible precisely those users who most need protection.
These are the users who don’t know how to report, don’t trust that reporting will help, or whose experience of harm isn’t captured by the categories your system was designed to recognise.
Survivorship Bias
A close relative is survivorship bias, which is the mistake of treating those who remain visible as if they represent the whole picture, while overlooking those who’ve disappeared from view.
This is about surviving data, which unfortunately often omits the experiences of the people who matter most — the survivors who don’t make it through the selection filter at all.
The best-known illustration comes from World War Two.
An artist’s impression of hit patterns, based on Wald’s analysis. Source: Wikipedia
The Army Air Corps was losing bombers and crews at an unsustainable rate and turned to a group of mathematicians working with the Statistical Research Group at Columbia University for help. A statistician named Abraham Wald helped to lead the work.
The Corps showed Wald exactly where on the airframes of returning aircraft enemy fire had hit. Their question: where should we add armour plating?
Too much plating and the flight characteristics of the aircraft would be impaired; too little and the exercise would have been futile.
The logical problem is easy to see once it’s named. The data only showed damage on aircraft that made it back. Wald argued the right question was where the damage on non-surviving aircraft was likely to fall. The place to look was on the areas left undamaged on the planes that survived. Strengthening what was already resilient would do nothing. It was the engines – conspicuously undamaged on returning planes — that were most likely to be fatal when hit.
The lesson: the data you have is derived from and shaped by those who got to tell you about it.
In an online safety context, that means asking not just ‘what are we seeing?’ but ‘what aren’t we seeing, and why?’
The Streetlight Effect
The streetlight effect is what happens when selection and survivorship bias ossify into institutional habit.
We start measuring what is measurable, instrumenting what is instrumentable. We then assume the resulting data is the full picture. We look for the truth where truth-seeking is easy, rather than where the truth actually lies.
Sociologist William Bruce Cameron put it well in 1963:
“Not everything that can be counted counts, and not everything that counts can be counted.”
Organisational ecologist Tom Geraghty frames the institutional stakes even more directly:
“Who gets to decide what’s important? When we flatten people’s lived realities into metrics, who decides what we quantify and what we disregard? Who has that institutional power? Measurement choices are about power. They determine whose experience is rendered visible, whose risks are legitimised, and whose realities remain structurally invisible.”
The Availability Heuristic
The fourth form of bias is the availability heuristic, which can operate at the level of entire institutions, not just individual analysts. It is the tendency to prioritise our response based on what is most recent, most vivid, most viscerally present in our consciousness. The events that dominate our attention aren’t necessarily the most significant. They’re simply the ones we can most easily call to mind.
I want to be clear about what I am and am not saying when I use the availability heuristic as a lens on the policy response to the Christchurch terror attack. I’m not saying the legislative response was wrong.
I’m saying it’s a near-perfect illustration of how a traumatic event can reshape an entire government’s sense of legislative priority — and in doing so, displace some of the harder, slower, less photogenic work of systemic reform.
On March 15, 2019, a gunman live-streamed the murder of fifty-one people at two mosques in Christchurch on Facebook. The footage spread rapidly across platforms. The horror was immediate, global, and indelible.
The Australian Government’s response was swift. Its instinct was to be seen as doing something, and so it enacted muscular amendments to the Commonwealth Criminal Code. This included criminal indictments for executives of services that don’t expeditiously remove abhorrent violent material once notified. That was a defensible response to what the world had just witnessed.
But the deeper questions — about hate, violent radicalisation, the role of recommender systems in producing extremism, the part played by sites such as 8chan and 4chan, the failures of platform responsibility that allowed the live-stream to spread — required sustained, systemic work that couldn’t be accomplished in the news cycle in which they were conceived. Those questions were, more or less, left unaddressed.
That is the availability heuristic at work in policy. The most vivid harm commanded the most visible response. The structural conditions that produced it remained in the dark.
For trust and safety teams, the lesson is the same: the harm you can see most clearly is not necessarily the harm doing the most damage.
Part Two: Listening to the Internal Experts
Knowing what you don’t know, and understanding why you don’t know it, is the necessary precondition for the second practice. Even when organisations have addressed the data problem there is a second failure mode that is just as common and just as costly.
The people who can see the right problem often cannot get the right people to listen.
I want to tell two stories.
Both are about safety leaders who raised serious concerns about what they were seeing, and were ignored. The first concerns a technical control. The second concerns an informational one.
The Challenger disaster
Cape Canaveral rarely sees the temperature drop below freezing. But on January 28 1986 it was so cold that the air itself seemed to freeze. Icicles draped from the booster rockets. Great clouds of steam engulfed the Challenger as teams finalised last-minute checks.
On board was a crew of seven, including for the first time a schoolteacher, Christa McAuliffe. In her crewbag, she carried a t-shirt: “I touch the future. I teach.”
The Shuttle was lifted into orbit by two solid rocket boosters running alongside it, jettisoned in sections as their fuel was exhausted. Each section was sealed from the next by rubber O-rings — flexible rings designed to absorb the colossal forces of takeoff and prevent gas escaping between the joints.
A diagram showing the sections of Challenger’s solid rocket booster. Source: NASA https://www.nasa.gov/history/rogersrep/v4p109.htm
Since 1977, engineers had been warning that the O-rings were problematic. Cold made the rings rigid.
The coldest previous launch day had been 12 degrees. After that launch, which involved Space Shuttle Discovery, the O-rings had partially failed. Warnings were made by engineers, alarmed at the degradation they found after components were recovered. They were ignored.
One engineer working for Thiokol, the company that made the O-rings, was especially persistent. His name was Roger Boisjoly.
Six months before the Challenger launch, he wrote to Thiokol’s VP of Engineering:
“It is my honest and very real fear that if we do not take immediate action… we stand in jeopardy of losing a flight along with all the launch pad facilities.”
He warned of a catastrophe, even loss of human life.
The day before the launch, sub-zero temperatures forecast. Deeply concerned about the weather, Thiokol engineers got one last meeting with NASA to push for a delay. NASA resisted hard.
One official snarled: “When do you want me to launch, next April?”
In a private briefing, Thiokol’s general manager Jerry Mason said: “We need to make a management decision.”
Mason then turned to his head of engineering and told him to take off his engineering hat and put on his management hat.
Boisjoly’s account of what followed the next day is devastating:
“It was approximately five minutes prior to the launch as I was walking past the room used to view launches when Bob Ebeling stepped out to encourage me to enter and watch.
At first I refused, but he finally persuaded me. The room was filled, so I seated myself on the floor closest to the screen and leaned against Bob’s legs as he was seated in a chair.
The boosters ignited, and as the vehicle cleared the tower Bob whispered to me that we had just dodged a bullet. At approximately T+60 seconds Bob told me that he had just completed a prayer of thanks to the Lord for a successful launch.
Just 13 seconds later we both saw the horror of destruction as the vehicle exploded. We all sat in stunned silence for a short time, then I got up and left the room and went directly to my office, where I remained the rest of the day.
Two of my colleagues inquired at my office to see if I was okay, but I was unable to speak to them and hold back my emotions, so I just nodded yes to them and they left after a short silent stay.”
All seven crew were lost.
Twitter and January 6
Anika Collier Navaroli was a lawyer and journalist by training, and in 2020 was Twitter’s most senior expert on its US Safety Policy Team. Her team wrote and enforced Twitter’s content moderation policies, which were the final word when no algorithm or human moderator could make a call.
As she put it: every day, her team had to decide whether a piece of content equated to yelling ‘fire’ in a crowded theatre.
In 2020, Navaroli and her colleagues identified a growing and coordinated use of ideologically coded ‘dog-whistle’ language by violent extremists. They wrote a ‘Coded Incitement to Violence’ policy to identify and remove phrases designed to sidestep existing guardrails.
Of particular concern were terms associated with far-right extremists. These included ‘The Day of the Rope’, a reference to mass executions drawn from the Turner Diaries, a neo-Nazi novel — more like a handbook, really — providing a blueprint for overthrowing the US government and initiating a race war. Other surfaced terms included ‘1776’, ‘locked and loaded’, and ‘civil war part two’.
Crowds gather outside the Capitol Building on January 6 2021. Source: Getty.
In November 2020, Navaroli’s team met with Twitter senior management, including the VP of Trust and Safety. They shared their findings and read live tweets, aloud.
Navaroli even asked senior leaders whether they needed someone to get hurt before they could be convinced of the threat.
Twitter’s leadership engaged devil’s advocate arguments in response. ‘Locked and loaded’ could refer to someone defending their home. Maybe they were getting ready to shotgun a beer?
Navaroli described the response as “magical thinking” — a willingness to not see what was right before their eyes, depending on what best suited a conclusion that created the least controversy for the company.
The policy was neutralised. Worse, actually: Twitter confirmed it had never been rolled out.
Shortly before January 6, an exasperated Navaroli bypassed her management line and wrote directly to Twitter’s legal team:
“I feel like I’ve been waving the red flag on this real potential for extreme violence for a while, and I don’t know what else to do.”
The response confirmed the platform had no monitoring in place for January 6 and no protocols to respond to violence if it occurred.
The following day, a violent mob stormed the Capitol, injuring police, hunting Vice President Pence, and attempting to overturn a democratic election. A protester was shot and killed.
Three Safety by Design Lessons
There is almost certainly someone (or a team) working in your organisation right now who can see something others can’t. Perhaps it’s you. Perhaps it’s one of your colleagues — someone in complaints, investigations, trust and safety, quality assurance, or risk. They know the data in a way the leadership does not.
In both cases, Boisjoly’s and Navaroli’s expertise was dismissed in favour of administratively convenient decisions by leaders who’d put on their management hats.
Here are three Safety by Design lessons I think you can draw from those cases.
First, the expertise must exist. Effective harm prediction requires people deeply versed in how harms are perpetrated. These are the people who can decode language, intent, and methodology in context. In practice, this means investing in people whose professional and lived experience has included actual exposure to harm: investigators, former law enforcement, survivor advocates, forensic psychologists, social workers.
Survivor consultation matters enormously here. This is because those who have been directly harmed can provide immensely valuable insight into the design of features, tools, and processes that no amount of data can substitute for.
All of this means resisting the pressure to staff trust and safety primarily with generalist policy analysts. It means keeping that expertise current. Perpetration methodologies evolve, language codes shift, new platforms generate new attack surfaces. It means developing people, developing teams, developing quintessentially human capabilities.
The expertise that serves you today has a shelf life without ongoing investment.
As Navaroli put it during her Congressional testimony:
“We need to create space to hear from people on the front lines. We need to give them protection so they can share their experiences. Only then can we begin to understand the full scope of the problem.”
Second, there must be formalised pathways for that expertise to reach decision-makers — and to be understood by them.
This isn’t just about org chart proximity.
As we’ve seen, Navaroli’s team didn’t fail to communicate. Far from it: they sat in a room with Twitter’s VP of Trust and Safety and read out live tweets calling for insurrection. The failure lay with the institutional reaction, which was predicated on the implicit question of what’s the minimum action that keeps us out of controversy.
Practically, this means designing for decision-maker comprehension, not just information transfer. Operational intelligence needs to be translated into board-level, strategic risk language. It requires escalation protocols that specify not just who receives a report, but what they’re required to do with it, and by when.
It also means investing in the methods that reveal what formal reporting channels miss.
Two methods are especially useful:
Red-teaming, which involves gaming out how a perpetrator would subvert your systems. (As an ex-armed robbery detective, I can tell you that the countless hours I spent with my partners planning the perfect armed robbery while undertaking long surveillance shifts was time exceptionally well spent.)
Pre-mortems, where a team assumes a project has catastrophically failed and works backwards through what went wrong.
Both of these aapproaches create psychological safety to surface what people already suspect might be true but haven’t been able to say.
This is important, as organisational cultures often promote the idea that we should all be ‘problem-solvers’ and ‘solutions-oriented’.
Often, these diktats produce a forced, organisation-wide, optimism. A culture of synthetic cheerfulness where healthy scepticism and doubt are unfairly labelled ‘pessimism’ or ‘negativity’.
Third, information must be acted upon within an accountability cycle. This is not the classic risk management logic where a risk is identified, treated, and put into cryofreeze until the next annual review. What I am referring to here is active risk management planning, where risks and treatments produce operational blueprints for change, and outcomes are evaluated for impact.
Under the Digital Duty of Care framework being developed in Australia, that kind of documented accountability cycle is precisely the evidence that will distinguish platforms exercising genuine duty of care from those merely performing compliance.
Part Three: Acting to Change — Moral Leadership in Action
These three lessons — that expertise must exist, that it must surface through formalised pathways, and that it must be acted upon —require something more before they can be realised in an organisation.
Good process alone doesn’t get you there. Neither does goodwill. Process can be gamed, sidestepped, or abandoned when it becomes inconvenient. Goodwill can be transient and sometimes fragile.
What makes these practices material, instead of rhetorical, is leadership that comes from conviction. That’s the third practice in my proposed operating model.
Moral leadership is driven by values. It starts with a question. Not what do we do, not how do we do it, but why do we do it? Simon Sinek observes that this why requires those who know how, which is the exact relationship between moral leadership and the first two practices in this model.
Here’s a distinction worth sitting with. Imagine a platform that builds Safety by Design systems because it faces harsh sanctions if it doesn’t. Now imagine a platform that does the same because preventing harm is incompatible with its values and ethos.
The first company is constrained by values. The second is constituted by them. That difference matters to everything they do: How quickly safety features are built, how seriously expert advice is taken, how the organisation behaves when no-one is watching.
Greenleaf’s 1970 ‘servant leadership’ model is useful here. Its central proposition is that the leader’s primary obligation runs not upward to shareholders, but outward and downward, to those they serve.
Applied to Safety by Design, the question becomes: who is this platform actually for? If the honest answer is ‘the most engaged users’ rather than ‘the most vulnerable ones’, that’s a values problem that can’t be fixed with an algorithm.
Moral leadership in this context suggests an action statement: user safety is a design priority rather than an afterthought; the interests of the most vulnerable users matter even when they’re not visible in the data; and foreseeable harm creates an obligation to act.
The Omegle Case Study
As a case study, the story of Omegle illustrates both the failure of moral leadership and its eventual, if belated, vindication.
Omegle was founded in 2009 by Leif K-Brooks, then eighteen years old, as a random video and text chat service pairing completely anonymous strangers.
That was the value proposition: connect any two users at random. The user experience was a massive dose of dopamine, never knowing who you might meet.
Otherwise, the service was entirely threadbare, with no registration, no moderation, and no accountability whatsoever.
It was, from the outset, a near-perfect grooming vehicle.
Reports of child sexual exploitation emerged almost immediately. For more than a decade, child safety organisations, regulators including eSafety, survivors, and law enforcement flagged Omegle as a significant vector for online child sexual abuse.
The platform’s response to these concerns was minimal: some keyword filtering, an optional ‘moderated’ section that remained largely unmonitored, and periodic statements from its founder asserting that misuse was the responsibility of users, not the platform.
Omegle operated on the premise that it was a tool, not a publisher, and that what people did with it wasn’t a design outcome. It was, of course, an entirely predictable design outcome.
In November 2023, Omegle shut down. The immediate cause was a lawsuit brought on behalf of a survivor known as A.M., matched at the age of eleven with a predator on the platform and subjected to years of sexual abuse.
In his farewell statement published to the Omegle website, K-Brooks wrote at length about the toll the litigation and regulatory environment had taken.
“Unfortunately, what is right doesn’t always prevail. As much as I wish circumstances were different, the stress and expense of this fight – coupled with the existing stress and expense of operating Omegle, and fighting its misuse – are simply too much. Operating Omegle is no longer sustainable, financially nor psychologically. Frankly, I don’t want to have a heart attack in my 30s.”
Omegle is an extreme case. But it is instructive precisely because the harm was foreseeable from the moment the product was designed.
The anonymous random pairing of strangers, with no age verification, no moderation, no reporting mechanisms, and no safety policies or guidelines was the design.
Clearly, technical capability wasn’t lacking.
What was lacking was the conviction that the users who could be harmed — children, in particular — were the platform’s responsibility.
The platform was constrained by values it didn’t hold, rather than constituted by the values it should have.
The Four Leadership Qualities
With Omegle as our cautionary example, let me turn to the leadership qualities that distinguish organisations that get this right.
These qualities go well beyond the algorithm. They shouldn’t be limited to product teams or trust and safety functions, but should be shared by everyone with a stake in safety – the entire organisation.
The first is curiosity and a desire to understand. What’s happening in our complaints, investigations, escalations? Who isn’t represented in the data? What might be the unintended consequences of a product change, and for whom?
The second is objectivity, or the capacity to reason from data rather than fit data to a conclusion. It requires an investigative mindset, weighing the evidence dispassionately, checking for fallacies, and resisting the temptation to look no further than what you expect to see.
The third is creativity and imagination. How could our systems be weaponised by a bad actor? What’s the worst-case scenario, and how do we work backwards from it?
The fourth is courage, or the willingness to make hard decisions that might conflict with revenue; to hold a position under pressure; to advocate upward for the why when questions are asked about the cost of the what and how.
When these qualities are in harmony, the result is mission alignment. This is the organisational condition in which safety decisions don’t require individual heroism because the institution is already committed. It is the inverse of Boisjoly and Navaroli, where one person or a small team shouldered a moral burden that should have been distributed across the whole organisation.
The Stakes
At this point it’s worth returning to what Tom Geraghty said about power: who gets to decide what’s important? Who has that institutional power?
When a newly-minted trillionaire calibrates an AI model and platform to automate industrial-scale, globalised image-based abuse, or invokes the tissue-thin rationalisation of ‘free speech!’ to urge civil war in the UK, these questions become critically urgent.
Urgent questions, too, need to be asked about how we think about the potential for change and reform. We must press for – or exercise ourselves – the moral leadership needed to overturn commercial models built on extraction and exploitation.
Lawrence Lessig, the Harvard law professor, captures something essential about this in Code is Law.
Every choice made about technology and its design reflects certain values. As he writes:
“The code regulates. It implements values, or not. It enables freedoms, or disables them. It protects privacy, or promotes monitoring. People choose how the code does these things. People write the code.”
The features and functions of technologies, their logics and their underlying goals are not received at the top of Mt Sinai; they are not inevitable. They reflect choices, and choices can be challenged, improved upon, and made more human.
Returning to the moral leadership action statement I offered earlier.
User safety must be a design priority rather than an afterthought.
The interests of the most vulnerable users must matter even when they are not visible in the data.
Foreseeable harm creates an obligation to act.
These are not compliance requirements. They are the values from which everything else in this operating model flows.