AI is pervasive but presents itself in schools in three distinct ways.

First, students use tools such as ChatGPT and AI companions for research, writing, homework support, and conversation. This is the most visible form of AI and the one that generates the most debate about academic integrity, wellbeing, and unsupervised access.

Second, teachers use AI to plan lessons, create materials, provide feedback, and reduce administrative workloads. Schools are beginning to balance these benefits with policies governing privacy, accuracy, and appropriate use.

The third application is less visible but potentially more important to student safety. AI is increasingly embedded in filtering and safeguarding systems.

Behind the scenes, AI can analyse large volumes of student activity, identify patterns associated with potential harm, and prioritise incidents for review. It can help surface indications of self-harm, bullying, violence, exploitation, or other safeguarding concerns that might otherwise be difficult to detect.

However, AI does not safeguard a student by itself.

AI identifies a signal. A person must decide what that signal means and what should happen next.

Monitoring Does not Equal Decision-Making

Filtering and monitoring are related, but they serve different purposes.

Filtering is preventative. It blocks or restricts access to content that has been classified as illegal, inappropriate, or harmful.

Monitoring is reactive. It reviews activity and generates reports or alerts when something may require attention. Modern behavioural monitoring can analyse searches, text, online conversations, and viewed content at a scale that no safeguarding team could review manually.

The scale of AI and its ability to process vast amounts of data at speed is AI’s strength. Judgement is not.

An alert should never be treated as a diagnosis, verdict, or disciplinary finding. It is an indication that a trained person may need to look more closely.

The data and output from monitoring systems is a flag, not a conclusion.

A Flag Is Not a Safeguarding Outcome

Technology can recognise words, behaviours, and patterns associated with risk. It cannot fully understand the student behind them.

AI may detect a concerning phrase, but it may not know whether the student is expressing an immediate intention to cause harm. It may not know whether the student is quoting a book, song, or classroom assignment. It may not know whether the student is describing an event that happened to someone else. It may not understand whether the student has a known history that changes the meaning of the activity. It may also fail to recognise humour, slang, or language that is specific to a particular community.

AI also cannot sit with a distressed student, contact a parent, consult a counsellor, or determine whether emergency intervention is necessary.

Those decisions require professional judgement, knowledge of the student, and an understanding of the wider circumstances.

The US Department of Education makes this distinction explicit:

“A top policy priority must be establishing human in the loop as a requirement in educational applications.” [1]

The point is not that AI is ineffective, AI and people perform different roles.

AI provides reach, speed, and consistency. Humans provide context, accountability, empathy, and action.

Human Oversight Must Be Designed Into the System

Human intervention cannot be an informal expectation that is added after a monitoring tool has been purchased. It must be part of the operating model.

Every alerting system will produce activity that requires interpretation. Some alerts will prove urgent. Others will be false positives, misunderstandings, or activity that does not require intervention. Systems may also miss risks that do not resemble previously recognised patterns.

Without effective oversight and governance, schools face two significant dangers.

The first danger is overreaction. A student may be labelled, disciplined, or escalated because an automated system misunderstood the context.

The second danger is a failure to act. A serious alert may sit unread, reach the wrong person, or lack enough information for someone to respond quickly.

As new as AI is, governance organizations have rushed to provide guidance as to how to ensure that humans partner with schools to act on AI generated alerts to provide contextual support for students, teachers, and administrators. Some recent guidance from educational policy leaders include:

In student safeguarding, responsibility must remain with a clearly identified and appropriately trained person:

The National Institute of Standards and Technology warns that converting complex human behaviour into mathematical models can remove necessary context. Its AI Risk Management Framework also states that human intervention may be required when an AI system cannot detect or correct its own errors. [2]

Australia’s national framework expresses the principle directly:

“Teachers and school leaders retain control of decision making and remain accountable.” [3]

Technology may support the decision. It must not become the decision-maker.

What Effective Monitoring Looks Like

A responsible monitoring process follows a clear sequence. Here are some recommendations on how to implement a process that combines the power of speed of AI with the awareness and intelligence of humans.

1. AI Detects and Prioritises Potential Risk
The system reviews activity at scale and identifies behaviour that may warrant attention. It assigns severity based on defined safeguarding criteria.

2. The Alert Includes Meaningful Context
The reviewer should receive more than a category or risk score. The alert should include enough relevant information to explain why it was generated.

This information may include the words used, surrounding activity, timing, device details, or conversation context.

A notification stating that a student used an AI chatbot is not sufficient. The safeguarding question is what occurred during the interaction and whether it indicates a credible risk.

3. The Alert Reaches a Named Person
High-risk alerts must be routed promptly to a designated safeguarding lead, counsellor, or other trained professional. Schools should also establish backup coverage for absences, evenings, and other periods when the primary contact may be unavailable.

4. A Professional Reviews the Circumstances then Decides Which Action to Take
The reviewer considers the alert with context and their perspective on the student, relevant history, school policy, including surveying the seriousness and immediacy of the possible harm.

The response might include speaking with the student, notifying parents, involving school leadership, or escalating the matter to emergency or social services.

In other cases, the reviewer may determine that no intervention is required. That can also be the correct outcome. Effective safeguarding is not measured by the number of escalations. It is measured by whether the right concerns receive the right response.

5. The Outcome Is Recorded and Reviewed
Schools should document who reviewed the incident, what action was taken, and what outcomes resulted. This provides accountability and helps determine whether alert thresholds, procedures, or staff training need adjustment.

The Department for Education (UK) reflects this model in its filtering and monitoring standards. It requires clear responsibilities, appropriately trained staff, designated safeguarding lead involvement, and processes for acting on reports and concerns. Its 2026 AI product safety standards also call for high-risk alerts to be sent to the responsible safeguarding contact within an agreed timeframe. [4]

Three Questions Every School Should Ask

1. Who Reviews an Alert Before Action Is Taken?
Schools should understand where human review occurs, who is accountable, and what happens when the primary reviewer is unavailable.
An automated flag should not independently trigger a high-stakes decision about a student.

2. What Information Does the Reviewer Receive?
A risk label without supporting context creates guesswork. Reviewers need understandable and relevant information that enables them to assess the situation quickly and proportionately.
Schools should ask whether the system provides the actual activity, the surrounding context, the severity, the timing, and the reason for the alert.

3. What Happens When the System Is Wrong?
Schools should expect vendors to explain false positives, missed detections, escalation thresholds, and the process for challenging or overriding an alert.
The system should support professional judgement. It should not discourage staff from questioning the technology.

The Standard Schools Should Expect

The strongest safeguarding system is not the one that removes people from the process.
It is the one that gets the right information to the right trained person quickly enough for that person to make a difference.
AI can help schools find signals hidden within enormous volumes of digital activity. However, it cannot know a child, understand every circumstance, or provide care.

AI can identify the risk. Human intervention protects the student.

Source Validation

[1] US Department of Education. Its national AI in education report states that teachers and learners must retain agency to determine what patterns mean and select a course of action. It describes establishing humans in the loop as a top policy priority for educational applications.

U.S. Department of Education, Office of Educational Technology. (2023). Artificial Intelligence and the Future of Teaching and Learning: Insights and Recommendations. https://tech.ed.gov/ai/

[2] NIST AI Risk Management Framework. NIST cautions that mathematical representations of complex human behaviour can remove important context. It also states that human responsibilities should be clearly defined and that human intervention may be needed when systems cannot identify or correct errors.

National Institute of Standards and Technology. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0) (NIST AI 100-1). https://doi.org/10.6028/NIST.AI.100-1

[3] Australian Framework for Generative AI in Schools. The framework’s accountability principle requires human agency, human responsibility, active monitoring, and the ability to question AI-supported decisions.

Australian Government Department of Education. (2025). Australian Framework for Generative Artificial Intelligence (AI) in Schools. https://www.education.gov.au/schooling/resources/australian-framework-generative-artificial-intelligence-ai-schools

[4] Department for Education (UK). Current filtering and monitoring standards require trained staff to act on reports, designated safeguarding leads to review reports and respond to concerns, and schools to maintain documented processes for handling incidents. The January 2026 product safety standards also require systems to alert local supervisors and send high-risk alerts to the responsible safeguarding contact within an agreed timeframe.

Department for Education. (2026). Meeting digital and technology standards in schools and colleges: Filtering and monitoring – core standard. GOV.UK. https://www.gov.uk/guidance/meeting-digital-and-technology-standards-in-schools-and-colleges/filtering-and-monitoring-core-standard

Department for Education. (2026). Generative AI: Product Safety Standards. GOV.UK. https://www.gov.uk/government/publications/generative-ai-product-safety-standards/generative-ai-product-safety-standards