The Double-Edged Sword of Functional Safety: Are We Ready for the Next Generation of Smart Machines?

Imagine a state-of-the-art, AI-driven manufacturing robot installed in a local factory. It is a marvel of modern engineering, boosting productivity by 30% through its ability to adapt to changing product lines and optimise its own movements. For months, it has operated flawlessly. Then, one day, it makes an unpredictable move, a slight unexpected deviation from its normal path, causing a near-miss incident that forces an emergency shutdown of the entire production line. An investigation reveals no hardware faults, no software bugs in the traditional sense. The machine’s intelligence itself behaved in a way its creators did not fully anticipate.

This scenario is no longer the stuff of science fiction. It represents the central challenge facing industrial automation today. The principles of functional safety, which have successfully protected workers and assets for decades, are built on a foundation of predictability. We design systems to fail in safe, predictable ways. But the introduction of adaptive, non-deterministic Artificial Intelligence (AI) shatters this foundation. For business leaders in Malta and across the European Union, this is not merely a technical problem for engineers to solve; it is a fundamental challenge to risk management, legal liability, and the very license to operate in the EU market.

As machines get smarter, are our safety rules becoming obsolete? The very standards that have been the bedrock of trust, like EN ISO 13849-1; which provides the methodology for designing and integrating safety-related parts of control systems and EN 62061, which guides designers to cover the functional safety of safety-related control systems, are now being stretched to their limits. They were designed for a world of circuits and code that followed deterministic rules, not for systems that can learn and evolve after they leave the factory. This raises profound questions for every business involved in the manufacturing, importation, or operation of modern machinery. Are our standards, our certification bodies, and our companies truly ready for this paradigm shift? How can we ensure safety is not just a paper exercise when a machine's brain can effectively change its own mind?

Before dissecting the disruption caused by AI, businesses need to understand the framework that AI is, in fact, disrupting. Functional safety is not just an engineering discipline; it is a critical function that underpins product liability, market access, and brand reputation. It is the active part of a machine's overall safety system; the automatic protection that intervenes to prevent harm when something goes wrong. A simple example would be the kettle’s thermostat that prevents it from overheating; this is an operational requirement, but also acts as a basic safety function. Now scale that concept to a complex industrial press or a high-speed robotic arm. The functional safety system is what detects a person entering the danger zone and commands the machine to return to a safe state, preventing a catastrophic failure, injury and, by extension, catastrophic financial and legal consequences.

For decades, the "rules of the road" for ensuring this protection in the European Union have been defined by two pivotal, harmonised standards: EN ISO 13849-1 and EN 62061. Compliance with these standards is, for all practical purposes, the passport to placing machinery on the EU market.

At the heart of these standards are two key metrics used to quantify the reliability of a safety function: Performance Level (PL) and Safety Integrity Level (SIL). While they stem from slightly different philosophies, they both aim to answer the same fundamental question: "How likely is this safety function to fail dangerously when it is needed?".

Performance Level (PL), is the more commonly used metric in the world of discrete machine building. It provides a pragmatic, probabilistic assessment of a safety function's reliability. A required PL is determined through a structured risk assessment that considers three factors:

• Severity of injury: Is the potential injury slight or serious/fatal?

• Frequency and/or exposure to hazard: Are people exposed to the hazard rarely or frequently?

• What’s the possibility of avoiding the hazard? Is it possible to avoid the danger or scarcely possible?

Based on these qualitative inputs, a required PL is determined. The machine designer must then engineer a safety-related part of a control system that achieves this level. The final PL is a calculated value based on the system's architecture, the reliability of its components, its ability to detect faults, and its resilience to single points of failure that could have a cascading affect.

Safety Integrity Level (SIL), provides a more rigorous, lifecycle-oriented approach to safety. It classifies safety functions into four levels based on their required risk reduction. The SIL determination process is more detailed, considering parameters like severity, frequency of exposure, probability of the hazardous event, and the ability to avoid harm. While PL is often seen as more suitable for individual machines, SIL is frequently applied to larger, more complex, or integrated systems.

The key takeaway is that both PL and SIL are ultimately expressed as a Probability of Dangerous Failure per Hour. They provide a tangible, quantifiable measure of risk reduction, allowing for a clear assessment of whether a safety system is adequate for the hazard it is designed to control. While the standards are probabilistic about the failure of a component, they are fundamentally deterministic about its function. The calculations derived all presume that if a safety component or system has not failed, it will execute its pre-defined, specified function correctly and repeatably. The risk being managed is that of a component breaking down, not of the component misbehaving while perfectly healthy. This foundational assumption, that a non-failed system is a correctly behaving system is precisely what AI invalidates. AI introduces a new, deeply problematic failure mode: functional insufficiency. The system's hardware can be 100% operational, yet it can still produce an unsafe action because its internal logic is probabilistic, adaptive, and potentially opaque. Therefore, the very premise of our established safety calculations is being challenged at its core.

The integration of Artificial Intelligence into safety-critical control systems is not an incremental evolution; it is a revolutionary break from the past. AI-based systems, particularly those using machine learning (ML), introduce characteristics that are fundamentally at odds with the deterministic worldview of traditional functional safety. This creates a chasm between the capabilities of the technology and the ability of our established standards to ensure its safety.

As discussed in previous articles, many of the most powerful AI models, especially the deep neural networks that excel at complex tasks like image recognition or navigation, are effectively "black boxes". While engineers can define the inputs and observe the outputs, the intricate, multi-layered decision-making process within the network is often too complex for any human to fully comprehend or explain. This is known as algorithmic opacity.

For a business ready to invest in advanced machinery, the implications are profound and unsettling. How can a manufacturer confidently sign an EU Declaration of Conformity, legally attesting to a machine's safety, when the logic of its primary safety function cannot be fully explained? How can a Notified Body, the independent third-party arbiter of safety, certify a system whose internal workings are inscrutable? This opacity introduces a level of legal and ethical ambiguity that the industry has never before faced. It shifts the safety argument from "we have verified the logic" to "we have tested the outputs" conclusion, which, as we will see, is an insufficient substitute.

A cornerstone of all engineering validation is repeatability. If you apply the same inputs under the same conditions, you expect the same result. Non-deterministic systems violate this principle. By design, many AI models are probabilistic; given the same input, they may produce slightly different outputs on different runs. While this variability is an acknowledged feature for creative applications like generating text or images, it is a critical flaw in a safety context.

Traditional safety validation relies on being able to reproduce a test case and get a consistent outcome. If a system passes a safety test 999 times but fails on the 1000th attempt under identical conditions, it cannot be considered reliable. This non-determinism makes it practically impossible to exhaustively validate the system's safety. You can never be certain that you have uncovered all potential failure modes, because you can never be certain that the system will behave the same way twice. This fundamentally undermines the confidence that a given Performance Level or Safety Integrity Level system provides.

Perhaps the most disruptive characteristic of advanced AI is the potential for continuous or online learning. Some systems are designed to adapt and modify their behaviour based on new data they encounter after they have been deployed and commissioned. The machine can literally "teach itself" new ways of operating.

This creates a compliance nightmare. A machine that is CE-marked and certified as safe on Monday could, through its learning process, develop a new, unsafe behaviour by Friday, without any physical or software modification by the manufacturer. This invalidates the original conformity assessment and raises intractable questions of liability. Who is responsible for the safety of a machine that modifies its own safety-critical logic? Is it the manufacturer who built it, the owner who operates it, or the data provider whose data it learned from? This capability for post-deployment evolution breaks the static, point-in-time model of certification that underpins the entire European regulatory framework.

Finally, it is crucial to understand that an AI system is not just software; it is software inextricably fused with data. The quality, quantity, representativeness, and freedom from bias of the data used to train an ML model are what fundamentally determine its performance and safety. A model trained on incomplete or biased data will inevitably have blind spots and may behave in discriminatory or unsafe ways when it encounters real-world situations not well-represented in its training set.

This means the safety case for an AI-driven machine now extends to its entire data lifecycle. Manufacturers must be able to prove the provenance and quality of their training, testing, and validation datasets. This is a new, complex, and potentially costly requirement that involves robust data governance, a discipline more familiar to IT and data science than to traditional machine safety engineering.

In response to the profound technological shift, the European Union has erected a new, formidable regulatory structure. For any local business involved in the machinery sector, understanding this new legal reality is not optional; it is a prerequisite for survival and growth within the EU market. Two pieces of legislation form the pillars of this new framework: the Machinery Regulation (EU) 2023/1230 and the AI Act (EU) 2024/1689. Together, they create a complex compliance gauntlet for manufacturers, importers, and distributors of smart machinery.

The new Machinery Regulation (MR), which will be fully applicable from 20 January 2027, replaces the long-standing Machinery Directive. The most significant change for business leaders is its legal form. As a Regulation, not a Directive, its rules are directly and uniformly applicable across all EU member states, without needing to be transposed into national law. This aims to increase legal certainty.

Crucially, the MR brings the concepts of AI and cybersecurity directly into the heart of machinery safety law. It explicitly defines "software that performs a safety function" as a "safety component," placing it on par with physical guards or emergency stop buttons. Furthermore, it introduces new Essential Health and Safety Requirements (EHSRs) that directly target the risks of intelligent systems. These require that a machine's safety control system be designed to prevent hazardous behaviours, including those arising from AI systems that learn or evolve. The regulation demands that these systems remain within the safe boundaries established during the initial risk assessment.

The EU AI Act, which entered into force in August 2024 and will be in full force by August 2026 with a staggered implementation, establishes a comprehensive legal framework for artificial intelligence. It employs a risk-based approach, categorising AI systems into four tiers; unacceptable risk, high risk, limited risk and minimal risk.

For the machinery sector, one point is of paramount importance: AI systems that serve as safety components in machinery are automatically classified as high-risk under the AI Act. This classification is not optional and is not based on a separate risk assessment; it is a default legal status. This triggers a cascade of demanding legal obligations for the provider of the AI system. The interplay of these two regulations creates a situation of "double jeopardy" for manufacturers. A smart machine's safety system must now undergo a dual conformity assessment. The machine as a whole must comply with the Machinery Regulation, while the AI component within it must separately comply with the stringent requirements of the AI Act. This is not a single, streamlined process. It means the technical documentation for the MR must now incorporate the evidence of compliance with the AI Act. This dual compliance track dramatically increases the complexity, cost, and time-to-market, posing a significant challenge for all manufacturers, but especially for SMEs that may lack large, dedicated compliance departments.

A second, equally profound shift is the legal codification of the link between cybersecurity and functional safety. For decades, safety engineering focused on protecting against random hardware failures and systematic design faults. The world of industrial machinery was largely air-gapped and isolated. Today's interconnected Industrial Internet of Things (IIoT) environment changes everything. A machine connected to a network is exposed to "intentional faults", malicious cyberattacks. A hacker could potentially exploit a vulnerability in a PLC or an insecure network protocol to remotely disable a light curtain or manipulate a robot's safety limits, with devastating consequences.

Recognising this, both the Machinery Regulation and the AI Act make cybersecurity a mandatory, non-negotiable component of safety. The MR's EHSRs explicitly require that safety-related control systems be protected against "intended or unintended corruption". The AI Act likewise lists cybersecurity as a core requirement for high-risk systems. This means a manufacturer's risk assessment must now treat malicious actors as a foreseeable hazard. The technical file must document cybersecurity measures, such as network segmentation, encryption, secure authentication, and vulnerability management, as integral parts of the overall safety case. Cybersecurity is no longer an "IT issue"; it has become a fundamental prerequisite for achieving and demonstrating functional safety. This forces a cultural and organisational fusion, requiring safety engineers to work hand-in-glove with cybersecurity experts.

So what happens now? To fill this critical gap, a new generation of standards is being forged by a joint effort between ISO and the IEC. These documents provide a roadmap to the future of AI safety engineering.

The first key document is ISO/IEC TR 5469:2024, "Artificial intelligence — Functional safety and AI systems." As a Technical Report (TR), it is not a normative standard that mandates requirements, but rather an informational "state-of-the-art" guide. It is the first formal attempt by the international standards bodies to map the landscape of AI safety. It provides an overview of functional safety in the context of AI, describes the properties and risk factors of AI systems, and explores potential control and mitigation measures. For example, it discusses architectural patterns for safe AI, such as using a simple, non-AI "fail-safe" backup function that takes over if the AI's behaviour is detected as unsafe, or using a supervisor function to constrain the AI's actions within pre-defined safe limits.

Building on this foundation is the forthcoming ISO/IEC TS 22440, "Artificial intelligence — Functional safety and AI systems — Part 1: Requirements." This Technical Specification (TS) is the crucial next step, as it will move from guidance to providing concrete, testable requirements for functionally safe AI systems. The draft work indicates it will focus on several core areas essential for certification: risk assessment and hazard analysis, transparency and explainability, validation and verification and safety analysis techniques to estimate the likelihood of AI-related failures.

These two documents signal the direction of future regulation and conformity assessment. Aligning internal R&D and quality management processes with the principles laid out in TR 5469 and the coming requirements of TS 22440 is the most effective way to prepare for the compliance landscape of tomorrow.

The most formidable technical hurdle in bringing AI-driven machinery to market is validation. The traditional engineering concepts of verification, "Did we build the system right?" and validation, "Did we build the right system?" are strained to their breaking point by AI's inherent complexity and unpredictability. How can you test for "unknown unknowns" in a system that might generate them spontaneously? The answer lies in a new frontier of validation tools that move beyond physical testing into the virtual world. A digital twin is a high-fidelity, dynamic virtual replica of a physical machine, its control system, and its operating environment. By feeding the twin with real-time data from its physical counterpart, it can mirror the asset's state and behaviour. For safety validation, its power is immense. Digital twins enable massive-scale simulation, allowing manufacturers to subject their AI systems to millions of "what-if" scenarios, edge cases, and fault conditions that would be far too dangerous, expensive, or time-consuming to test in the real world.

Want to know how an AI-powered autonomous forklift reacts to a sudden spill on a polished floor under low-light conditions? A digital twin can simulate that scenario a thousand times with slight variations, providing a statistical basis for its safety performance. This virtual proving ground is essential for building confidence in an AI's ability to handle the near-infinite permutations of the real world. Case studies in industries from wastewater management to aerospace are already demonstrating the power of this approach to improve performance and safety.

While simulation tests what a system does, formal methods aim to prove what it cannot do. In simple terms, formal methods use rigorous mathematical logic to analyse a system's model and prove that it cannot enter a defined unsafe state, regardless of the inputs. Instead of relying on a finite number of test cases, this approach seeks to provide an exhaustive guarantee of certain properties, effectively proving the absence of specific bad behaviours.

For AI, particularly neural networks, this is a game-changer. While computationally intensive, formal methods are one of the few techniques that can provide strong, verifiable guarantees about an AI's robustness against adversarial attacks or its behaviour when faced with out-of-distribution data (inputs it has never seen before). This directly helps to mitigate the "black box" problem by allowing engineers to place a mathematically proven boundary around the AI's possible outputs, ensuring it stays within a safe operating envelope. While still an emerging field, practical applications in robotics and other critical domains are demonstrating its feasibility.

The rise of these advanced techniques points to a fundamental change in the nature of safety evidence. For traditional machines, physical test reports were a cornerstone of the safety file. For complex AI systems, the sheer number of possible states makes exhaustive physical testing an impossibility. Consequently, the burden of proof for safety is shifting from the physical world to the virtual one. The safety case will increasingly depend not on a log of physical tests, but on the quality, fidelity, and rigour of the virtual validation process. This elevates the simulation and the digital twin to a position of critical importance. The central question from a regulator or Notified Body will no longer be just, "What were the results of your tests?" but rather, "How can you prove your virtual world is a valid and trustworthy representation of reality?" This means manufacturers must now invest heavily in validating their validation tools. The digital twin itself becomes a safety-critical asset, requiring its own meticulous verification and validation process, a significant and complex new overhead.

The cumulative effect of these challenges, dual regulatory frameworks, complex new validation techniques, and a scarcity of expertise falls disproportionately on Small and Medium-sized Enterprises (SMEs). This difficult landscape may lead to an unintended and paradoxical consequence: a "flight to simplicity." Faced with the high cost, complexity, and uncertainty of certifying an AI-based safety function, a rational business decision for many SMEs could be to deliberately avoid using AI in safety-critical applications altogether. To escape the most stringent and costly tier of regulatory scrutiny, a manufacturer might opt for a traditional, well-understood safety system, such as a standard PLC and light curtain, over a potentially more effective but regulatorily daunting AI solution. This is not a hypothetical risk; industry leaders have explicitly stated that the administrative burden will make companies question whether to use AI in their products. The very regulations designed to ensure safe AI could, in the short-to-medium term, disincentivise the use of AI in safety-critical functions, potentially slowing the adoption of technologies that could, if validated correctly, make workplaces even safer.

The era of intelligent machinery has irrevocably altered the landscape of functional safety. The journey from predictable, deterministic systems to adaptive, non-deterministic ones represents a true paradigm shift. Safety is no longer solely about preventing hardware failure; it is about managing the behaviour of complex intelligence. This shift is now enshrined in a demanding dual-regulatory framework, the Machinery Regulation and the AI Act, that places unprecedented responsibility on every actor in the supply chain.

For the business leaders of Malta, the message is clear: a superficial, "tick-box" approach to compliance is a recipe for commercial and legal disaster. The inherent unpredictability of AI means that a deep, genuine, and demonstrable culture of safety, backed by robust engineering and state-of-the-art validation, is the only true defence against catastrophic failure and the immense liability that would follow.

This challenge, however, presents a distinct opportunity. As a nation characterised by agile SMEs deeply integrated into the wider European supply chain, Malta is well-positioned to turn this regulatory burden into a competitive advantage. Companies that become early, expert adopters of best practices in AI safety will signal a level of trustworthiness, quality, and forward-thinking that will be highly valued by larger European partners and customers. Mastering this new domain is not just about avoiding fines; it is about building the foundation for future growth.