This post is part of a new series explaining concepts in Corporate Governance, and how they can be applied to AI firms. 

Understanding the three lines of defence

The three lines of defence model is a framework used to understand the roles that various functions have in managing the risk of activities within a firm. It is a valuable paradigm for structuring risk management functions, separating roles by the level of oversight they provide. This post describes this model in a context relevant to an AI firm. I will also go into details of where this model can help firms understand and highlight failures of their risk management and governance processes.

First Line

The first line is composed of those roles and functions that conduct the primary business objectives of the firm – that is to say, the parts of a business that are required to create and maintain its products, services and research. The first line involves almost everyone in the firm, from researchers, engineers and their managers to cybersecurity experts, in-house solicitors and the COO. Even cleaners and administrative staff serve an important role in the first line.

Perhaps the biggest, most important takeaway from this model, from any part of corporate governance, is that every employee in the firm has a key responsibility to manage risk. The negligence or malice of an individual has the potential to create gaps in a firm’s control framework, whether this is due to lax physical security, buggy code or gaps in a customer due diligence assessment. As such, effective risk management requires awareness of this responsibility to be baked into the culture of an organisation.

Second Line

The second line consists of oversight roles, specifically set up to monitor risk and determine where controls may need to be strengthened. These roles define the way that risk is measured in a company and sets broader policies and objectives for the first line to implement. As part of this goal, the second line works with a board of directors to determine the risk appetite of the firm – the amount of risk that is considered reasonable, or even desirable, in the pursuit of a business strategy. 

The second line is characterised by a balance of independence and context. Functions in this part of the model sit separately to the first line and are concerned with the accurate assessment of risk, though still ultimately report to the senior management team. As such, they provide a level of independence by the definition of their roles, but will often have some conflicts of interest in carrying out their assessment and management of risk. This trade-off allows the second line to have access to a level of context that is necessary to fully determine the level of risk that exists in their organisation, and to work much more collaboratively with the first line in its management. Though the second line has a responsibility to consider all stakeholders¹, it is still not in a position to place main priority on protecting the interests of society, and is ultimately seeking to implement the risk appetite policy determined by the board.

Third Line

The third line consists of a firm’s internal audit function, which is sometimes conducted entirely in-house and sometimes outsourced to another firm, often one of the Big Four or other large accounting practices. The trade-off between independence and context is even more crucial here – for an internal auditor to provide an effective level of challenge independence is essential. However, this makes the lack of context more present, especially when internal auditing is outsourced. It is very difficult for an internal auditor to have the full knowledge and understanding of a firm’s internal technologies and processes, even more so when the audit is conducted by a third party. As such, the challenges that they can provide will often be stymied as they are directed at less significant areas of the business, or formed with more limited access to data. 

Furthermore, even when the third line is outsourced there are still conflicts of interest. Ultimately, the firm that is being audited is still the client of the auditors, and the depth of challenge will come into conflict with the desire to continue the business relationship. As such, though the priorities of an internal auditor are more closely aligned with that of society at large, there are still inherent limitations to their ability to pursue this. Independent model evaluators should be aware that these limitations are also likely to apply to any future engagements they have with AI firms.

---

Assessing holes in the lines of defence

Framing a firm’s risk management process in this way allows for a greater examination of where issues may lie. Ultimately, it is impossible to fully prevent risk in an organisation – this is a property of the imperfect information available to all actors, a property inherent to the world in which we all live. Understanding risk frameworks using the three lines model opens an opportunity for firms to better manage their vulnerabilities. Some examples of these vulnerabilities are given below, accompanied by suggested solutions to address them.

First Line

Risk Awareness and Responsibility

Employees working in first line functions often lack a sufficient awareness of potential risks and their responsibility to prevent and report them.

There are many firms where a large number of employees may have issues with the way they are working, or concerns in the research and products they are producing. However, for many reasons the majority of these issues are not discussed with those who have the ability to address them, so the problems persist – and usually worsen. Some reasons this can happen include:

• Vague or cumbersome reporting processes
• Lack of belief that change will happen
• Apathy to the outcomes of the organisation
• Stress, or being overburdened with work
• A “shoot the messenger” culture, where reporting issues is discouraged, or even punished, by others.

As an addendum to the last reason, this is especially true when the person considering reporting is the person responsible. There are also many cases where people will see evidence of risky activities and behaviour, yet not even be aware that they are observing bad practice. A future post will provide some suggested items for a risk register in an AI lab and will illustrate this point clearly.

Solutions

There are two methods to addressing this vulnerability, to address two separate issues:

• Increase the risk awareness of individuals. This means making any actors in an organisation aware of both the potential risks they can encounter in their job, and their responsibility to report them. Some remedies include:
  • Training on these risks (such as anti-money laundering training mandatory in the finance industry)
  • Regular email reminders
  • Risk processes that require regular active engagement
  • Frequent sharing of risk incidents with all employees.
• Make it as easy as possible to report risks. This can be achieved through many different methods, including:
  • Clear and fast risk reporting processes
  • Risk functions actively reaching out to employees
  • A culture that rewards incident reporting, especially for those responsible
  • Clear owners for the different types of risk

In achieving this, it can be tempting to build systems that punish those who do not report risks. In practice, many studies² have demonstrated that punishment is usually far less effective than reward, and can actually disincentivise desired behaviours. As such punishment mechanisms should only be used with caution and forethought.

---

Risk Ownership and Information Flow

For a risk to be reported it first needs to be spotted. Although it is everyone’s responsibility to be aware of risks, many can only be discovered and managed through proactive attention. Without a strong story of ownership for each specific risk, mistakes will be unnoticed and repeated, and deficient controls will not be improved. Even for those risks to which proactive attention is given, the ineffective flow of information can make it impossible for incidents to be reported.

Solutions

• Create a risk register. Good risk ownership stories first require clear risk definitions. For this, a register of risks needs to be created. Best practices for doing this will be discussed in a future post, but broadly the register should have coverage of all of an organisation’s risks, with the greatest depth and specificity of individual risks being reserved for those that have the largest potential impacts to stakeholders.
• Assign a clear individual owner to each risk. Assigning individuals to take ownership of certain risks has the benefit of delineating liability, as well as empowering risk owners with the ability to make changes to prevent further risks. This gives the owners both the incentive to detect risk incidents³ and maintain a strong control framework to prevent future risk incidents.
• Create specific monitoring for each risk. The bespoke nature of different types of risks means that it is vitally important for a risk owner to consider what information they need to be able to detect risks and ensure they have access to it. This could for example take the form of clear reporting lines between employees, management information and alerts.

---

Second Line

Ambiguous Risk Appetite and Tolerance

An unclear definition of an organisation’s risk appetite and risk tolerance will lead to a second line that is unable to work effectively, as the standards of success have not been made clear. Risk appetite refers to the level of risk that a firm is actively seeking as part of its business strategy, where risk tolerance refers to the acceptance of inevitable side effects to the business strategy that are not intended. For a safety critical industry such as frontier AI research, risk tolerance must be as low as possible as it applies to society.

Solutions

• Establish a risk committee that regularly meets with the board of directors. Definitions of risk appetite and tolerance will evolve as the business evolves. As such, regular meetings between senior members of the second line risk functions and the board of directors are essential. This should occur at least once a quarter.
• Create risk metrics, but don’t rely on them. Metrics are necessary to create stronger definitions of the appetite and tolerance, but they are subject to Goodhart’s law⁴. Assessments of current risk appetite and tolerance must also contain qualitative measures that take into account the nuances of such a complex field.

---

Lack of Independence

As discussed above, a lack of independence will create conflicting incentives for second line functions. The second line is required to find the balance of providing challenge to the first line and board of directors, whilst also recognising the realities of operating in a complex environment. When the board wishes to update its risk appetite and tolerance, risk management functions must ask whether this is because the nature of the business has changed or because there are those who are motivated to inappropriately move the goalposts. For AI firms, it is especially important to consider all stakeholders, putting the safety of the public above any other interests.

Solutions

• Separate second line compensation calculations from profit incentives. As far as possible, remove the direct incentives for a second line to benefit from the obfuscation of risks to drive profit.
• Create separate risk and audit committees. To ensure independence from the third line, it is considered best practice in many industries to separate the roles of risk and audit committees.
• Increase challenge to the board by appointing non-executive directors. Non-executive directors, or NEDs, are vital to maintaining a board that acts in the interests of all. As such, they will be able to lend support to any challenges that a second line function may wish to make.

---

Lack of Context

Independence will often come at the cost of context. If a second line function does not have the requisite insight into the processes and technologies of the first line, it will be very limited in its ability to assess and challenge the risk and control frameworks as they currently exist. For a new field such as frontier AI research, the lack of standardisation and best practices makes this vulnerability especially relevant.

Solutions

• Engage first line risk owners with second line responsibilities. By increasing the engagement of those responsible for risky processes in the first line with risk and control assessments, incident reporting and even risk appetite setting, the barriers of communication between the first and second line are lowered. This pairs well with the solutions presented in both the sections on first line vulnerabilities described above.
• Conduct risk interviews with the first line. Similar to the point above, risk interviews with members of the first line can be very insightful in getting a better understanding of a risk and control framework. The more specific that questions can be, the less likely it is that the second line will miss important details due to a lack of context.
• Hire more technical and first line experts into the second line. Moving employees from the first line into the second means that their knowledge is imported with them. It also makes it much easier to engage with other members of the first line, leveraging the understanding of processes and terminology that an expert provides.

---

Third Line

Lack of Independence

As with the second line, internal audit teams that exist in-house will suffer from many of the conflicting incentives that come from needing to report to individuals that may not be motivated to act in the interests of all stakeholders. The definition of the third line means that there is more scope for independence, though this creates a potential issue akin to moral licensing where the audit team may be less aware of the influence of such competing interests.

Solutions

• Separate third line compensation calculations from profit incentives. In theory this should be easier than it is for the second line. However, care should be taken around reporting lines – though the compensation of a third line employee may not be tied to company profit, if their manager’s compensation is they will still experience pressures to downplay risks. For any outsourced internal audit function, there are strong profit incentives to maintain the business relationship that can potentially come at the cost of providing sufficient challenge to their clients. Fixed term contracts could help to address this, though I do not believe they are currently an industry standard.
• Focus on a diverse and clearly defined set of tests, mapped to industry best practices. The more well defined that tests of a company’s practices are, the harder it is for conflicting interests to influence their outcomes. Mapping these to industry benchmarks and best practices further provides a legitimate route of challenge.
• Delineate the second and third lines. If an audit team and an audit engagement team work too closely together, the third line has the potential to essentially become an extension of the second line. This will neuter an essential aspect of the firm’s risk management processes, so care must be taken to define these roles clearly and separately.
• Engage with third-parties often. This may be through outsourcing the third line, but consultants can also be extremely effective at providing independent challenge to certain processes. They will also carry the context of seeing how things work in many different organisations, so they are often able to present insights that would not be spotted by internal teams.

---

Lack of Context

The lack of context that internal audit teams can suffer from is substantial, especially for outsourced third lines. This hampers the ability for an audit to provide an effective challenge of processes, yet is often considered to be inevitable as a consequence of independence. The solutions suggested below offer some evidence that this need not be so.

Solutions

• Use a diverse set of testing methodologies. Testing of processes should take many forms, including quantitative tests (analysing metrics), demonstrative tests (meetings with first line employees to show how they do their work) and methodological tests (examinations of process descriptions and risk assessments for  weaknesses in the control environment). This should be true for every area of the business that is in scope of a given audit. For the model evaluators of the future, it will be important to develop a broad range of methodologies such as those described above. Though they do not currently exist, evaluations based on prompt engineering, mechanistic interpretability, and staged scaling to detect emergent capabilities could all be considered.
• Engage directly with the first line. Though second line audit engagement teams can be valuable in understanding control deficiencies, an audit that relies solely on information gathered from these teams is too removed from the actual work that gives rise to risks. Interviews with first line workers and demonstrations are examples of some methodologies that allow direct access to these processes.
• Create and actively maintain detailed documentation. Documentation provides another effective route of challenge when it is maintained correctly. There are many great reasons for a company to invest heavily in documentation, something that will likely be discussed in a future post, but one benefit is that it allows a greater understanding of how processes work at multiple levels. So-called “folk knowledge” in companies is inevitable – it would be impossible to truly write down everything every employee knows about the business – but documentation is an important method of sorting information and giving auditors the base knowledge of “how things should work”. When a process then works differently in reality, exploring why this is the case can be extremely valuable for strengthening control frameworks.

---

The Fourth Line

One criticism that is often applied to the three lines model is that it exists in separation to external auditors and regulators. To some extent this is fair – the relationships between these entities and the three lines can be valuable to understand. As such, many researchers and firms add those actors to the existing three lines to create a fourth line. 

Though this will be useful to consider in the future, I have avoided it here for a few reasons. First, there currently are next to no regulators for frontier AI, and the legislative environment is extremely underdeveloped. Therefore, it is difficult to critically examine these relationships in a way that will not be inherently speculative. Without some knowledge of what a regime may even look like, there are very few valuable insights that can be gained from this analysis.

Second, and perhaps more important, is that the purpose of the three lines model as far as I see it is to be able to understand and critically assess the risk management and control framework of an organisation. To include regulators as a fundamental part of that framework is to take away responsibility from the companies themselves in managing their risk, something that I believe could be very dangerous. I strongly recommend that regulators and external auditors do not become considered as part of a firm’s control processes, but instead are thought of as part of the market that the business exists in. 

1. In the UK this is a legal responsibility here (as with the directors under company law) to the shareholder members of the company, in the contextual awareness of the wider stakeholder environment (including public interest, particularly for a charity or not-for-profit). ↩︎
2. Though I encourage this to be explored further by the reader, one such study can be found here. ↩︎
3. Presuming that risk reporting is encouraged and rewarded. ↩︎
4. https://en.wikipedia.org/wiki/Goodhart%27s_law ↩︎