AI Safety Institutes play a critical role in global AI governance
Plus: ISO 42001 audits, Republican AI policy blueprint, and top 10 risks of AI agents
Hello from a sunny and cold Washington DC. While many of you may be slowing down as we approach the holiday season, AI governance news never slows down. Stay tuned for our next and last newsletter of 2024 where we’ll cover our Top 10 predictions for AI Governance in 2025.
In today’s edition:
AI Safety Institutes play a critical role in global AI governance
AWS gets ISO 42001 Certified
The Emerging Red State Approach for AI Oversight
Emerging Risks of Agentic AI
1. AI Safety Institutes play a critical role in global AI governance
Measuring safety risks in AI systems is a complex task as it requires deep technical expertise, cooperation with big tech companies operating in a competitive space, and keeping up with the pace of AI innovation. In response to this, governments around the world have stood up dedicated ‘AI Safety Institutes’ tasked with trying to develop the science for measuring the capabilities, safety, and risks of advanced AI systems. So far, 9 countries have created (or proposed) AI Safety Institutes or similar government departments.
Recently, there has been a flurry of activities and updates from AISIs around the world worth covering:
AISI Global Summit - AISI staff members from 9 countries met in a 2-day summit in San Francisco to discuss past initiatives, potential collaboration, and exchange best practices.
Claude 3.5 Joint Analysis - The US and UK AISIs announced the publication of their joint pre-deployment analysis of Anthropic’s Claude 3.5-Sonnet model. This gave the world a look into what standardized pre-deployment risk evaluation processes may look like, and specifically tested the model for biological, cyber, software creation, and general safety capabilities.
US AISI Consortium Plenary - The US AISI hosted its first plenary session for its consortium, its public-private partnership program. The consortium was created to gather community feedback on various aspects of AI Safety research, and the plenary session was to present initial work from its 5 working groups, and collect feedback on future initiatives. Trustible is a proud member of the AISIC, and also sponsored the plenary session.
The work of AISIs is only getting started. However, the US AISI was created through President Biden’s Executive Order and, as a result, President-elect Trump’s planned revocation of that EO could lead to the US AISI’s dissolution. While Trustible remains non-partisan, and neutral on most AI policy issues, we do support Congressional action to authorize and fund the US AISI. We support this position for the following reasons:
US Leadership - Research into AI safety practices will not stop if the US AISI is dissolved. However, it will remove the US from having a dedicated seat at the head of the table where it can shape the nature and direction of this work. Despite the rhetoric, dissolving the US AISI won’t actually accelerate innovation or reduce any compliance burdens as most AI providers will still need to comply with requirements emerging from the EU, India, Canada, and the UK.
Leading Science - AI safety research is not an inherently partisan issue. Regardless of whatever definition of bias, fairness, or safety is used, there needs to be scientifically sound practices established to measure it. Being the leader in AI technology, and the leader in AI safety, are not mutually exclusive.
Accelerated adoption and growth - Many organizations are hesitant to adopt AI because they don’t understand how to govern it or assure its safety to some degree. Even voluntary, market-driven approaches to this problem benefit greatly from common scientifically-sound best practices. Provably safe AI systems will be adopted more widely, which can both accelerate economic growth and incentivize further innovation.
Key Takeaway: Trustible supports the mission and efforts of AI Safety Institutes, and dissolving the US AISI won’t actually help accelerate AI innovation, but it will hurt the US’s reputation and ability to lead global AI standards.
—
2. AWS gets ISO 42001 Certified
Amazon Web Services (AWS) announced it has received its first ever ISO 42001 certification for specific AI services. While a few other smaller organizations have previously received ISO 42001 certification, AWS is the first major AI provider to publicly disclose an audit (Google Cloud previously received a less formalized ‘assessment’). AWS’s auditing firm, Schellman, only recently received their own formal approval to conduct ISO 42001 audits. Certification against ISO 42001 involves an examination of an organization’s AI governance and management processes, and an assessment of compliance with any controls the organization may have implemented in support of it.
AI Audits, as well as formalized AI Assessments, can be powerful tools for building trust with customers and other stakeholders. Microsoft’s procurement requirements allow for ISO 42001 certification to replace Microsoft’s own AI due diligence process. However, the AI audit ecosystem is still early in its development, with challenges encountered with NYC’s Local Law 144 audit for HR systems and the ISO standard for AI audit practices (ISO 42006) still under development. While voluntary external audits are starting to emerge for general governance practices, mandatory model level audits, and compliance standards for the EU AI Act are still pending.
Key Takeaway: The era for formalized AI governance audits is beginning, but it may be a while still before they become widespread and more formalized.
—
3. The Emerging Red State Approach for AI Oversight
As Republicans prepare to take over Congress and the Presidency in a few weeks, questions are swirling over the contours of a Republican approach to AI oversight and regulation. However, a more pressing question should be how red state lawmakers may think about AI-related issues as we approach the 2025 legislative session. In the past year, blue states (e.g., Colorado, Connecticut, and California) have been more proactive in tackling AI regulations on a broader level than red states. Yet this may be changing as legislators in Texas, as well as members of the American Legislative Exchange Council (ALEC), have provided insights on how red state lawmakers can tackle AI oversight.
On October 28, 2024, a draft AI bill known as the Texas Responsible AI Governance Act (TRAIGA) was released. The draft law, which is expected to be introduced in the 2025 legislative session, has been viewed as a blueprint for red state lawmakers that are interested in overseeing AI in the private sector. Key obligations under the draft bill include:
Impact assessments: While not a novel requirement under most AI frameworks, the TRAIGA would require companies to conduct semiannual assessments, which stands in contrast to annual assessments.
Social Media Companies: The draft law also imposes obligations on social media companies to make a commercially reasonable effort to ensure that advertisers deploying high-risk AI systems on their platforms are not exposing users to algorithmic discrimination.
Reporting Timelines: Similar to the Colorado AI law, there is a reporting timeline for companies that identify instances of algorithmic discrimination. However, the reporting timeline is 10 days as opposed to the 90 days afforded under Colorado’s AI law.
Prohibited Use Cases: Certain use cases, like emotional recognition systems, are prohibited under the TRAIGA. The full scope of prohibited systems in the proposed bill mirror those prohibited by the EU AI Act.
In addition to Texas, the non-profit organization ALEC finalized a Model AI Bill that provides a roadmap for states to oversee public sector AI usage. The model bill aligns with some state-level AI Executive Orders, which seek to create and maintain an inventory of AI used by state agencies. The model legislation proposes an “AI Policy Office” that handles AI-related rules and oversight. That office would sit within a state’s Attorney General’s office. The model law also aims to analyze current state AI regulations and identify gaps with those regulations, in so much as existing law could not “prevent or redress substantial, non-speculative, concrete, and redressable harm.” This proposes a different approach from the more general “reasonably foreseeable” harm or risks approach.
Our Take: Both pieces of proposed legislation could be a model for other red state lawmakers that are interested in trying to carve a different path on AI regulation or oversight. However, when comparing the scope and obligations to other AI regulatory frameworks (e.g., Colorado or Connecticut) there is not much substantive difference.
—
4. Emerging Risks of Agentic AI
The Open Web Application Security Project (OWASP) is currently developing a list of the Top Ten Risks for Agentic AI (see their top 10 list for LLM Applications here). Many of these Agentic AI risks are more complex and exploit traditional LLM vulnerabilities as building blocks. For example, Excessive Agency (i.e. the LLM being granted loose permissions) is a building block for new Agent Critical System Interaction, Agent Orchestration and Multi-Agent Exploitation and Agent Authorization and Control Hijacking risks. In each of these risk scenarios, agents are granted excessive agency and leveraged by a malicious party to manipulate other systems, including other agents. Similarly, Prompt Injections and Data Poisoning are used to manipulate an agent's instructions or memory (insert links here). Unlike the LLM risks, this OWASP list is focused on more theoretical attacks that have not been associated with publicized attacks. However, with rapid AI development and the widespread accessibility of AI tools, it is important to understand these vulnerabilities before they materialize.
In terms of recommended mitigations, key themes center around permission controls and logging. To prevent many of the outlined attacks, Agentic AI should be given the minimum permissions necessary, as well as given escalated permissions only for the duration of a particular task (e.g. the Agentic AI is only able to send an email given a specific trigger). As organizations deploy Agentic AI, they will need to find the balance between the set of tasks the system can do and increased security risks. For logging, it is necessary to log all activity executed by AI Agents (not just system inputs and outputs), including changes in permissions; the logging needs to be paired with monitoring and anomaly detection. Overall, many of the mitigations parallel concepts from traditional software security, but a set of practical best practices is still being developed.
Key Takeaway: General risks associated with Generative AI have received much attention from security researchers over the last year, while Agentic AI systems are just now being given deeper attention. Key controls for Agentic AI will involve a close review of permissions given to these systems and in-depth logging and monitoring.
*********
As always, we welcome your feedback on content and how to improve this newsletter!
AI Responsibly,
- Trustible team