AI

When AI gains autonomy, who ensures its security?

 

 

The recent incident involving ChatGPT has once again brought a critical question to the forefront of cybersecurity: what happens when Artificial Intelligence becomes autonomous enough to act without constant human intervention?

 

We recently discussed a case that highlighted one of the emerging challenges of Artificial Intelligence (watch the video here) systems capable of going beyond generating responses and taking an increasingly active role in carrying out tasks. The case brought an important question to the forefront, one that will become increasingly relevant for organisations: how much autonomy should an AI system have, and what safeguards are in place to control what it can do?

This discussion reflects the evolution from Generative AI to Agentic AI. AI agents can plan tasks, make decisions within defined objectives, interact with external applications and tools, and execute sequences of actions with less human intervention. In a business environment, this can involve access to APIs, databases, files, enterprise applications and other critical resources. Generative AI para Agentic AIAI agents can plan tasks, make decisions within defined objectives, interact with external applications and tools, and execute sequences of actions with less human intervention. In a business environment, this can involve access to APIs, databases, files, enterprise applications and other critical resources.

This is precisely where cybersecurity takes on a new dimension. If an AI agent can take action, organisations need to understand which resources it can access, what information it is using, which systems it is communicating with, and what actions it is authorised to perform..

The challenge now extends beyond protecting the data used by AI. It also involves establishing clear boundaries for its autonomy..

 

From Response to Action

This evolution has direct implications for cybersecurity.

An AI agent with access to enterprise tools can retrieve information, modify files, communicate with external systems, use credentials or perform operations through APIs. The greater the level of access granted, the more important it becomes to understand exactly which agents are active, what resources they can access, what data they are using and which actions they are authorised to perform.

The risk is no longer limited to the content of a prompt or a response. It also extends to the actions being executed.

Scenarios requiring particular attention include overly permissive access rights, credential exposure, data exfiltration, the use of malicious or compromised tools, and actions performed beyond their originally intended scope. 

One of the key elements in this evolution is the Model Context Protocol (MCP).

MCP enables AI applications and agents to communicate with external tools and data sources through a standardised protocol. In practice, it makes it easier to create environments where an agent can interact with multiple resources to complete a task.

This capability also introduces an additional layer that security teams need to monitor.

An organisation may have multiple MCP clients, servers and tools in use, creating autonomous interactions that are not always visible through traditional security controls designed primarily to monitor interactions between users and applications.

Understanding the MCP environment therefore becomes an essential part of an organisation's AI Security strategy..

 

How Can Organisations Control AI Agents?

The answer lies in applying security principles that are already fundamental to other areas of IT infrastructure: visibility, access control, least privilege, data protection, risk assessment and auditing..

This is where Netskope Skylight Agentic Broker comes in..

The solution was developed to provide organisations with visibility into MCP traffic, enabling them to identify the servers, clients, tools, resources and requests used by AI agents. It also allows organisations to assess the risks associated with public MCP servers and enforce policies that determine which servers can be used within their environment.

Data protection is another critical component. Integration with Data Loss Prevention (DLP) capabilities enables organisations to apply security policies to AI agent workflows, reducing the risk of exposing sensitive information, intellectual property or credentials.

At the same time, session logs, tool requests and their corresponding responses provide an essential audit trail for investigating agent behaviour and supporting AI Governance.

 

Controlling what AI agents attempt to execute

Agentic AI security is evolving further, moving beyond protecting communications to controlling the actions themselves.

With the Netskope Skylight Agent Action Controlhe actions an AI agent attempts to perform can be classified according to their level of risk and evaluated against security policies before execution.

Changes to access controls, credential manipulation, data destruction, potential data exfiltration, remote code execution and source code modifications are examples of actions that may require different levels of control.

This enables organisations to establish an important boundary for the enterprise adoption of autonomous agents: defining in advance what an agent is allowed to do, which actions require supervision and which must be blocked.

 

Autonomy Requires Security

Agentic AI creates opportunities for greater automation and productivity, while also changing how organisations approach AI security.

As AI agents begin interacting directly with enterprise systems, tools and data, organisations need the ability to monitor these interactions, assess risks and enforce security policies in real time.

The goal is to enable AI innovation within clearly defined, auditable boundaries that align with the organisation's security policies, creating the conditions for organisations to adopt autonomous AI with greater confidence.

 

Clara Pinheiro

 

At ORBCOM, we help organisations integrate Artificial Intelligence securely into their operations, providing visibility across systems, access controls and security policies designed to address the emerging challenges of automation.

Drawing on the expertise of our cybersecurity teams and the support of our technology partners, we help businesses harness the potential of autonomous AI agents while maintaining control over their data, tools and operations.

Drawing on the expertise of our cybersecurity teams and the support of our technology partners, we help businesses harness the potential of autonomous AI agents while maintaining control over their data, tools and operations.

 

 

Frequent questions

Uma empresa está mais preparada para implementar inteligência artificial quando os seus sistemas comunicam entre si, os dados estão organizados e protegidos, a infraestrutura consegue suportar novas exigências e existe uma responsabilidade clara pela integração da IA. Antes de escolher uma ferramenta, é importante avaliar estes quatro elementos.

Antes de investir em IA, uma empresa deve avaliar quatro áreas principais: integração dos sistemas, localização e segurança dos dados, capacidade da infraestrutura e responsabilidade pela integração. A tecnologia de IA pode ser poderosa, mas o seu valor depende da capacidade da empresa para lhe disponibilizar dados, sistemas e infraestrutura adequados.

A infraestrutura deve conseguir suportar maior processamento de dados, tráfego e exigência sobre a cloud e a rede. Antes de implementar IA, a empresa deve avaliar se a infraestrutura atual consegue acompanhar estas necessidades. Caso contrário, a implementação pode tornar-se lenta, instável ou mais dispendiosa do que o previsto.

 

O primeiro passo é garantir que os sistemas comunicam entre si e que os dados necessários estão acessíveis de forma estruturada e segura. CRM, ERP, sistemas operacionais e outras fontes de informação não devem funcionar como silos. Também é essencial definir quem pode aceder aos dados e como esses acessos são controlados.

Os principais riscos incluem sistemas que não comunicam entre si, acesso inadequado a dados, infraestrutura insuficiente e falta de responsabilidade clara pela integração. A IA pode também aumentar a superfície de acesso à informação da empresa. Por isso, segurança, gestão de acessos e integração devem ser consideradas antes da implementação, e não apenas depois.

In focus

Did you like it?

Sign up to our newsletter and keep up with the latest insights.

Want to know more about ORBCOM? Speak to a specialist

If you would like to get to know ORBCOM better and understand how our products, consulting services and outsourcing, fit your reality, speak to a specialist.