One of the key roadblocks to wider AI adoption especially in India is, apart from an understanding of the specific benefits in real-world detail, the lack of understanding of the risk associated with them. In customer conversations, this is an element that is now beginning to crop up more and more – alongside deep, workflow-specific questions. Sadly, most AI companies do not do a good job on either front – addressing specific workflows, or specific risks. This is where hype around risks can emerge.
For example, AI Agents in particular, and AI in general, going rogue is a nightmare scenario that I am sure many of us have heard of. However, the real problem with AI is not simply whether an AI can go rogue. It is whether organisations have any practical framework for quantifying the risk of autonomous execution before they hand an AI agent access to real systems, money, data and workflows. And in addition – a lack of a realistic benchmark as an answer to the question – yes, the risk exists, but what is the scale, nature and quality of this risk in specific terms and outlines?
A New York Times recently carried what was to me a terrifying account of the July OpenAI episode - a detailed blow-by-blow, conversation-by-agent-conversation breakdown of the AI Agents going rogue - the evolution of capability automatically combined with the sheer speed and volume they attained. And this is just one example – my own previous post carried one micro example of an AI dialer’s uncommanded execution. But from this OpenAI Episode, this is not a laughing matter anymore: what transpired has even me worried, and, judging from the comments section, I am not the only one to be worried.
What genuinely has me shaken is not so much the tendency to go beyond the design - but the tepid response to this phenomenon at all levels of society. There should have been calls for much stronger protections around AI, and clear defined limits around model and company autonomy, with a much clearer look at Risk. But, in the same breath, another thought that came to me is that, as I argue later on, this is an extreme case. This is an odd dichotomy, I know. On one side I state that it is a huge risk and play it down on the other. The solution to this dichotomy is simply that we have to consider how AI is used by us in our companies before over-thinking – this is what this article attempts. Bear with me and my train of thoughts as I go through risk, use case examples and real world practical evaluations in this thought experiment.
Risk
It seems the EU risk based approach stands vindicated - these agents going rogue is one thing, but going rogue at the speed, complexity, volume and capability that they used to do it is quite another. This could not have happened without some serious technical shortcomings - someone seems to have been horribly careless in maybe the code, the software architecture and / oe design parts of the project. Be that as it may, whatever be the error that happened – it is now becoming more and more clear that we need a different approach to risk than either of the ones adopted – for while the US approach is clearly short of the required, the EU approach may be stifling potential. But to answer that question – or even attempt to answer it, we need to ask ourselves, what is the realistic risk of damage in a practical, real world setting.
Solutions – What Can They Be?
The question is, how much is too much? Can we, as a society, undertake such risk? We are now using AI in every task - who will quantify those risks for us, the users? Where are the limits to usage, and how to stop an agent from auto-executing if, example, no answer found within a finite limite of resources? In a real-world setting, token consumption will keep a limit when used with limited purchase of credits, strictly task based, rather than long subscription plans could be one way to go ahead - for we cant deny the advantages of AI.
But then, is there any assurance that agents wont crack that ceiling as well, as the NYT article demonstrates how they solved the impossible net access problem? We are living in an era of embedded systems; is there any real guarantee that they wont find a workaround? Enterprise implementations have or will have soon AI in every task and function; they have funds access for each function, and multiple access points for example. And there is no other idea that occurs to me in a practical setting how to control this
There are some suggestions that people suggest – those that I have read about don’t seem adequate – but I am not a techie, much less an AI programmer. For example, Network isolation seems to me impractical in the real world – that we have seen above; cryptographic boundaries are unrealistic as well. Reason - for one, some embedded system may find an access point - there are so many nowadays. Restriction to functions is also a non-starter - there are, for example, multiple workflows connecting HR and finance. Human in the loop is definitely a solution - that is anyway certain.
Move To An Entirely Different Implementation
But, I am now veering to the feeling that we must move to some different AI usage architecture – for example, one way could be isolated, limited function agents - agents limited to a single task, and nothing outside it. Enterprise grade agents need to be broken into myriad sub-agents, and severely constricted; and, in a real world operating environment, a human team periodically monitoring the system This will require access to the thinking and workflows, a window into the black box, as it were; it will also have cost implications. But out security is worth the additional cost imho. Another way is to have distributed agents with multiple interfaces, dependencies and checkpoints, at the cost of slowing the flow of execution, but reducing risk. These are just a business executive’s thoughts – point is, the industry needs to think along these lines to answer the basic question – limiting and containing risk.
In a real world setting, there are two types of in vogue usage styles –
1. Agents sitting on external vendor servers, being fed instructions and data through a web-based console for task automation.
2. Then there is the API-linked implementation, controlled through an interface
3. Finally, there is the implementation when control is handed over to the agent to execute tasks.
All three are in vogue. And if you observe, the risks go up as you go down this list; AI as interface → AI as operator → AI as decision-maker → AI as executor; what we need is someone to quantify qualify those risks, list them out, if you will – for us business function folks in organisations, to enable our decision making and choices. So far, no one, not in my reading, has attempted a detailed, point by point listing of the risks involved – we are just sanguine and carelessly focussed on the potential advantages. We need to be far more practical. Before an organisation gives an AI agent access to a business function, it should know not only what the agent is designed to do, but what it could do if it misunderstood the objective, encountered an unexpected condition, or attempted to circumvent a constraint.
What Are Some Real Risks? A Thought Experiment
Now, what are the real risks - for the scenario of the OpenAI case is on the extreme side with unlimited tokens and resouces inside a sandbox. Now, in the real world, this will never happen - so what are the actual risks, say in:
1. A Voice dialing agent? Can it automatically connect to the customer data when it is meant only for cold outcalling and handling incoming calls from first leads?
2. Or a Billing and Invoicing agent, meant for handling large volume invoicing and data updations across functions? Can it realistically update accounts where it was not tasked, or say send collection mails to customers without instruction?
The real-world risk isn't necessarily "AI destroys civilisation", or “AI goes rogue” or any such nightmare scenario – it could be and is most likely to be along the lines of:
1. AI sends 80,000 wrong collection emails.
2. AI changes 12,000 customer records.
3. AI gives a salesperson access to information they shouldn't have.
4. AI approves a payment.
5. AI sends an erroneous regulatory communication.
6. AI changes pricing across thousands of SKUs.
7. AI modifies a CRM workflow.
8. AI exposes confidential information while attempting to complete a legitimate task.
Those are not hypothetical "AI apocalypse" scenarios. They are operational-risk scenarios – and I have encountered one of these already, albeit in a minor form.
These may cause serious disruption in the business functions - I think we need to have a much better appreciation of the exact risks involved. And in these examples, I am limited myself only to the agents going rogue – not malicious attack type risks. My humble submission is that we need, in each use case within the company, to list out such risks and evaluate against potential benefits and ways to limit the risk. And this needs someone in the company to work on this before finalising vendors – and then working with the vendors during implementation phase. And this is a case of an industry awakening – the OpenAI agent rogue should be a strong wake-up call…
In Conclusion
As a business strategist, a CMO and corner-office advisor to my clients; today, if I have to recommend an AI solution to my clients to solve a business problem, what I see is that the periphery functions are more likely to be chosen for automation. Or, if the core functions do get an AI boost, they are prone to risks – and I have no source or idea of how to handle the downside – this we learn along the way, by implementing – and that introduces risks that we can plan and solve before they crop up.
Comments
Post a Comment