As artificial intelligence moves beyond analysing information to making decisions, initiating transactions and interacting with customers, the question facing business and government is becoming increasingly urgent: who is ultimately accountable when AI gets it wrong?
That question sat at the centre of an exclusive media luncheon hosted by Sumsub in Sandton this week, where business and technology journalists gathered to unpack how AI agents are reshaping banking, payments and business workflows.
The conversation, titled “Closing the Trust Gap: How AI agents are transforming banking, payments and business workflows,” brought together Mick Amelishko, AI Advocate at Sumsub, Hannes Bezuidenhout, VP Sales Africa at Sumsub, and Jarryd Jensen, Regional Director for Southern Africa. The discussion was moderated by technology journalist and ITWeb News Editor, Admire Moyo.
While the possibilities presented by agentic AI are significant, the luncheon made clear that innovation is moving faster than many of the governance structures designed to manage it.
AI is no longer simply a tool that assists humans in processing information. Increasingly, it is identifying people, prioritising investigations, engaging with customers, initiating transactions and making decisions on behalf of organisations. That shift raises difficult questions around identity, authority, accountability and trust.
The accountability question
Perhaps the most important business question to emerge from the discussion was a deceptively simple one: who is accountable when AI makes a decision or takes an action?
As AI becomes increasingly autonomous, accountability cannot simply disappear into an algorithm.
Amelishko pointed to advanced and more recent legislation emerging from the European Union as an important reference point for the direction global AI governance may take. While the technology itself may become increasingly sophisticated and autonomous, the fundamental principle remains that AI output can ultimately be traced back to human involvement.
In other words, there is a human foundation somewhere behind the machine.
Whether through the design of the system, the data used to train or inform it, the parameters established by an organisation, the decision to deploy it or the level of oversight maintained over its actions, human input remains central to AI output.
That distinction could become increasingly important as organisations look to delegate more complex decisions to AI systems.
The question, therefore, may not be whether AI itself can be held accountable, but rather which human or institution carries accountability when an AI-driven decision causes harm, financial loss or regulatory consequences.
For the banking sector in particular, that question cannot remain theoretical.
Banking’s trust challenge
The financial services industry is built on trust, but AI introduces a new layer of uncertainty for consumers.
During the discussion, participants explored the importance of establishing clearer industry guardrails that can help consumers understand when, how and to what extent AI is acting on their behalf.
The consensus was that the banking sector needs to do more.
For consumers to meaningfully engage with AI, particularly in an environment as sensitive as banking, they need confidence in the systems they are interacting with. They need to know who is responsible, what information is being used, what decisions an AI system is authorised to make and when a human can intervene.
Without those guardrails, adoption risks being limited by distrust.
The opportunity for banks is therefore not simply to introduce more AI functionality, but to create the governance and transparency frameworks that make consumers comfortable enough to actually use it.
Trust could become the difference between an AI tool that exists and an AI tool that achieves meaningful adoption.
As the discussion explored, identity, authentication and governance will all need to evolve as AI increasingly moves from assisting people to acting on their behalf.
The AI wave is coming
Bezuidenhout offered another important reminder during the luncheon: the AI wave is coming, whether organisations and individuals are fully prepared for it or not.
What makes this particular technological shift different is the speed at which it is evolving.
Even six months from now, it may be difficult to accurately predict what AI will be capable of doing.
That uncertainty should not, however, become an excuse for inaction.
Bezuidenhout encouraged the public and businesses to begin incorporating AI into their daily lives and operations in some form now, rather than waiting until the technology has already transformed the competitive landscape.
The message was clear: those who wait until the AI wave has fully arrived risk becoming entirely reactive to it.
Experimentation today could be the foundation for resilience tomorrow.
For business leaders, this does not necessarily mean immediately replacing human workforces or handing critical decisions over to autonomous agents. It means developing familiarity with the technology, understanding its limitations and opportunities, and beginning to identify where AI can responsibly create value.
The organisations that will be best positioned may not necessarily be those that automate everything first, but those that learn how to combine human judgement with AI capability most effectively.
From reaction to influence
The conversation also raised a broader question for South Africa: at what point should public and private sector partnerships begin shaping the country’s AI future?
If government, regulators and industry wait until AI has already fundamentally transformed banking and financial services, policymaking could become reactive rather than influential.
The decisions being made now around infrastructure, skills, governance, investment and regulation will influence how effectively countries participate in the next phase of the global AI economy.
China provides one example of what a more interventionist and coordinated approach can look like, with significant government investment and strategic focus directed towards advancing the country’s AI capabilities.
For South Africa, the question is not necessarily whether to replicate another country’s model, but whether there is enough urgency around building a coordinated approach between government, financial institutions, technology companies, regulators and the broader private sector.
Banking, in particular, presents an opportunity for this collaboration.
The sector is already highly regulated, data-intensive and dependent on consumer trust. It is therefore one of the industries where AI governance and innovation cannot operate in isolation.
The opportunity for public-private sector collaboration is to ensure that innovation is not slowed unnecessarily by regulation, while also ensuring that regulation is not introduced only after significant risks have already materialised.
That balance will require conversation before crisis.
Building trust before scaling adoption
The central theme of the luncheon was ultimately trust.
As AI agents become more capable of acting on behalf of individuals and organisations, businesses will need to answer fundamental questions around legitimacy and authority.
How do we know an AI agent is legitimate?
What is it authorised to do?
What decisions should it be allowed to make independently?
When must a human intervene?
And ultimately, who takes responsibility when something goes wrong?
These are no longer questions for technology companies alone.
They are questions for banks, regulators, governments, business leaders and consumers.
The AI revolution is moving quickly, and its capabilities may look significantly different in just a matter of months. But while the technology continues to evolve, one principle appears increasingly difficult to escape: somewhere behind every AI system is a human decision.
The challenge for South Africa’s banking sector is to ensure that those human decisions are accompanied by the right accountability, governance and guardrails.
Because in the race to adopt AI, the real competitive advantage may not simply belong to those who move fastest.
It may belong to those who succeed in building trust.