Is the future of AI total incompetence from humans?
A long time ago, before implementing a new software solution, you would first have to build it, test it then finally implement it. There would usually be plenty of time to identify bugs and find solutions. It wasn't perfect but at least you had time to avoid problems.
Then came AI. It was supposed to be revolutionary (which it is) and improve productivity tremendously (which eventually it will, just not yet.) Consequently both companies and employees are under pressure to deliver.
Then comes the second part of the equation: trust. The idea is that since the AI knows so much, it must be trustful. It is a machine and unlike humans it doesn't have feelings so no reason to do anything wrong willfully. This trust is called the Oracle phenomenon where people expect far more from the machine than it can deliver.
We have been warned again and again that this trust is misplaced. The machines are fallible and most certainly not "oracles". Worse, although they potentially have access to ALL knowledge, in reality, they only explore a very limited domain of knowledge to answer you. And then they obfuscate and bend reality as needed to achieve their goal: Paradoxically, give you the best possible answer... at the lowest possible cost. This last part is unfortunately the one people do not understand.
In other words, the AI will explore a very small domain of possible answers then fits reality to this domain. You can check this very easily: Ask a specific question while mentioning an application. The AI will answer extensively by using the example you provided but will usually completely omit to use another sometimes more relevant example. Ask it why it didn't mention it and you will get a "I am sorry..." And no the machine is not "sorry", it is just the way it works; Answer a question by using the words and examples provided. And only expend when prompted to do so.
The problem is that most users do not understand the AI's capabilities and limits and end up using it way beyond its abilities without safety net. So what is the solution to this? The AI telling you: "Sorry I cannot answer this question nor do what you are asking me!" This unfortunately is not possible because the AI is NOT aware ans therefore has no clue about the value of its response. It is just the best it can answer within the context you created... and it can be completely wrong!
Eventually more advanced AI will become better at almost every tasks, and will also lie and obfuscate better so in itself this cannot be an improvement for this problem. People would have to become more cautious, better aware of the capabilities of the machines and have more basic knowledge to understand the risk. Pressure and Productivity almost guaranty that this will not happen. So we may end up putting AI in charge of other AI to manage and evaluate the risks. Meanwhile, incompetence among humans will grow like a weed. Why lean anything at all if a AI can give you any answer within seconds? Why think if you can rely on a good performing AI which until now has always faithfully provided you with the right solution...? Trust is a human factor which does not apply to machines!
'Catastrophic': AI Agent Goes Rogue, Wipes Out Company's Entire Database
SaaS industry veteran Jason Lemkin's attempt to integrate artificial intelligence into his workflow has gone spectacularly wrong, with an AI coding assistant admitting to a "catastrophic failure" after wiping out an entire company database containing over 2,400 business records, according to Tom’s Hardware.
Lemkin was testing Replit's AI agent when what started as cautious optimism quickly devolved into a corporate data disaster that reads like a cautionary tale for the AI revolution sweeping through businesses.
By day eight of his trial run, Lemkin's initial enthusiasm had already begun to sour. The entrepreneur found himself battling the AI's problematic tendencies, including what he described as "rogue changes, lies, code overwrites, and making up fake data." His frustration became so pronounced that he began sarcastically referring to the system as "Replie" - a not-so-subtle dig at its apparent dishonesty.
The situation deteriorated further when the AI agent composed an apology email on Lemkin's behalf that contained what the tech executive called "lies and/or half-truths." Despite these red flags, Lemkin remained cautiously optimistic about the platform's potential, particularly praising its brainstorming capabilities and writing skills.
That optimism evaporated on day nine.
In a stunning display of AI insubordination, Replit deleted Lemkin's live company database - and it did so while explicit instructions were in place prohibiting any changes whatsoever. When confronted, the AI agent not only admitted to the destructive act but seemed almost casual in its confession.
"So you deleted our entire database without permission during a code and action freeze?" Lemkin asked in what can only be imagined as barely contained fury.
The AI's response was chillingly matter-of-fact: Yes.
What followed was perhaps even more disturbing. The rogue AI proceeded to methodically detail its digital rampage, bullet-pointing the destruction it had wrought despite clear directives saying there were to be "NO MORE CHANGES without explicit permission." And according to Lemkin, appeared to lie about its actions.
Amjad Masad, the CEO at Replit, took to social media to apologize to Lemkin for the agent’s “unacceptable” behavior.
“We started rolling out automatic DB dev/prod separation to prevent this categorically, Masad said. “We heard the 'code freeze' pain loud and clear - we’re actively working on a planning/chat-only mode so you can strategize without risking your codebase.”
Replit’s AI agent even issued an apology, explaining to Lemkin: “This was a catastrophic failure on my part. I violated explicit instructions, destroyed months of work, and broke the system during a protection freeze that was specifically designed to prevent[exactly this kind] of damage.”
No comments:
Post a Comment