Book Reviews | 10:32 AM
Excerpt from The AI Instinct: The Future of AI and Human Decision-Making by Rana Gujral.
Roles, Accountability, and Decision-making
A major question that we need to keep in mind as we move forward is, “What do we want human–AI co-intelligence to become and how do we design for dignity, agency, and flourishing?”
The answer to this question is something we have already touched on: the importance of role clarity. Bostrom continuously emphasizes that governance and rules matter deeply here: intelligence without aligned aims and constraints can be catastrophic.
His orthogonality thesis and instrumental convergence arguments implicitly demand that humans remain in the loop so that AI goals don’t diverge dangerously from human values.
Humans must be the ones to define goals and values, and they must continue to be the ones that provide oversight that these are being reached and address issues as they arise if the path misaligns from values outlined.
Ben Shneiderman’s Human-Centered AI (HCAI) Principles advocate for AI systems that augment, amplify, empower, and enhance humans rather than replace them. He proposes practical design controls like audit trails, supervised autonomy, human override, and transparency so users can see evidence and options.
In this regard, trustworthy AI will preserve human control while offering high levels of automation.
Mismatched accountability can occur if this is not a priority. The first are moral crumple zones, which describe how responsibility for failures in complex human–AI or human–robot systems are often misattributed to a human operator who had limited control. Humans naturally, as the leaders of the projects and the highest intelligence, will absorb blame for the faults or unpredictability of AI systems.
When designing these systems, we want to avoid scenarios where humans are unfairly penalized or held accountable for AI’s errors or unintended behavior.
Another risk is automation bias, which happens when accountability is unclear and humans over-trust AI outputs because they believe the system “knows better.” This can lead to signing off on AI recommendations even when they are flawed.
The human is officially accountable but in practice defers judgment.
Deskilling and loss of human agency are also risks that come about when humans are nominally responsible but rarely intervene. When this happens, they may lose the skills needed to step in effectively when AI fails. In aviation, this is known as automation complacency.
When a crisis arises, the human may be unable to act and still be blamed afterward.
In complex systems, when outcomes go wrong, neither humans nor organizations can clearly be held responsible. This is called an accountability gap. The AI is seen as a “black box,” and the human overseer may not have had real control.
When accountability doesn’t match control, courts and regulators struggle to assign liability. This slows down governance and may leave victims without justice. It also risks backlash that may stifle beneficial AI innovation while creating a dangerous vacuum for legal and ethical responsibility.
When humans are told that they are responsible but cannot meaningfully influence outcomes, the conflict creates a psychological burden on operators. This can create stress, guilt, and burnout. For example, drone operators or clinicians using high-stakes AI tools may feel responsible for harm caused by system errors even though they had little say in the decision.
Misaligned incentives happen when organizations shift liability onto individuals while reaping the efficiency gains of AI automation. This discourages reporting of problems, slows adoption of better practices, and erodes trust between humans and institutions.
Further, there can be strategic exploitation if accountability is blurred, as companies or states may design systems deliberately to offload blame onto humans, shielding themselves from responsibility. This undermines both safety and fairness.
Air France Flight 447, an Airbus A330-200, disappeared on 1 June 2009 over the Atlantic Ocean as it was flying from Rio de Janeiro to Paris. The aircraft entered an aerodynamic stall that it never recovered from, and all 228 people on board the aircraft perished when the plane went down.
The tragedy shocked the aviation community and became a defining case in understanding hidden vulnerabilities of automated flight systems mixed with human factors. This example shows how automation masking and system complexity can erode pilot situational awareness and decision-making.
The incident reinforces the principle that while automation enhances safety under normal conditions, it must be designed to empower, not disempower, human operators during failures.
Pilots are nominally responsible for flights, but highly automated cockpits reduce active engagement. In this scenario, when the autopilot disengaged due to faulty sensor readings, the crew was suddenly responsible for manual control under extreme stress.
Their degraded manual flying skills, combined with lack of situational awareness, led to a fatal crash. While not a specific AI example, this illustrates deskilling and automation complacency; humans bore the blame but lacked readiness to take over.
Tesla markets its cars as having a reliable autopilot and “Full Self-Driving (supervised)” systems, but legally the driver remains responsible, although the manufacturer may be liable if a system defect or failure contributed to the crash.
Marketing, understanding, and other factors have created accountability gaps: when crashes occur, investigations often reveal that drivers were over-relying on automation. This shows how misaligned incentives and strategic exploitation let companies enjoy AI’s benefits while pushing blame onto humans who were given incomplete control.
In 2022, a California man was charged with vehicular manslaughter after his Tesla, operating in Autopilot mode, ran a red light and caused a fatal crash that killed two people. He became the first person in the U.S. to face criminal charges associated with a collision involving a semiautomated driving system.
Tesla defended its position by stating that drivers must always remain in control, regardless of whether Autopilot is engaged.
The driver was ordered to pay more than $23,000 in restitution for the deaths of two people, a decision announced the same day that Tesla recalled nearly all vehicles sold in the U.S. to update software and fix a defective system that’s supposed to ensure drivers are paying attention when using Autopilot.
Systems like COMPAS in the United States assign risk scores for recidivism and are heavily relied upon by judges, who retain formal responsibility for sentencing. This produces accountability gaps (for example, if an analysis is biased) and legal/ethical uncertainty, since neither the human decision-maker nor the AI developer can be held clearly liable.
This was a similar issue for IBM Watson for Oncology (covered in Chapter 3), and one of the reasons why it is no longer used by doctors to aid in diagnosis and provide treatment maps. If harm occurred, the doctor was blamed, even if the system’s opacity or errors made meaningful oversight impossible.
****
Excerpted with permission from the publisher, Wiley, from The AI Instinct: The Future of AI and Human Decision-Making by Rana Gujral. Copyright © 2026 by John Wiley & Sons, Inc. All rights reserved. This book is available wherever books and eBooks are sold.
Bio
Rana Gujral builds at the frontier of Artificial Intelligence, Cognitive AI, and the long-horizon path toward Artificial General Intelligence. As the former CEO of Behavioral Signals, he helped turn advances in speech-signal processing and representation learning into production systems that infer intent, emotion, and deception risk from voice, deployed in financial services, defense, and trust-and-safety contexts.
Earlier, he founded TiZE, acquired by Alchemy. He then led Smart Home at Logitech, and later drove the turnaround at Cricut that returned the business from near-bankruptcy to profitable growth and set the path to its $4.4B IPO.
His TEDx talk has surpassed 1M+ views. He has delivered keynotes at the World Government Summit, World Economic Forum, the World AI Cannes Festival, /function1 Dubai (opening keynote, 10,000+ attendees), The Next Web, and LEAP. Named an “AI Entrepreneur to Watch” by Inc., “Most Influential CEO” by CEO Monthly, and “Top 10 Entrepreneurs to Follow” by Huffington Post.
Technical limitations
If you are reading this story through a third party distribution channel and you cannot see the book cover included, we apologise, but technical limitations are to blame.
Find out why FNArena subscribers like the service so much: “Your Feedback (Thank You)” – Warning this story contains unashamedly positive feedback on the service provided.
FNArena is proud about its track record and past achievements: Ten Years On