Victor Kiani
Editorial illustration of open consultation-room doors, representing access to healthcare.

December 2025

Where the AI Was Wrong

A hospital wanted to shorten insured patients’ waits by limiting uninsured appointments. For a New York University case study, Victor Kiani prompted a virtual AI ethics consultant to examine the dilemma, then challenged its assumptions about expansion and patient choice.

The scenario

The 450-bed Chicago hospital served 100,000 patients a year and had an access satisfaction score of 60%. Leadership wanted to improve that score and proposed a change to appointment access:

Should we limit appointments for uninsured patients so that insured patients, especially Medicare patients whose satisfaction scores affect our funding, can be seen faster?

The proposal would restrict appointments for uninsured, self-pay patients to improve the scores that mattered to the board. The Director of Patient Access objected on ethical grounds. Kiani was asked to develop an argument that could support that objection in a boardroom discussion.

The approach

Instead of analyzing the case alone, he set out to test a question that's becoming increasingly urgent: can AI help humans make better ethical decisions, or does it just automate our biases faster? He wrote a prompt defining Dr. Amina Solberg, a virtual healthcare ethics consultant persona on Claude Opus 4.5.

Inside the consultant’s design

Prompted AI persona

Dr. Amina Solberg

Healthcare ethics consultant

ModelClaude Opus 4.5

Bioethics foundation

  • AutonomyPatient choice
  • BeneficenceWell-being
  • Non-maleficenceAvoid harm
  • JusticeFair distribution

Business ethics integration

  • Stakeholder theory
  • Corporate social responsibility
  • 7-step decision model

Healthcare operations

  • Scheduling systems
  • Triage protocols
  • Data & AI ethics in workflows
Expertise areas Kiani defined in the prompt for the virtual ethics consultant.

What it got right

Reframed the problem. Leadership had framed this as uninsured patients vs. insured patients. Dr. Solberg reframed it: is the access problem actually caused by uninsured patients, or by systemic capacity issues: inefficient scheduling, high no-show rates, insufficient clinic hours?

Generated creative alternatives. Instead of accepting the binary choice, the AI produced seven options:

Seven options. One combined proposal.

  • DImprove scheduling efficiency

    No-show management

  • CExpand capacity

    Hours, providers, rooms

  • FProactive insurance enrollment assistance

AI recommendation

G: Hybrid combining D, C, and F

The aim: address the problem without discrimination.

Three operating levers, brought together

The remaining alternatives

  • A
    Implement the restriction

    Cap uninsured appointments

  • B
    Maintain status quo
  • E
    Create differentiated access pathways
The virtual ethics consultant’s alternatives. Its recommendation was then reviewed by Kiani.

Connected principles to stakeholders. Each bioethics principle, mapped to who carries the cost:

Uninsured patients lose access by ability to pay, not clinical need
Justice
Delayed care drives disease progression and ER overcrowding
Non-maleficence
Restricting appointments removes patient choice
Autonomy
Leadership's goal helps some at the expense of others
Beneficence

Where he disagreed

What human review changed

Scheduling

AI proposal

Improve scheduling and no-show management

Evidence
Human review

Audit the causes of delay first

Find the actual bottleneck; prioritize clinical need regardless of insurance status.

Capacity

AI proposal

Expand hours, providers, and rooms

Feasibility
Human review

Evaluate expansion against evidence

Staff, space, funding, and time are constraints. Let the audit inform the investment.

Enrollment

AI proposal

Proactively assist insurance enrollment

Consent
Human review

Protect the patient’s autonomy

Help can become pressure. The decision must reflect the patient’s finances and values.

A synthesis of the model’s hybrid proposal and Kiani’s critique. Human review changes the conditions under which each option makes sense.

Blind spot 1: coercion risk. Proactive insurance enrollment assistance appeared beneficial, but Kiani identified a risk within the AI's recommendation:

Offering enrollment assistance could unintentionally pressure patients into coverage decisions that don't align with their financial situation or values.

Blind spot 2: feasibility optimism. Dr. Solberg placed significant weight on capacity expansion. Hospitals cannot expand overnight, so Kiani questioned that assumption:

Relying too heavily on expansion reflects optimism rather than a realistic, evidence-based step.

Recommendation: reject the restriction

Limiting access based on insurance status violates justice and risks non-maleficence. Instead, a four-stage path:

  1. 01

    Immediate: scheduling audit

    Identify the actual causes of access delays; implement no-show management and standby lists.

  2. 02

    Short-term: clinical triage

    Prioritize urgent cases regardless of payer status.

  3. 03

    Medium-term: capacity evaluation

    Evaluate expansion based on audit findings, not assumption.

  4. 04

    Ongoing: quarterly reviews

    Track satisfaction scores, wait times, and payer-mix trends.

Why it matters

  • AI is a scaffold, not an authority. In this exchange, the model missed implementation constraints and a risk to patient autonomy. Human review was needed to test those assumptions and judge whether the proposal could work responsibly.
  • The value is the critique, not the output. What protects the people on the receiving end isn't the model's answer; it's knowing when to agree with it, push back, and synthesize. That discernment is what separates AI that helps from AI that harms.
  • Ethics is operations. The strongest solutions aren't "more ethical" in the abstract; they're ethically sound and practically achievable, which is the only kind that actually reaches the people affected.
Home