California Limits AI Use by Health Plans

In the fall of 2024, California Governor Newsom signed into law Senate Bill 1120. With that action, California became one of the first states to enact a law that regulates the use of artificial intelligence by health and disability insurance plans. This bill is also known as the Physicians Make Decisions Act.

The new law regulates California health and disability insurers who use artificial intelligence, algorithms, or other similar software (AI tools) for analyzing medical necessity as part of utilization reviews or other utilization management functions. The law does not prohibit the use of AI by health and disability insurers; rather, it regulates it when used.

What Does the Law Say?

The law states that AI tools may not supplant a human healthcare provider in making decisions for analyzing medical necessity. Specifically, insurance carriers may not deny, delay, or modify health care services based on a medical necessity determination made by an AI-based decision-making tool.

The plain purpose of the law is to safeguard patient access to treatments by mandating a certain level of health care provider oversight when payors use AI tools to assess the medical necessity of a requested service.

This law is directed toward health and disability insurers. However, employers should be aware of the impact on the claims administration practices of their plans.

Use of AI in Utilization Management

Health plans are increasingly relying on AI tools to streamline internal operations, including automating prior authorizations and utilization management processes. These AI tools have been marketed with the promise of both reducing costs and increasing efficiency. However, since their implementation, concerns have been raised about the stringent application of algorithms resulting in delays or incorrect denials of claims. Additionally, there are concerns about the lack of oversight by healthcare professionals whose experience and training would recognize nuances and complexities that AI tools might miss.

SB 1120 does not prohibit insurers from using AI tools. It prohibits sole dependence on it for medical decisions and aims to correct the lack of oversight by healthcare professionals. The law mandates that determinations of medical necessity under health plans be made by a licensed healthcare professional. A licensed healthcare professional is defined as a physician or other licensed healthcare professional who is competent to evaluate the specific clinical issues involved in the healthcare services requested by the provider. In short, AI cannot supplant the role of a healthcare provider in decision-making.

The Details

In addition to prohibiting AI tools from replacing physicians in the decision-making process, the new law incorporates specific requirements to regulate the use of AI tools that are employed to analyze medical necessity. These include: 

• Medical Necessity: AI tools must base decisions of medical necessity on:
  • Individual medical history and other clinical history,
  • Individual clinical circumstances as presented by the requesting provider, and
  • Other relevant clinical information as contained in the medical or other clinical records. 
    
    
• No Group Dataset: AI tools cannot make determinations based solely on a group dataset.

• Cannot Supplant Professionals: AI tools cannot supplant health care provider decision-making. A determination of medical necessity may only be made by a licensed healthcare professional.

• No Discrimination: AI tools may not discriminate, directly or indirectly, against individuals in a manner that violates federal or state law.
  
  
• Fair and Equitable: AI tools must be applied fairly and equitably.
  
  
• Open to Inspection: AI tools, including specifically the underlying algorithm, must be open to inspection for audit or compliance reviews.

• Disclosure Required: Disclosures pertaining to the use and oversight of AI tools must be provided in written policies and procedures that are disclosed to providers, enrollees, and the public, upon request. They must also be filed with the regulator for review and approval.

• Review and Revise: AI tool outcomes must be periodically reviewed and revised to ensure “maximum accuracy and reliability.”
  
  
• Limits on Patient Data Use: Patient data is prohibited from being used beyond its intended and stated purpose. Specifically, AI tools must limit the use of patient data to be consistent with California’s Confidentiality of Medical Information Act as well as HIPAA.
  
  
• No Harm: AI tools cannot directly or indirectly cause harm to enrollees.

Requirements for Insurance Carriers

Insurance carriers are mandated by the law to disclose the methodologies used by utilization review or utilization management contractors. This information must be provided to the insurance commissioner and made available to policyholders upon request.

The criteria employed by an insurer or contractor for utilization review or utilization management purposes, to decide whether to authorize, modify, delay, or deny healthcare services, must adhere to all of the following requirements:

• Be developed with involvement from actively practicing health care providers.
• Be consistent with sound clinical principles and processes.
• Be evaluated, and updated if necessary, at least annually.
• If used as the basis of a decision to modify, delay, or deny services in a specified case under review, be disclosed to the provider and the policyholder in that specified case.
• Be available to the public upon request.

The required upon request must be accompanied by the following notice: “The materials provided to you are guidelines used by this insurer to authorize, modify, or deny health care benefits for persons with similar illnesses or conditions. Specific care and treatment may vary depending on individual need and the benefits covered under your insurance contract.”

Applies To . . .

The law applies to health plans and disability insurers, both in-state and out-of-state, that use AI tools to analyze medical necessity affecting California enrollees. Applicability is not conditioned on whether the contract is written or filed in California. It applies to California enrollees in plans written in any jurisdiction.

Commentary

The swift adoption of artificial intelligence tools is being met with both enthusiasm and concern across various industries. Insurance carriers have been among the early adopters of this technology. This bill reflects California’s initiative to safeguard patients amidst the rapid adoption of this evolving technology.

References

CA SB 1130