§Why CasetextPricing

Pulse on AI + Healthcare: An Introduction to Healthcare AI Innovation in an Evolving Regulatory Landscape

By Benesch
Jul 12, 2024

The expansion of artificial intelligence (“AI”) applications in healthcare (“Healthcare AI”) has made recent headlines, from the transformation of clinical diagnostics with increased accuracy, to use of personalized medicine with the potential for life-changing preventative care, and hope to remedy, in some degree, the pervasive healthcare workforce shortage.[i] AI is certain to continue its healthcare delivery revolution; by 2028, the growth of the Healthcare AI market share is expected to reach approximately $102.7 billion USD.[ii] Despite the perception that AI is a futuristic concept, one of the earliest applications of AI was a 1950s checkers-playing computer program that was enabled for self-learning – a precursor to modern applications.[iii] Healthcare AI presents vast opportunity for innovation, but also carries considerable risk for providers, patients and other industry stakeholders if such risks are not mitigated prior to its use.

This article—part of our “Pulse on AI + Healthcare” series—seeks to provide a brief overview of Healthcare AI applications and legal and regulatory efforts to govern Healthcare AI, while emphasizing a “proceed with caution” approach prior to utilizing AI in the delivery of healthcare services.

Current Applications of Healthcare AI

Although there is no universally accepted definition for AI, the United States National Artificial Intelligence Initiative Act defines it as, “A machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations or decisions influencing real or virtual environments.”[iv]

In the healthcare industry, AI is incorporated into various functions for providers, payors and patients. The following examples illustrate how Healthcare AI augments aspects of clinical and non-clinical operations:

  1. Machine learningtechniques train AI algorithms for task performance[v] used in disease prediction.
  2. Natural language processing allows computer programs to understand, interpret and translate oral or written language in medical record transcribing or extraction of data patterns in patient files.
  3. Deep learning uses large data sets filtered by layers to create neural networks used in x-ray analysis.
  4. Generative AI emulates the structure and characteristics of input data to generate synthetic content[vi] and can automate routine administrative tasks such as scheduling appointments or processing claims.
  5. Large language models are trained on voluminous textual data to understand language and generate novel content, such as clinical treatment education communicated to patients with no staff intervention.
  6. Software as a medical device is used for one or more medical purposes without being part of a hardware medical device,[vii] such as cancer detection software.
  7. Clinical decision support software tools provide general knowledge and patient-specific information to enhance healthcare,[viii] and can create patient summaries and clinical guidelines for diagnoses.

The Legal and Regulatory Landscape of Healthcare AI
There is currently no uniform framework of laws or regulations governing Healthcare AI at the federal or state level. Aside from a prior Executive Order and guidance from the White House Office of Management and Budget (“OMB”) AI Memorandum, federal activity concerning AI stems primarily from specific agencies regulating target industries, with some notable overlap. U.S. states provide a patchwork of laws and regulations with trends starting to take shape around consumer protection, insurance and data privacy.

Federal Executive Directives.
In October 2022, the White House Office of Science and Technology Policy issued its “Blueprint for an AI Bill of Rights” intended as a model for both government and private sector entities.[ix] A year later, the federal Executive Order on the “Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence,” was issued (the “EO”). The EO directed federal agencies to assess potential risks and implement regulations and policies on advancing and using AI. The EO’s overarching message is to protect individual privacy and consumer rights, promote equity and innovation, and mitigate the risks of AI prior to its use.[x] Following the EO, the White House confirmed it received industry sector commitments from fifteen (15) leading AI companies and twenty-eight (28) payors and providers in AI initiatives under the FAVES principles – aiming for healthcare outcomes that are Fair, Appropriate, Valid, Effective, and Safe.[xi]

Federal Agency Authority.
Certain federal agencies have provided regulations and guidance impacting critical aspects of Healthcare AI, such as clinical decision-making, data privacy, potential bias and ethical use. These regulations are crucial in ensuring the safe and effective integration of AI applications in healthcare and will subsequently be covered in more detail by Benesch’s AI Commission in our Pulse on AI + Healthcare article series (in no particular order):

The Office of Civil Rights (“OCR”) – On May 6, 2024, the OCR issued a Final Rule strengthening protections against discrimination and bias in healthcare, specifically with regard to AI clinical decision-making tools. Specifically, the OCR’s responsibility in enforcing the Affordable Care Act § 1557 will extend to covered entities using AI, clinical algorithms, predictive analytics and other tools in health program activities. This Final Rule took effect on July 5, 2024.[xii]

The Federal Trade Commission (“FTC”) – Last year, the FTC released its “Privacy and Data Security Update” and in April 2024, with other federal agencies, a “Joint Statement on Enforcement Efforts Against Discrimination and Bias in Automated Systems.” These statements reflect the FTC’s focus on AI and automated systems, concerns for discrimination and violations of unfair business practices.[xiii]

The Food and Drug Administration (“FDA”) – In March 2024, the FDA released “Artificial Intelligence and Machine Learning in Software as a Medical Device” and previously, “Clinical Decision Support Software: Guidance for Industry and FDA Staff.[xiv] This guidance indicates FDA’s commitment to protect public health and promote ethical medical product innovation using AI.[xv] The FDA regulates AI software depending on the type or use and excludes some software functions under the 21stCentury Cures Act.[xvi]

The Office of the National Coordinator for Health Information Technology (“ONC”) – The ONC issued a Final Rule, effective March 11, 2024, that implements provisions of the 21st Century Cures Act and makes updates to the ONC Health IT Certification Program regarding AI algorithm transparency, a new standard for public health data interoperability, and enhances information blocking requirements.[xvii]

Centers for Medicare and Medicaid Services (“CMS”) – Earlier this year, CMS released FAQ guidance regarding its prior Final Rule on coverage criteria and utilization management requirements.[xviii] In sum, the Final Rule and guidance indicate that Medicare Advantage plans may not solely rely on AI algorithms or software to perform utilization review and make medical necessity determinations or terminate service.

State Oversight Efforts.
More than one-third of states have consumer protection, insurance or data privacy laws in place that intersect with targeted Healthcare AI uses of data. Among these states, some prohibit forms of consumer profiling or “algorithmic discrimination” for health insurance or other purposes. Utah’s Artificial Intelligence Policy Act went into effect May 1, 2024, and puts “generative AI” under its consumer protection laws. The Utah Act requires private companies to disclose use of generative AI in interactions with consumers, upon request, but licensed professionals have an affirmative obligation to disclose use of generative AI at the outset of services.[xix]Colorado’s Artificial Intelligence Act was recently signed into law on May 17, 2024[xx] and applies to “developers” and “deployers” who are responsible for such AI systems. The Colorado AI Act will be enforced by the Attorney General and takes effect Feb. 1, 2026.[xxi] Other AI bills relating to healthcare are pending in at least ten (10) states.[xxii]

Professional Industry Guidance.
The American Medical Association (“AMA”) is a leading healthcare industry member that highlights “Principles” that all stakeholders should consider when using AI tools. The AMA’s guidance emphasizes use of “augmented” rather than “artificial” intelligence, to underscore the importance of AI as a supporting, assisting role in healthcare, rather than as a replacement or substitute to providers.[xxiii] The AMA advocates for rigorous standards in AI implementation, a focus on patient safety and equity[xxiv], and maintaining integrity of the patient-provider relationship.[xxv] The AMA’s balanced advice invites assessment of some potential benefits and risks of Healthcare AI in the healthcare industry at large:[xxvi]

Potential Benefits

    • Increased efficiency in healthcare delivery
    • Reduced costs for providers and patients
    • Optimizing workforce resources
    • Increased accuracy in diagnosis and treatment
    • Automation of routine administrative tasks
    • Better patient outcomes

Potential Risks

    • Technical or operational errors
    • Ethical concerns (e.g. misuse or bias)
    • Data privacy and cybersecurity breaches
    • Liability regarding:
      • Practice of medicine scrutiny (e.g. scope of practice, licensure, corporate practice)
      • Fraud, waste and abuse (e.g. up-coding, no personally performed services, false claims)
      • Patient safety (e.g. consent, misdiagnosis)

In addition to the above examples, where not already required by law or contract, providers should consider the need and adequacy for transparency in AI-use disclosures to patients, payors and insurance carriers.[xxvii]

Five Key Takeaways on Healthcare AI

While there are many unknown consequences and variables of Healthcare AI, here are five fundamental considerations:

  1. AI applications offer diverse opportunities and benefits for the healthcare industry;
  2. The use of Healthcare AI requires careful thought and consideration due to legal risks and liability;
  3. AI governance is in flux and varies at the state level, causing compliance challenges for providers;
  4. Providers can expect healthcare transactions, insurance policies and contracts to involve AI terms;
  5. Compliance professionals and providers should be proactive in mitigating potential risks posed by AI.

[i] Akinrinmade AO, et al., Artificial Intelligence in Healthcare: Perception and Reality, Cureus, September 19, 2023;15(9):e45594, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10587915/; Medical Economics: How AI can fill gaps caused by the health care worker shortage, https://www.medicaleconomics.com/view/how-ai-can-fill-gaps-caused-by-the-health-care-worker-shortage.

[ii] Markets and Markets, Artificial Intelligence in Healthcare Market by Offering (Hardware, Software, Services), Technology (Machine Learning, NLP, Context–Aware Computing, Computer Vision), Application, End User and Region – Global Forecast to 2028,” January 2023,marketsandmarkets.com/Market-Reports/artificial-intelligence-healthcare-market-54679303.html.

[iii] Encyclopedia Britannica, Early milestones in AI: The first AI programs,

https://www.britannica.com/technology/artificial-intelligence/The-Turing-test (The program was initially written in 1951 and ran at the University of Manchester, England and subsequently rewritten by Arthur Samuel for the IBM 701 prototype computer in 1952; self-enabling features were added in 1955).

[iv] 15 USC § 9401(3); Pub. L. 116–283 Section 5002(3) (The definition further explains that “artificial intelligence systems use machine and human-based inputs to— (A) perceive real and virtual environments; (B) abstract such perceptions into models through analysis in an automated manner; and (C) use model inference to formulate options for information or action”).

[v] U.S. Federal Food and Drug Administration: Software as a Medical Device.

[vi] Executive Order on Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, October 30, 2023.

[vii] FDA: Software as a Medical Device (SaMD) (We note that the definition follows the International Medical Device Regulators Forum).

[viii] Health IT: The Office of the National Coordinator for Health Information Technology Clinical Decision Support; AI for Health Care Providers: Overview, Practical Law Practice Note Overview, Westlaw, Thomson Reuters, w-025-7755, 2024.

[ix] White House Office of Science and Technology Policy, Blueprint for An AI Bill of Rights, October 2022.

[x] Executive Order on Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, October 30, 2023; 88 Fed. Reg. 75191, November 1, 2023.

[xi] White House Briefing: Delivering on the Promise of AI to Improve Health Outcomes, December 14, 2023 (Payor and provider commitments include: Allina Health, Bassett Healthcare Network, Boston Children’s Hospital, Curai Health, CVS Health, Devoted Health, Duke Health, Emory Healthcare, Endeavor Health, Fairview Health Systems, Geisinger, Hackensack Meridian, HealthFirst (Florida), Houston Methodist, John Muir Health, Keck Medicine, Main Line Health, Mass General Brigham, Medical University of South Carolina Health, Oscar, OSF HealthCare, Premera Blue Cross, Rush University System for Health, Sanford Health, Tufts Medicine, UC San Diego Health, UC Davis Health, and WellSpan Health).

[xii] 89 Fed. Reg. 37522 (May 6, 2024), https://www.govinfo.gov/content/pkg/FR-2024-05-06/pdf/2024-08711.pdf.

[xiii] Federal Trade Commission, Department of Labor, Department of Justice, Consumer Financial Protection Bureau, Department of Education, Department of Homeland Security, Department of Health and Human Services, and Equal Employment Opportunity Commission, JOINT STATEMENT ON ENFORCEMENT OF CIVIL RIGHTS, FAIR COMPETITION, CONSUMER PROTECTION, AND EQUAL OPPORTUNITY LAWS IN AUTOMATED SYSTEMS, April 4, 2024; Federal Trade Commission, Privacy and Data Security Update, 2023.

[xiv] 21 U.S.C. § 360j(o)(1)(E).

[xv] U.S. Food and Drug Administration, Artificial Intelligence and Medical Products: How CBER, CDER, CDRH, and OCP are Working Together, March 15, 2024; Clinical Decision Support Software, September 28, 2022.

[xvi] U.S. Food and Drug Administration, Clinical Decision Support Software, September 28, 2022, pp. 4-6.

[xvii] Office of the National Coordinator for Health Information Technology, Health Data, Technology, and Interoperability: Certification Program Updates, Algorithm Transparency, and Information Sharing (HTI-1) Final Rule, 89 FR 1192, March 11, 2024.

[xviii] CMS Final Rule (CMS-4201-F), 88 Fed. Reg. 22120, April 12, 2023; Frequently Asked Questions related to Coverage Criteria and Utilization Management Requirements in CMS Final Rule (CMS-4201-F), February 6, 2024.

[xix]Utah S.B. 149 “Artificial Intelligence Amendments”; The Artificial Intelligence Policy Act defines generative AI as “an artificial system that: (i) is trained on data; (ii) interacts with a person using text, audio, or visual communication; and (iii) generates non-scripted outputs similar to outputs created by a human, with limited or no human oversight.”

[xx] Colorado Artificial Intelligence Act, S.B. 24-205; The Colorado Artificial Intelligence Act defines algorithmic discrimination as “any condition in which the use of an artificial intelligence system results in an unlawful differential treatment or impact that disfavors an individual or group of individuals on the basis of their actual or perceived age, color, disability, ethnicity, genetic information, limited proficiency in the English language, national origin, race, religion, reproductive health, sex, veteran status, or other classification protected under the laws of this state or federal law.”

[xxi] Id.

[xxii] Lexis Nexis, Artificial Intelligence Legislation Tracker, maintained, 2024; CA AB3030, CA AB2930, CA AB2058, CA SB1120, GA HB887, IL SB2795, IL HB5116, MA H1974,NY AB9149, NJ A3858, OK HB3835, OK SB1975, RI H7521, VA HB747, VT H710.

[xxiii] American Medical Association: Principles for Augmented Intelligence Development, Deployment, and Use Approved by AMA Board of Trustees, November 14, 2023; AMA Augmented Intelligence Research: Physician sentiments around the use of AI in health care: motivations, opportunities, risks, and use cases, November 2023.

[xxiv] American Medical Association, Advancing health care AI through ethics, evidence and equity, https://www.ama-assn.org/practice-management/digital/advancing-health-care-ai-through-ethics-evidence-and-equity.

[xxv]Principles for Augmented Intelligence Development, Deployment, and Use Approved by AMA Board of Trustees, November 14, 2023.

[xxvi] American Medical Association, Policies: Augmented Intelligence in Health Care,H-480.939; H-480.940.

[xxvii]Principles for Augmented Intelligence Development, Deployment, and Use Approved by AMA Board of Trustees, November 14, 2023.