Prescription for Innovation: The Path to Regulating Generative AI in Medical Devices
Healthcare as the Next Frontier for AI: AI-Enabled Devices
Social determinants of health, including access to quality healthcare, account for up to 60% of premature deaths globally. In privatized economies like the United States where decreased access to preventative healthcare worsens health disparities, the advent of publicly accessible artificial intelligence tools creates an exciting bridge between the demand for medical expertise and a growing supply of health AI solutions.
“AI-enabled devices” are categorized as combination products that are defined as “therapeutic and diagnostic products that combine drugs, devices, and/or biological products.” However, due to regulatory, policy, and review management challenges, government agencies like the FDA have struggled to both meet the pace of technological development surrounding AI-enabled devices and codify a successful regulatory framework for the expedited commercialization of these innovations.
As of September 2026, the FDA has authorized over 1,600 AI-enabled devices for marketing. The FDA recognizes the potential of AI enabled medical devices to rapidly improve performance through iterative adjustments, and has published multiple papers and gathered public comments in order to synthesize an approach to safely expediting these innovations.
In addition to increasing access to diagnostic and supportive medical care, these AI-enabled/ GenAI-enabled medical devices are also finding their way into the practice of medicine itself. Physicians are increasingly utilizing AI for clinical care to bridge gaps in expertise and training, with a Harvard study showing that physicians, especially those in early in their career, frequently utilize AI in making treatment decisions outside of their immediate specialties. With the reliance on these technologies developing faster than the laws governing them, these studies emphasize the importance of studying the pioneering AI-enabled medical devices pushing for commercialization right now.
What Bodies of Law Govern AI Enabled Devices and What Are the Commercialization Implications?
The advent of healthcare AI, especially generative AI, has elicited many new concerns about legal compliance. From privacy to patents to patient safety, companies pioneering AI medical technology have had to navigate a legal maze before pushing their products to the market. Many existing federal and state laws constrain the proliferation of AI-enabled medical devices. Here we focus on a select few bodies of law: HIPAA, patent law, and FDA regulations.
HIPAA
The Health Insurance Portability and Accountability Act, more commonly known as HIPAA, is the nation’s main federal law on patient privacy. It mandates the privacy and security of individually identifiable health information. The requirements of HIPAA directly conflict with AI development. While AI models work more accurately if they are trained on more detailed data, HIPAA requires that healthcare providers only release data that is anonymized and nonidentifiable.
Not all parties that collect health information are subject to HIPAA. HIPAA applies to healthcare clearinghouses, health plans, healthcare providers, and business associates. Business associates are parties that perform functions on behalf of a party from one of the other categories. Healthcare providers can share protected health information with business associates, but only for the purpose of executing the provider’s healthcare functions. The business associate cannot legally use the protected information for its own purposes. Therefore, companies creating AI-enabled medical technology typically must work around HIPAA. These companies are third-parties using medical data for their own purposes of training AI models that will be marketed across the healthcare market. Therefore, they cannot train the models off of protected health information.
Healthcare providers can share health information with third-parties if the information has been properly de-identified. De-identified information is defined under statute as “[h]ealth information that does not identify an individual and with respect to which there is no reasonable basis to believe that the information can be used to identify an individual.” Companies training AI-enabled medical devices typically rely on de-identified data made available to them by healthcare providers.
Critics have pointed out that there are many gaps in the current de-identification process that may allow identifiable patient information to leak through. De-identification requires either expert approval or the removal of eighteen specific identifiers (including items such as date of birth and zip code). Since HIPAA’s enactment in 1996, many advancements have been made in data analytics and machine learning. These advancements now make it possible for third-parties to make legally de-identified data identifiable again by combining it with other datasets. For example, one 2020 study found that a machine learning model could identify 99.98% of individuals in an anonymized dataset using fifteen demographic attributes. As data analytics continues to advance, the risk for data misuse and privacy violations continues to grow. Data breaches, which constitute the basis for the majority of HIPAA enforcement claims, also now carry with them increased consequences. A breach of de-identified data could turn out to now carry the same consequences as a breach of protected health information. This issue is further compounded by the cybersecurity concerns around AI. State and federal legislators should consider how the law can keep up with technological advances to avoid rendering HIPAA protections meaningless. More requirements should be added to the de-identification process or restrictions should be imposed on the sharing of information with third-parties.
Patent Law
AI-enabled medical technology also faces unique barriers to patenting. Patents are generally available for “any new and useful process, machine, manufacture, or composition of matter.” However, some judicially-created exceptions apply. Patents may not be granted for “laws of nature, natural phenomena, and abstract ideas.” When determining if a patent is not available under these grounds, courts engage in a two-part test. First, the court decides if the patent claim is “directed to” one of the ineligible categories, so that that category is the focus of the claim. Second, the court decides if “the elements of each patent claim contain additional aspects that ‘transform’ it into a patent-eligible application of an ineligible concept.” Simply applying AI to a different problem within the excluded categories is not enough; the company must demonstrate that it has added additional technical aspects that make its product unique.
Many of the currently available AI-enabled medical devices are used for diagnostics. Diagnostic methods are more likely to be unpatentable because they only reflect “abstract ideas or laws of nature.” Logically, the ability to recognize a disease is not something to be owned by one company. Companies seeking to patent AI-enabled diagnostics would need to demonstrate that their devices contain some other unique aspect in its technology or process that is patentable. AI diagnostic tools account for only 5% of patents for medical machine learning devices. AI devices focusing on lower-end tasks, such as measurement, task analysis, detection, and classification, comprise over half of the granted patents.
AI-generated technologies also risk being unpatentable since the U.S. Court of Appeals for the Federal Circuit has stated that AI cannot be listed as an inventor for the purposes of patenting. Under the USPTO’s released guidelines, AI can be used as a tool for invention but the human inventors must independently fulfill inventor requirements. A human inventor must conceive of the invention, “meaning to possess in their mind a definite and permanent idea of the complete invention, such that only ordinary skill is required to reduce the invention to practice.”
FDA Approval
Finally, if a company ever wishes to see their AI-enabled medical device go to market, they have to seek out FDA approval. Most AI-enabled devices fall under the Software as a Medical Device (SaMD) category for FDA regulation. SaMD are “software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device.” On the other hand, some software does not qualify as a “device” and therefore is not subject to FDA regulations. Under the 21st Century Cures Act, software intended for administrative support, for encouraging a healthy lifestyle unrelated to any specific disease, for electronic patient records, or for transferring, storing, converting formats, or displaying data and results does not qualify as an FDA-regulated “device.” Certain “clinical decision support” software is also excluded if it meets specific criteria, such as requiring the independent review of a physician.
The FDA sorts SaMD into three different classes according to risk level. All classes are subject to similar reporting requirements, but have different premarket review requirements. Most AI-enabled devices have fallen under Class II (moderate risk). Therefore, they are subject to 510(k) premarket notification. 510(k) requires a company to demonstrate that their device is “substantially equivalent” to an existing “predicate” device on the market. This process is meant to ensure that the new device is equally safe and effective. Class II devices may also come to market through a de novo classification request. The de novo pathway exists for devices that are so novel that they would have no similar “predicate” device to compare to. The FDA must determine that its own regulations are enough to control the risk of allowing the new device to go to market. Rarely, AI-enabled SaMD may fall under Class III (high risk). Class III devices must pass premarket approval, which requires clinical evidence of safety and efficacy.
The FDA has approved over 1,600 AI-enabled medical devices. Notably, the FDA has not yet authorized a medical device that uses generative AI for commercialization.
RecovryAI and the “Breakthrough Device” Designation Path to Commercialization
The company RecovryAI, which markets itself as “physician-prescribed AI for post-operative recovery,” has created virtual care assistants (“VCAs”) for patients recovering from surgery. VCAs are a unique kind of provider- and patient-facing AI that independently provides recovery guidance to patients while keeping liability and responsibility with the clinician. The models were trained by measuring concordance between responses by VCAs and physician judgment across more than 43,000 real-life patient conversations, and they alert clinicians when a patient’s recovery deviates from the pattern expected.
On March 3, 2026, RecovryAI’s total joint arthroplasty recovery VCA was granted the FDA “breakthrough device” designation, which is reserved for medical devices that address serious medical conditions and demonstrate a path to improved standards of care. RecovryAI is a particularly fascinating case study because it stands as the first patient-facing, autonomous generative AI chatbot to receive the breakthrough device designation. If it succeeds in being granted FDA authorization, it will become the first of its kind in a De Novo Class II software as a medical device (SaMD) category.
Breakthrough devices are required to meet one of the following statutory criteria under the FDA in addition to the first criterion of providing for “more effective treatment or diagnosis of life-threatening or irreversibly debilitating human disease or conditions”:
Represents breakthrough technology
No approved or cleared alternatives exist
Offers significant advantages over existing approved or cleared alternatives
Device availability is in the best interest of patients
Inherently, the breakthrough device designation enables more intensive review for technological solutions that show the potential to improve the lives of patients with debilitating conditions. Surprisingly, it is the first criterion that is often difficult to meet: the requirement that the solution addresses must be severe and specific enough to warrant the more efficient review process.
However, for those mission-driven companies addressing highly urgent and specific medical conditions, the breakthrough device program holds value because it “enhances transparency, interaction, and flexibility throughout development.” It is not without its challenges; developers are often expected to present stronger clinical data, narrow the scope of their product (thereby limiting their customer pool and business) and maintain equally, or more, rigorous standards for safety and effectiveness. For AI-enabled medical device developers driven to get their products in the hands of those with rare or debilitating illnesses, however, this pathway is nonetheless an exciting opportunity to help those struggling with access to healthcare or the daily burdens of managing their conditions.
Will RecovryAI Establish a Pathway For This Market?
RecovryAI is the first of its kind to go through the regulatory path it is on: it is a procedure-specific, patient-facing, autonomous generative AI that does not require a licensed provider to review every interaction. For this reason, the company is pursuing a De Novo classification and is currently in the last steps prior to their final De Novo submission. The company just concluded a large non-significant risk trial (a trial that does not pose significant risks to its subjects) and is in the process of physician review and data analysis. If the data is favorable and can help answer questions posed by the FDA in previous rounds of review, RecovryAI will likely have a substantial chance at becoming the first generative-AI-enabled SaMD authorized by the FDA.
The prospect of this authorization could set precedent for the growing medtech AI market that is expected to be valued at $615 billion by 2034. The FDA additionally recently closed the public comments for their discussion paper on generative AI-enabled medical devices on October 19, 2026. This feedback will likely, in live time, be considered in the case-by-case decisions to create new regulatory pathways for companies such as RecovryAI. In addition to the FDA authorization process, RecovryAI will also face HIPAA and patent law constraints in their path to commercializing their product for the public. While the company is aiming to receive FDA authorization by Q2 of 2027 and is positioned for a more intentional timeline due to their breakthrough device designation, delays are still highly likely due to the complexity of the device and the lack of an established framework for evaluation.
Public Concerns and Suggestions About GenAI and FDA Regulation
In analyzing the responses submitted to the FDA regarding regulation of generative AI in medical devices, three main questions emerged:
How should a device be monitored after the launch, and can another AI be responsible for the monitoring?
Because AI-enabled devices, especially autonomous ones like RecovryAI, are unique in their continually changing nature, companies looking to commercialize their AI product must also consider how the product will be regulated as it evolves or degrades. Here, industry experts suggested that postmarket monitoring should not be left to another AI, but rather should combine periodic third party review with event-triggered regulation.
How many dimensions does it take to evaluate the risk of an AI?
The FDA paper considers evaluating two dimensions in risk assessment: the consequences from erroneous output and what the AI actually does. Clinicians generally supported this framework, but would not adopt it without accounting for additional dimensions such as the traceability and detectability of the AI-enabled device’s errors.
Is a benchmark score sufficient to predict how a device behaves with real patients?
The FDA proposes “benchmarking,” or using published and publicly available datasets as a reference point for the device’s clinical performance, as its main way of evaluating an AI-enabled device’s competency. The majority of experts and clinicians contest this method as counterintuitive: while publicly available benchmarking is valuable, providing it to a learning model can contaminate accurate evaluation results and therefore benchmarking should not be treated as evidence of competency on its own.
Overall, experts and clinicians are alert and pushing for more stringent evidentiary standards to ensure that AI-enabled devices enter the market in a way that benefits, rather than further harms, their patients. With the recent closing of public commentary, exciting developments in the law through the integration of this feedback are likely to follow in the coming months.
History in the Making: RecovryAI and Looking to the Future
Advancements in AI and machine learning technologies have revolutionized the current medical technology industry. Laws written before the generative AI boom are now stretching to accommodate this new technology, with experts fighting for more stringent evidentiary guidelines to protect patients while concurrently supporting innovation. With the emergence of devices like RecovryAI, it is clear that the industry is at a turning point. FDA approval of one generative AI device will pave the way for future uses of generative AI in medical technology, since companies could refer to RecovryAI as a predicate in the De Novo classification process. In the coming years, the practice of healthcare and the patient experience may be widely transformed by the introduction of generative AI. As this shift happens, lawmakers, clinicians, and businesses will need to continue to adjust to the dynamic nature of wielding generative AI in high-stakes environments.
*The views expressed in this article do not represent the views of Santa Clara University.
Comments