Regulator‑Ready Language: How to Write Performance Claims for ML Models in SaMD and Enterprise AI

Step 1: Anchor the claim—Intended use, context, and dataset

Before you can state any performance numbers, you must anchor the claim in a clear description of what the model is meant to do and under what conditions. Regulators read performance claims through the lens of intended use. If this anchor is weak or ambiguous, the same numbers can be misinterpreted, overstated, or rejected. A regulator-ready anchor includes the population, the setting, the inputs and outputs, and the model’s functional role in decisions.

• Population and context: Specify who the model is for (e.g., adults in acute care, SMEs in an enterprise workflow), the clinical or enterprise scenario (e.g., triage, retrospective decision support, transaction monitoring), and any exclusions (e.g., pediatric cases, non-English records). Use language that makes it clear the claim does not extend beyond the defined context. If the model is intended to be used by trained professionals, state this. If it is not a standalone decision-maker, state that it is an adjunct and that final decisions rest with qualified personnel.
• Input and output: Define exactly what data the model consumes (e.g., ECG waveform, chest X-ray DICOM, tabular HR data, unstructured text) and what it produces (e.g., binary classification, probability score, risk tier). This is important because performance depends on data modalities and formats. If preprocessing or feature engineering is required, mention it so others can reproduce the conditions under which performance was measured.
• Decision role and operating context: Clarify how the output is intended to influence decisions (e.g., flag for review within 15 minutes, prioritize queue, suggest next step). If the model provides a score that must be thresholded to produce an actionable decision, state that an operating point is defined and will be reported. If the model supports risk communication, explain that outputs are calibrated probabilities and not deterministic diagnoses.
• Study design and dataset sources: Identify the study type (prospective, retrospective, randomized, enrichment strategies) and where data came from (number of sites, geographies, time windows). Regulators will ask if the dataset reflects the intended use environment. Describe inclusion and exclusion criteria and how prevalence compares to expected real-world prevalence. Prevalence shapes positive predictive value (PPV) and negative predictive value (NPV). Without it, a reader cannot interpret those measures.
• Data partitions and independence: State how data were split (development, validation, test) and how independence was maintained. Ensure that patient- or entity-level independence is guaranteed to avoid leakage. Clarify whether the claim is based on internal validation, external validation, or both, and whether the external validation includes sites not used in development. The strength of the claim grows when external validation matches the intended use setting.

The goal of this step is to make your performance interpretable and bounded. A precise intended use, clear inputs/outputs, explicit decision role, and transparent study design prevent the impression that the model works in all populations, settings, or data regimes. This is the foundation on which all performance metrics must rest.

Step 2: State core performance with statistics—metrics, operating point, and inferential evidence

Once the claim is anchored, present the core performance. Regulators expect you to align metrics with the task and to present them with uncertainty. Do not rely on a single headline number. Provide a comprehensive picture that covers discrimination, operating point performance, calibration, and stability.

• Discrimination (overall ranking ability): Report AUROC for binary classification or AUPRC when prevalence is low and false positives are costly. Include a 95% confidence interval (CI) for each. The AUROC shows how well the model distinguishes positives from negatives across thresholds, but it does not define actionability. It must be accompanied by operating point metrics.
• Operating point selection: Pre-specify the threshold used for decision-making. Document how it was chosen (e.g., based on Youden’s J on a development set, constrained by a clinical requirement for minimum sensitivity). Then report sensitivity, specificity, PPV, and NPV at that threshold. Each metric must include a 95% CI. Explain any ties between the operating point and risk control measures (e.g., if a higher sensitivity is chosen to reduce missed cases, acknowledge the expected trade-off in specificity and PPV).
• Hypothesis framing (if applicable): If your evaluation was hypothesis-driven, align your claim to the pre-specified endpoints. Use clear phrases like “superiority” or “non-inferiority” and provide p-values and CIs that match those endpoints. State how margins were set for non-inferiority and confirm that Type I error was controlled. Regulators will examine whether you deviated from the statistical analysis plan; your reported claim should mirror that plan.
• Calibration: Report how well predicted probabilities match observed outcomes. Provide calibration measures (e.g., calibration slope, intercept) and a narrative summary of calibration curves. A model with strong discrimination but poor calibration can mislead users about risk. If you recalibrated for a new setting, document the method (e.g., Platt scaling, isotonic regression), the dataset used, and the impact on performance.
• Drift monitoring and stability: State whether and how prediction quality changes over time. Provide a summary of drift detection methods (e.g., population stability index for features, KL divergence for score distributions, periodic re-estimation of calibration). If your claim depends on a fixed version, say so. If you intend post-market or post-deployment monitoring, identify the triggers for alerting, thresholds for revalidation, and procedures for updating the operating point.
• Censoring, missingness, and handling rules: Describe how missing data were handled (imputation rules, exclusion policies) and how censored outcomes were addressed, if relevant. These choices influence bias and should be transparently disclosed.

The purpose of this step is to present a transparent, statistically grounded performance profile. Report discrimination, thresholded performance, uncertainty, calibration quality, and stability. If claims are hypothesis-based, ensure they align with pre-specified endpoints and control for multiplicity. Use clear statistics, not marketing language.

Step 3: Address fairness and generalizability—subgroups, bias methods, and boundaries

Regulators expect you to evaluate how performance varies across meaningful subgroups and to disclose any material disparities. They also expect clarity about the limits of the data and conditions under which the model is or is not expected to generalize. Treat this as part of risk control, not an optional appendix.

• Subgroup definition and selection: Predefine subgroups based on clinical relevance and legal considerations (e.g., sex, age bands, race/ethnicity where permitted and appropriate, comorbidities, site, device manufacturer, language). Explain the rationale for each subgroup. Opportunistic post-hoc subgroup analyses risk bias and should be clearly labeled as exploratory.
• Fairness metrics and thresholds: Specify which fairness or subgroup performance metrics you used (e.g., difference in sensitivity across subgroups, equalized odds gap, PPV disparity). State thresholds for what you consider material disparity before analysis. Do not rely on vague assurances like “no bias detected.” Quantify differences, provide CIs, and indicate whether disparities are statistically and clinically meaningful.
• Methods for bias assessment: Describe the methodology (e.g., stratified CIs, bootstrap for subgroup estimates, hierarchical modeling to borrow strength across small subgroups). Address sample size limitations and how you mitigated them, such as aggregating related categories when justified. If protected characteristics are not directly available, explain proxy methods responsibly and their limitations.
• Mitigation status and residual risks: If you identified disparities, state what mitigations were applied (e.g., reweighting, threshold adjustment per subgroup where permitted, additional calibration, data augmentation, or targeted data collection). Be explicit about whether mitigations were applied in the evaluated version and whether they will be part of the deployed configuration. Acknowledge residual risks and how users should interpret outputs in affected subgroups.
• Generalizability boundaries: Define the domains where evidence supports performance: data sources (vendors, scanners, sensors), geography, language, prevalence ranges, workflow integration, and latency constraints. Identify settings not evaluated. If the model is not validated for pediatric populations or for images from a certain manufacturer, say so. Provide guardrails such as input QC checks, out-of-distribution detection, or minimum data completeness thresholds. If the model is brittle to distribution shifts, disclose this and the monitoring plan.
• Versioning and operating conditions: Link claims to a specific version of the model and software environment. If quantization, compression, or deployment hardware differs from the evaluation environment, explain potential impacts. Define acceptable tolerance ranges (e.g., inference latency limits, minimum resolution) and describe automated safeguards when inputs fall outside those ranges.

This step ensures readers understand not only the average performance but also for whom, where, and under what constraints that performance holds. It converts fairness into measurable assurances and sets explicit boundaries for safe generalization.

Step 4: Write regulator-ready sentences—concise, reproducible claims with do/don’t patterns and a self-audit checklist

After you gather the evidence, translate it into labeling language that is concise, reproducible, and free from marketing embellishments. Your goal is to make the claim verifiable by an independent reviewer using the described data and methods.

• Phrasing patterns for intended use:

  • “This software is intended for [population] in [setting] to [action] using [inputs] to produce [outputs] that support [decision role]. It is not intended as a standalone diagnostic.”
  • “Performance characteristics were established using a [study design] on data from [sites/timeframe] with a prevalence of [value], reflecting the intended use setting.”

• Phrasing patterns for core performance:

  • “On the prespecified test set, AUROC was [value] (95% CI [lower, upper]). At the prespecified operating point (threshold = [value]), sensitivity was [value] (95% CI [lower, upper]), specificity was [value] (95% CI [lower, upper]), PPV was [value] (95% CI [lower, upper]), and NPV was [value] (95% CI [lower, upper]).”
  • “The primary endpoint demonstrated [superiority/non-inferiority] with a margin of [value]; difference = [value] (95% CI [lower, upper]), p = [value], per the statistical analysis plan.”
  • “Predicted probabilities were well-calibrated with calibration slope [value] and intercept [value]. No recalibration was performed post hoc / Recalibration via [method] was applied prior to evaluation.”

• Phrasing patterns for drift and monitoring:

  • “Performance stability was assessed across [time/window]. No clinically meaningful drift was observed. The system monitors [metrics] and triggers revalidation if [thresholds] are exceeded.”
  • “This claim applies to model version [ID]. Any model update will undergo revalidation prior to use.”

• Phrasing patterns for fairness and generalizability:

  • “Subgroup analyses were prespecified for [list]. The maximum observed disparity in sensitivity was [value] (95% CI [lower, upper]). [Mitigation] was/was not applied; residual risks are described in the labeling.”
  • “This claim is supported in [domains: sites, devices, languages] and is not established for [non-evaluated domains]. Use outside these domains is not recommended without additional validation.”

• Phrasing patterns for guardrails and use conditions:

  • “The system rejects inputs that fail [QC checks] and will not return a score under these conditions. Users should not override this behavior.”
  • “Minimum data completeness is [criteria]. If unmet, no decision is produced.”

• Do’s:

  • Do link all performance claims to a specific study design, dataset, and version.
  • Do provide 95% CIs alongside all reported metrics and align p-values to prespecified endpoints.
  • Do disclose calibration quality, drift monitoring, and triggers for revalidation.
  • Do quantify subgroup disparities and explain mitigation status and residual risk.
  • Do define operating points, guardrails, and input quality checks.

• Don’ts:

  • Don’t use unqualified superlatives (“state-of-the-art,” “best-in-class”) or vague assurances (“no bias detected”).
  • Don’t report unweighted or pooled averages that hide material subgroup differences.
  • Don’t mix development and test data or change endpoints post hoc without labeling them as exploratory and providing rationale.
  • Don’t imply generalizability to settings, devices, or populations not studied.
  • Don’t omit prevalence, as it is necessary for interpreting PPV and NPV.

• Self-audit checklist before submission:

  • Intended use clearly states population, setting, inputs, outputs, and decision role.
  • Study design and dataset sources are transparent; prevalence is reported and justified.
  • Discrimination metrics (AUROC/AUPRC) and operating point metrics (sensitivity, specificity, PPV, NPV) include 95% CIs.
  • Hypothesis testing aligns with prespecified endpoints; non-inferiority margins and Type I error control are documented.
  • Calibration quality is reported; any recalibration is justified and documented.
  • Drift monitoring plan, metrics, thresholds, and versioning are specified.
  • Subgroup analyses are prespecified where possible; disparities are quantified with CIs.
  • Fairness mitigation status and residual risks are clearly stated.
  • Generalizability limits are defined across domains (sites, devices, languages, time).
  • Operating points, QC checks, and guardrails are defined and match the evaluated configuration.
  • All claims are reproducible using the described methods and datasets; no marketing adjectives are used.

By following these steps, you convert raw experimental results into regulator-ready language. Anchoring the claim ensures relevance and scope. Presenting complete statistics with uncertainty and calibration provides a reliable picture of performance. Addressing fairness and generalizability manages risk and avoids misleading interpretations. Finally, using precise phrasing patterns, do/don’t guidance, and a self-audit checklist helps you deliver claims that are reproducible, reviewable, and aligned with regulatory expectations for Software as a Medical Device and enterprise AI systems. This approach does not inflate results; it communicates them responsibly, with the context required for safe, compliant use.