Changing the Method, Defending the Rationale: What to Say When Changing Disproportionality Method in Regulatory English

Step 1 — Grounding: metrics and concepts regulators need to understand

When inspectors read a regulatory submission that changes a disproportionality method, they need quick, plain-English clarity about what each metric is, what it measures, and what it implies for public health decision-making. Keep the definitions short, non-technical, and directly tied to regulator concerns: interpretability (can a reviewer understand the signal), reproducibility (can the analysis be rerun), and public-health protection (does the change better identify true signals without swamping reviewers with false positives).

• EBGM (Empirical Bayes Geometric Mean): A shrinkage estimator that compares the observed count of a specific drug–event pair to the expected count, adjusted by an empirical Bayes prior. Plain English: it gives a stabilized ratio so rare events do not look deceptively large. Regulatory implication: EBGM helps reduce false positives from low-count cells while preserving larger signals; inspectors will want to know the prior and the degree of shrinkage used.

• PRR (Proportional Reporting Ratio): The ratio of the proportion of reports of a specific event for a product to the proportion of that event for all other products. Plain English: PRR is a simple, easy-to-calculate measure of disproportionality that is intuitive but sensitive to reporting biases. Regulatory implication: PRR is transparent and easy to reproduce but can over-call signals when reporting patterns change.

• ROR (Reporting Odds Ratio): The odds of reporting a specific event with the product versus all other products. Plain English: ROR is mathematically similar to PRR but framed as an odds ratio, which makes it comparable to logistic regression outputs. Regulatory implication: ROR is interpretable and commonly used in signal detection, but like PRR it is more sensitive to small counts and reporting biases.

• IC025 (Information Component lower bound): The lower 95% credibility bound of the Information Component, a Bayesian measure used to quantify disproportional reporting (commonly associated with UMC/VigiBase). Plain English: IC025 tells you where the lower bound of the Bayesian signal measure lies; if it is above zero, the association is unlikely due to chance. Regulatory implication: IC025 explicitly reports statistical certainty and is conservative—inspectors will expect an explanation of the prior and the credibility interval method.

• Bayesian shrinkage: A technique that pulls extreme estimates toward a central value based on the strength of prior information and observed data. Plain English: shrinkage tempers apparent strong signals that are based on very few reports, reducing false positives. Regulatory implication: Shrinkage improves calibration across low- and high-count cells; inspectors will ask how the prior was specified and how sensitive results are to the prior choice.

• Sensitivity analyses: Systematic checks that test how results change under reasonable alternative assumptions (e.g., different priors, thresholds, exclusion of noisy data, stratification by time). Plain English: sensitivity analyses show whether your conclusions are robust or fragile. Regulatory implication: Sensitivity analyses are essential for reproducibility and for demonstrating that a method change does not hide clinically important signals.

• Governance rationale: The documented, auditable reasoning and approval process that justifies analytic choices (minutes, version control, impact assessments). Plain English: governance shows that the change was considered, reviewed, and accepted by appropriate stakeholders. Regulatory implication: Governance records let inspectors verify that the method change was institutionalized with appropriate oversight and risk management.

Step 2 — When and why you may validly change a disproportionality method

Changing a disproportionality method is a substantive decision that must be defensible across statistical, data-quality, operational, and governance dimensions. Each driver should be framed as addressing a specific regulatory concern, and for each driver you should describe the expected inspector question and how to preempt it with evidence.

• Statistical drivers (bias control and calibration):

  • Rationale: A method change may be needed to reduce known biases (e.g., small-cell instability, disproportional influence of high-volume reporters) or to improve calibration (alignment between expected false positive rate and observed). For instance, replacing a raw PRR with EBGM or IC025 reduces overcalling from low counts by applying principled shrinkage.
  • Inspector concern: Does the new method alter signal detection sensitivity for clinically important associations? Provide statistical metrics (e.g., true/false positive rates on labeled test sets, calibration plots) and show head-to-head comparisons.

• Data-driven drivers (changes in reporting patterns or new data sources):

  • Rationale: If the reporting ecosystem changes (e.g., introduction of patient reporting, new spontaneous-reporting systems, linkage to electronic health records), the assumptions underlying older methods may no longer hold. A method better suited to heterogeneous reporting intensity or to integrating multiple data sources can be justified.
  • Inspector concern: Does the new method handle the new data characteristics without introducing artifacts? Present stratified analyses and sensitivity checks that isolate the effect of the data change.

• Operational drivers (scalability, harmonization, computational constraints):

  • Rationale: Large-scale surveillance may require methods that scale or that are harmonized with partner agencies. For example, if the previous method is computationally infeasible at larger database sizes or prevents timely re-runs, selecting an alternative that preserves detection properties but improves runtime is defensible.
  • Inspector concern: Is the change driven solely by convenience, and does it compromise safety? Provide benchmarks, runtimes, and demonstrate equivalent or better signal capture on archived datasets.

• Regulatory/governance drivers (alignment with guidance, audit findings):

  • Rationale: New guidance (EMA, MHRA) or audit recommendations may require method alignment. Governance imperatives like auditability, version control, and transparency may make a formally documented change necessary.
  • Inspector concern: Was the change driven by regulatory compliance and appropriately accredited? Provide governance documentation: change requests, impact assessments, approval minutes, SOP updates.

In each category, present a short, factual description of the observed problem, the analytic limitations of the previous method, and why the alternative is expected to mitigate the issue. Emphasize that method choice should be the least-change solution that adequately addresses the problem while preserving historical interpretability.

Step 3 — Inspector-ready phrasing: what to say and how to structure the justification

Inspection teams expect clear, consistent language that links identified problems to decisions and evidence. Use a reusable sentence structure that flows from problem to solution to evidence:

• Reusable structure (one-sentence template): “Because we observed [concise problem statement], we changed from [old method] to [new method]; we evaluated the impact by [head-to-head comparison, sensitivity analyses], found [quantitative summary of impact], and documented approval in [governance record].”

• Expandable paragraph template for reports:

  • Start: Describe the driver succinctly: “During routine monitoring we identified [e.g., increasing proportion of single-report cases and increased volatility in PRR for low-count pairs], which creates an elevated false-positive rate.”
  • Middle: Describe the methodological change: “To address this, we replaced PRR with EBGM (empirical Bayes geometric mean), applying prior parameters X that produce moderate shrinkage for counts <N.”
  • Evidence: Summarize comparative analysis: “A head-to-head analysis on the historical dataset (2015–2024) comparing PRR and EBGM showed that EBGM reduced low-count alerts by Y% while preserving X/ Y known positive signals; calibration plots and ROC-like summaries are provided in Appendix A.”
  • Sensitivity checks: “We performed sensitivity analyses varying the prior by ±Z, varying thresholds (EBGM>=1.5, >=2.0; IC025>0), and re-running historical alerts; results were robust across plausible settings and flagged no clinically relevant signals missed under the new method.”
  • Governance: “The change was submitted as Change Request CR-2024-XX, reviewed by the Safety Analytics Board on DATE, and approved with action items to re-run historical alerts and to implement a 6-month post-change monitoring plan (minutes attached).”

• Phrases to state thresholds and interpretable meaning:

  • “We selected EBGM ≥ 2.0 to balance sensitivity and specificity based on calibration against labeled known positive associations; this threshold corresponds to an approximate X% estimated positive predictive value in our validation set.”
  • “We interpret IC025 > 0 as evidence that the lower bound of the 95% credibility interval excludes chance; this conservative choice minimizes false positives and aligns with VigiBase practice.”
  • For Bayesian shrinkage: “We applied empirical Bayes shrinkage to stabilize estimates in low-count drug–event pairs; shrinkage parameters were estimated from the data using method M and documented in Appendix B.”

Always link threshold choice to concrete validation results or to harmonized practice (e.g., EMA/MHRA precedent), rather than asserting an unsupported rule. Where thresholds are pragmatic, say so and explain the trade-offs.

Step 4 — Defending choices with evidence and transparency: what to present to inspectors

Inspectors will expect a structured, auditable package that demonstrates the change was necessary, evaluated, and controlled. Present evidence in both tabular and visual forms and include governance artifacts. The packages should enable rapid verification and deeper re-analysis.

• Minimum evidence bundle to include:

  • Comparative tables: side-by-side counts of alerts under old and new methods, metrics of overlap (e.g., proportion of historical alerts retained, lost, or newly created), and sensitivity-specific tables showing threshold effects.
  • Plots: calibration plots (observed vs. expected false positive rate), rank-change plots (showing how signal ranks move between methods), and forest/heatmap style visualizations of key drug–event pairs across scenarios.
  • Sensitivity scenarios: clearly defined scenarios (different priors, thresholds, exclusions) with short summaries of impact on key signals and on overall alert volume.
  • Re-analyses: explicit re-run results for historical signals of regulatory interest (e.g., those leading to label changes or regulatory actions). For each such signal give the old result, the new result, and an interpretation of whether regulatory conclusions change.
  • Governance artifacts: change request, impact assessment, minutes of decision-making committee, version-controlled code and documentation, and a rollback plan.

• Recommended visual and textual checklist for EMA/MHRA expectations:

  • Reproducible code and runtime environment specified (packages, versions, seed values).
  • Data provenance: exact database snapshots, extraction queries, and any de-duplication or mapping logic documented.
  • Threshold rationale: linkage between chosen cutoffs and validation results or harmonized guidance.
  • Impact assessment: quantitative summary of changes to alert volume, false-positive rate proxies, and effect on known positives.
  • Post-change monitoring plan: time-limited commitments to track missed signals, to re-run archival analyses if needed, and to report back any emergent issues.

Conclude your submission narrative by reiterating transparency and continuity: state how you preserved backwards comparability (e.g., by re-running historical periods with the new method and producing crosswalk tables), how stakeholders were engaged, and what monitoring will ensure no degradation in public-health protection. Use clear signposting language so inspectors can quickly find the evidence behind each claim.

Final note on tone and style: prioritize concise, factual sentences that make causal links explicit (problem → method → evidence → governance). Avoid defensive language; present the change as an evidence-led improvement with clear, auditable trails. This is what to say when changing disproportionality method: frame the need, state the solution, show the comparative evidence, and document the governance — then make it easy for inspectors to verify each step.