Pre-Trained, Not Pre-Qualified: The QA Risk Behind Third-Party AI Models
Key Takeaways
- A pre-trained AI model should be treated as a governed AI component, not an invisible technical feature.
- Pre-trained models should enter through the company’s AI Governance Policy: inventory, risk classification, supplier qualification, integration controls, monitoring, change control, and retirement.
- QA does not need to repeat every vendor test, but it must confirm intended-use fit, integration behavior, workflow impact, and local performance expectations.
- Supplier qualification must address model identity, version, provenance, documentation, limitations, update practices, and lifecycle controls.
- Human-in-the-loop is only meaningful when the human role is trained, documented, monitored, and empowered to challenge the model output.
- A supplier release note is not a change-control record. Vendor updates still require impact assessment against the implemented intended use.
What a Pre-Trained Model Is — and Why It Accelerates Innovation
A pre-trained AI model is a model that a developer has already trained on one or more datasets — often large, broad, or domain-specific datasets — so it has learned patterns before any single organization adapts it to a narrower task. Instead of building a model from nothing — collecting data, labeling it, designing an architecture, and training it at scale — a team starts from a model that already works and shapes it to the job at hand. In practice, the model may be used as delivered, configured, connected through an API, or fine-tuned on domain-specific data.
That shift is why pre-trained and foundation models have moved quickly from research into commercial products. They provide a starting point that can reduce the burden of building advanced AI capability from scratch. For many organizations, the benefits are concrete:
- Speed to value — capabilities that may take significant time and specialized resources to build in-house can often be evaluated or deployed more quickly.
- Potentially lower development burden — the organization may avoid some of the upfront data, compute, and specialized talent required to build a base model from scratch.
- Access to advanced capability — image analysis, language understanding, and pattern detection that few organizations could develop on their own.
- Focus on the real problem — scarce in-house expertise can be spent adapting the model to the business or scientific question, not rebuilding the base.
- A foundation for innovation — a strong base model puts advanced AI within reach of smaller teams and non-AI-first companies.
In regulated and scientific settings, that can translate into faster evaluation of medical image analysis, document and record handling, laboratory data interpretation, and case triage — work that once demanded highly customized development may now build on a capable model from day one.
In practice, most organizations do not build these models themselves. To accelerate development and reduce the burden of building a model from scratch, a company may acquire a selected third-party pre-trained model — proprietary, open-source, cloud-hosted, or embedded in a vendor platform — and incorporate it as one component of its own software solution. The pre-trained model becomes a core analytical component; the organization supplies the data handling, logic, interface, and controls around it. That division of labor — an external model running inside an in-house solution — is exactly the arrangement this article is about.
These are real advantages, and they help explain why life sciences organizations are evaluating pre-trained and foundation-model capabilities more seriously. But the same properties that make a pre-trained model powerful — trained somewhere else, on data the organization did not choose, and able to change on the supplier’s schedule — are exactly what turn it into a quality and compliance question.
The Model Arrived Trained, Not Qualified
Pre-trained AI models can make development faster. They may also make quality oversight harder.
A third-party pre-trained model may arrive with model weights, configuration files, technical documentation, release notes, known limitations, and supplier performance claims. The model may already be capable of classifying images, summarizing records, prioritizing cases, detecting patterns, generating recommendations, or supporting a regulated workflow.
But that does not mean the model is qualified for your use. Its behavior may have been shaped upstream by supplier-controlled training data, labeling methods, tuning decisions, architecture choices, thresholds, performance tradeoffs, and update practices. By the time QA sees the final software solution, part of the system behavior may already exist inside the model.
That changes the QA question. Not only:
Does the software work?
But also:
Can we justify that this specific model, from this specific supplier, integrated into this specific workflow, is fit for this specific intended use?
That is why pre-trained models should not enter a regulated environment through a side door. They should enter through the company’s AI governance process.
Pre-trained does not mean pre-qualified.
1. Govern Before You Integrate
A pre-trained model is not only a technical component. It is a governance object before it is a validation object. The first control is not a test script — it is deciding whether the model belongs in the company’s governed AI inventory.
A company AI Governance Policy should define how pre-trained models are identified, approved, risk-classified, sourced, integrated, monitored, updated, and retired. The policy does not need to contain every metric or test case, but it should establish the expectation that pre-trained models are controlled before regulated use.
This matters because the model’s risk depends on context. A model that only helps sort low-risk work queues is not the same as one that supports a clinical, laboratory, manufacturing, quality, or regulatory decision. The same technical model can carry different risk depending on how it is used. A governance review should capture at least:
- model name and version;
- supplier or source;
- business owner and system owner;
- intended use and excluded uses;
- affected system or workflow;
- whether the model is advisory, assistive, triage-oriented, decision-supporting, or autonomous;
- whether human review is required;
- whether the use case affects a regulated process, GxP record, patient workflow, quality decision, or inspection-relevant output;
- whether the model is third-party, open-source, cloud-hosted, API-based, embedded, fine-tuned, or part of a larger foundation-model platform.
Once the model is in the AI inventory and its risk is classified, the organization can determine the appropriate level of supplier qualification, local assurance, monitoring, human oversight, and change control.
A pre-trained model should not be integrated before it is governed.
Is your organization ready to govern the AI models it inherits?
Identify gaps in AI governance policy, model inventory, risk classification, and lifecycle control — before a pre-trained model enters a regulated workflow ungoverned.
2. Qualify the Supplier You Cannot Fully See Inside
Traditional software supplier qualification is not enough when the supplier provides a model whose behavior depends on training data, architecture, tuning, labeling, and future updates. Cybersecurity posture, quality certifications, contractual terms, and commercial reputation still matter — but they do not answer the full AI question. QA also needs to know what evidence comes with the model and what the supplier may control after implementation.
Supplier and Model Identity
QA should understand who developed the model and what role each party plays. The supplier may be the model developer, reseller, cloud host, system integrator, data provider, or only the interface layer between the customer and another model provider. That distinction matters. At minimum, the qualification record should identify:
- the model developer;
- the model provider;
- the data provider, where known;
- the cloud or hosting provider, if applicable;
- the system integrator, if different from the model provider;
- the owner of model updates, incident communication, and change notification;
- the exact model version, endpoint, or release being used.
If the supplier cannot clearly explain what model is being used, who controls it, and how updates are managed, QA should pause before treating the system as ready for regulated use.
Model Documentation
QA should request documentation proportionate to intended use and risk: a model card or system card, technical documentation, intended-use and excluded-use statements, validation and performance summaries, cybersecurity and dependency information, known limitations, release notes, and update policy. The point is not to collect documents for their own sake — it is to determine whether the supplier can provide enough evidence to support the model in the customer’s intended use.
Training and Data Provenance
For pre-trained models, data provenance becomes part of the quality story. QA may not always receive the full training dataset, especially for commercial or cloud-hosted models, but it should still understand what can be known about the data that shaped the model:
- What data were used to pre-train the model?
- What is known about the source, quality, and representativeness of the data?
- Were labels reviewed, adjudicated, or quality-checked?
- Were different sites, instruments, scanners, populations, workflows, or sample types represented?
- What gaps or limitations are acknowledged?
- What is the model not intended to do?
- Was the model later fine-tuned or adapted — and what data, parameters, or configuration changes were involved?
This is not a theoretical concern. FDA’s draft guidance on AI-enabled device software functions — which is specific to AI-enabled medical devices — recommends that submitters explain the pre-trained models used, including the dataset used for pre-training and how the model was obtained, and the same provenance question applies in any regulated context. NIST’s Generative AI Profile likewise treats third-party and pre-trained models as a governance concern, including how such models are adapted — through fine-tuning, retrieval-augmented generation, or other modifications — for a specific task. The quality question is straightforward:
What are we inheriting from the supplier, and is it appropriate for our intended use?
Supplier Lifecycle Control
The supplier’s responsibilities do not end at implementation. Pre-trained models may change through retraining, tuning, calibration, threshold changes, dependency updates, API changes, hosting changes, cybersecurity patches, or model replacement. Some updates are visible to the customer; others are abstracted behind a cloud service or platform release. QA should know how often the supplier updates the model; whether updates are automatic, optional, or customer-controlled; whether customers are notified before updates; whether prior versions can be retained and rolled back; whether release notes are detailed enough for impact assessment; and how dependency, API, hosting, threshold, or calibration changes are communicated.
Supplier qualification must move from“Can this vendor support software?” to“Can this supplier provide enough evidence for us to trust the model in our intended use?”
Is your supplier qualification ready for third-party AI?
Identify gaps in model documentation, provenance, intended-use fit, and update policies — the supplier evidence that standard vendor onboarding does not capture.
3. Integrate, Do Not Just Install
A pre-trained model is rarely the whole system. It usually sits inside a broader software solution with input data handling, pre-processing, model inference, thresholds, post-processing, business rules, user-interface design, human review, output storage, audit trails, reporting, monitoring, and change control. That means the model may perform well in supplier testing and still behave differently after integration.
The model is one component in the package. The supplier controls the model; the manufacturer owns the integration, the human workflow, and the controls around it.
This is where many organizations underestimate risk. Integration choices can materially affect the final output. Examples include pre-processing changes that alter the model input; threshold changes that affect false positives or false negatives; user-interface design that encourages overreliance; post-processing logic that changes the meaning of the output; failed model calls that are not handled properly; missing traceability between inputs, model version, configuration, and outputs; weak audit trails; and no fallback process when the model is unavailable.
Model must be validated in context, not in isolation.
4. Do Not Repeat the Vendor’s Tests — Own Local Assurance
There are two weak extremes. One is to accept vendor performance claims without local assessment. The other is to repeat the supplier’s entire model-development and validation program without considering what evidence is actually needed. The better approach is risk-based local assurance.
Vendor testing usually focuses on model development, training and tuning, supplier benchmark datasets, general performance claims, supplier-controlled validation, and model release readiness. Local assurance should focus on what only the regulated organization can confirm:
- intended-use fit and local data compatibility;
- integration behavior and workflow impact;
- user interaction and the human review process;
- output traceability;
- performance in representative operating conditions;
- regression after updates;
- alignment with the organization’s quality system.
The goal is not to redo the supplier’s model development. It is to verify that the received model, once integrated, remains fit for the regulated workflow. For a low-risk assistive use case, local assurance may be limited and focused. For a high-impact workflow, it may require formal acceptance criteria, representative test data, subgroup review, human-AI workflow testing, and post-deployment monitoring.
The point is not more testing. The point is better evidence.
5. Monitor What You Inherited
A pre-trained model does not become safe, reliable, or inspection-ready just because it passed a go-live test. Performance can change over time. Incoming data can shift. Workflows change. Users may learn to overtrust or ignore the output. Supplier updates may change model behavior. A system can perform well on average while performing poorly for a subgroup, site, scanner, assay, sample type, or workflow variation. That is why monitoring is part of governance.
Metrics should be selected based on intended use, risk, and the model’s role in the workflow — not copied blindly from a vendor report.
| Area | Metrics to Monitor | Why QA Should Care |
|---|---|---|
| Baseline performance | Accuracy, sensitivity, specificity, precision, recall, F1, AUC, false positive rate, false negative rate | Confirms whether the received model performs within expected limits |
| Intended-use performance | Performance by site, user group, instrument, scanner, assay, sample type, population, geography, or workflow | Confirms the model works in the actual operating context |
| Data similarity | Input distribution, image quality, acquisition parameters, missingness, protocol differences, sample characteristics | Detects mismatch between supplier training context and production context |
| Calibration & confidence | Confidence distribution, calibration error, uncertainty rate, reject/abstain rate | Helps detect overconfident incorrect outputs |
| Integration behavior | Pre-processing errors, failed model calls, latency, output formatting errors, threshold behavior, interface errors | Confirms the model works correctly inside the full software solution |
| Human-in-the-loop | Review rate, override rate, acceptance rate, escalation rate, disagreement rate, review turnaround time | Confirms whether human review is effective and meaningful |
| Data integrity | Model/configuration version traceability, input/output traceability, audit trail completeness, record retention, reproducibility | Supports traceability and inspection readiness |
| Drift | Input drift, output drift, performance drift, subgroup drift, workflow drift | Detects whether model reliability changes after deployment |
| Supplier lifecycle | Release frequency, update notices, patch history, dependency changes, known issues, vulnerabilities | Tracks supplier-controlled risk after go-live |
| Change control | Regression test pass rate, open defects, validation exceptions, rollback success, impact assessment completion | Confirms updates remain controlled and justified |
Not every system needs every metric. Every regulated use needs a justified monitoring strategy. The question is not“What metrics did the vendor report?” but“What metrics tell us whether this model remains credible in our intended use?”
Is your monitoring strategy built for AI you didn’t train?
Identify gaps in performance metrics, drift indicators, human-in-the-loop measures, and supplier-lifecycle signals — and build risk-based monitoring tied to your intended use, not the vendor’s benchmark.
6. Keep the Human in the Loop — As a Real Control
Human-in-the-loop is often treated as a comfort statement. It should not be. A clinician, analyst, operator, reviewer, or QA professional cannot be a decorative control. Human review is only meaningful when the human has authority, context, training, time, and a documented process to challenge the model. QA should ask:
- Who reviews the model output, and what qualifications and training are required?
- What information does the reviewer see? Can the reviewer override the model?
- Are overrides documented and disagreements trended?
- Is escalation defined? Is alert fatigue monitored? Is automation bias considered?
- Is the human-AI workflow tested? What happens if the human disagrees, or if the model is unavailable?
A human reviewer reduces risk only if the workflow gives the reviewer enough information and authority to act. If the interface hides uncertainty, pressures speed over review, or makes override difficult, the control may be weaker than it appears. It must be designed, trained, tested, and monitored.
Human-in-the-loop is a control only when the human can meaningfully challenge the model.
7. Control the Change You Did Not Make
Pre-trained models can change after implementation. The supplier may retrain the model, adjust tuning, change thresholds, update dependencies, modify API behavior, change hosting infrastructure, revise known limitations, patch vulnerabilities, or release a new version. Even if the regulated organization did not initiate the change, it may still own the impact on its validated workflow. That is why supplier updates must connect to change control.
A supplier release note is not a change-control record.
QA still needs an impact assessment against the actual intended use of the implemented system. Useful questions: Did the supplier retrain the model, or change its training data, labeling, tuning, architecture, weights, or configuration? Did thresholds or calibration change? Did API behavior, dependencies, or hosting change? Did cybersecurity controls, known limitations, or performance claims change? Were new issues introduced?
The impact assessment should consider intended use, risk classification, the model’s role in decision-making, affected workflows and users, data integrity impact, the human review process, validation impact, regression testing needs, rollback or fallback requirements, and documentation, training, and monitoring updates.
For AI-enabled medical devices,FDA’s Predetermined Change Control Plan (PCCP) guidance is an important reference, because it addresses how planned modifications can be managed while maintaining reasonable assurance of safety and effectiveness. Even outside the medical-device context, the principle holds: model change must be anticipated, controlled, assessed, and documented.
Change control should also include retirement. If the model is replaced, disabled, superseded, or no longer fit for purpose, the organization should define how it is retired, how records are retained, how outputs remain interpretable, and how downstream dependencies are handled. The lifecycle does not end at go-live — it ends when the model is retired and the evidence remains understandable.
Real-World Signals: This Is Not Theoretical
Public FDA summaries already show pre-trained and foundation-model concepts in regulated AI contexts.
FDA’s 510(k) summary forQure.ai qER-Quant (K211222) describes an analysis module containing a set of pre-trained convolutional neural networks, coupled with pre-processing and post-processing modules.
FDA’s 510(k) summary forQure.ai qXR-BT (K212690) separately describes an analysis module consisting of pre-trained convolutional neural networks with pre-processing and post-processing modules for chest X-ray analysis.
FDA’s 510(k) summary forAidoc BriefCase-Triage: CARE Multi-Triage CT Body (K252970) describes the device as using a foundation model-based AI system to analyze CT images and flag suspected findings.
These examples should be read carefully. They do not mean every healthcare AI tool, foundation model, or pre-trained model carries the same regulatory status — that depends on intended use, claims, workflow, risk, jurisdiction, and the model’s specific role in the final system. The point is simpler:
The regulatory question is not whether pre-trained models exist. They do. The QA question is whether the organization has enough evidence to control them.
Inspection-Ready Evidence Package
For regulated use, the evidence package should prove that QA understood the model, qualified the supplier, controlled the integration, justified the metrics, monitored performance, and remained informed when the supplier changed the model. A practical package may include:
- AI inventory record; AI governance review or approval record;
- supplier qualification record; supplier quality agreement or equivalent control;
- model identity and version record; model card or equivalent documentation;
- training-data summary or supplier attestation;
- intended-use and excluded-use statements; risk assessment;
- integration architecture diagram; requirements and traceability matrix; configuration specification;
- performance acceptance criteria; local assurance test results;
- human-in-the-loop procedure; monitoring plan; drift-monitoring plan;
- change-control procedure; supplier update notification process; regression testing strategy;
- release approval; periodic review record;
- deviation, incident, and CAPA linkage where applicable;
- retirement or decommissioning record when the model is removed or replaced.
The evidence package should prove that QA did not simply trust the model — it controlled the conditions under which the model was used.
Final Takeaway
Pre-trained AI models can accelerate innovation, but inheriting a capable model is not the same as qualifying it for a regulated workflow. The organization may not control the supplier’s original training process, yet it remains responsible for the full lifecycle: govern the model, qualify the supplier, validate the integration context, monitor the output, control human oversight, and assess supplier updates before the model can be trusted.
A pre-trained model may arrive trained, but it does not arrive pre-qualified.
In regulated AI, trust without traceability does not survive inspection.
Integrating AI Into a Regulated Workflow? Start With the Evidence.
Xevalics helps regulated organizations evaluate AI-enabled systems, third-party model suppliers, CSV/CSA strategies, data integrity controls, and lifecycle governance. Our support can include:
- AI governance policy and procedure development;
- AI inventory and risk classification strategy;
- AI supplier qualification strategy and pre-trained model evidence review;
- intended-use and risk assessment; integration control strategy;
- model monitoring and metrics planning; human-in-the-loop control design;
- change-control and impact-assessment procedures;
- inspection-ready validation and governance documentation.
If your organization is integrating a pre-trained model into a regulated workflow, the time to define governance, evidence, metrics, monitoring, and update controls is before the model goes live.
Frequently Asked Questions
What is a pre-trained AI model?
A pre-trained AI model is trained in advance on one or more datasets before being integrated, configured, or fine-tuned for a specific use case.
Why are pre-trained AI models relevant to QA?
Because part of the model’s behavior may be inherited from supplier-controlled training data, architecture, tuning, labeling, limitations, and update practices.
Should pre-trained models be included in a company AI Governance Policy?
Yes. The policy should define how pre-trained models are inventoried, risk-assessed, sourced, approved, integrated, monitored, updated, and retired.
Does QA need to repeat the vendor’s model testing?
Not necessarily. Use a risk-based approach: rather than repeating all supplier testing, confirm intended-use fit, integration behavior, monitoring controls, human oversight, and lifecycle governance.
What should supplier qualification include for pre-trained AI models?
Model identity, version control, documentation, intended use, known limitations, training-data summary, performance evidence, update policy, incident communication, and lifecycle controls.
What metrics should QA monitor after receiving a pre-trained model?
Performance, intended-use fit, data similarity, calibration, integration behavior, human review performance, data integrity, drift, supplier lifecycle activity, and change-control effectiveness.
Is human-in-the-loop enough to control AI model risk?
No. It is only effective when the reviewer has authority, training, context, time, documented procedures, and the ability to override or escalate model outputs.
How should vendor model updates be controlled?
They should trigger documented review, impact assessment, risk evaluation, regression testing where needed, release approval, and documentation updates before use in production.
Selected References
- FDA, Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations (draft, January 2025).
- FDA, Marketing Submission Recommendations for a Predetermined Change Control Plan for Artificial Intelligence-Enabled Device Software Functions.
- NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile.
- IMDRF/AIML WG/N88 FINAL:2025, Good Machine Learning Practice for Medical Device Development: Guiding Principles.
- ICH, Q9(R1) Quality Risk Management.
- ICH, Q10 Pharmaceutical Quality System.
- FDA 510(k) summary, Qure.ai qER-Quant (K211222).
- FDA 510(k) summary, Qure.ai qXR-BT (K212690).
- FDA 510(k) summary, Aidoc BriefCase-Triage: CARE Multi-Triage CT Body (K252970).
- Google Search Central, Article structured data.
Implementation note: use Article structured data, not FAQ schema as a rich-result strategy (per Google Search Central). Keep the FAQ as visible on-page content.
