When Black-Box Prediction Is the Wrong Tool

The question is not whether a black-box model can predict accurately. Many can.

The question is whether predictive accuracy is sufficient evidence for the decision the system is being asked to make.

For a low-consequence task with abundant representative data, immediate feedback and reversible errors, an opaque predictor may be entirely appropriate. Product ranking, document routing and perceptual classification can all benefit from models whose internal representations are difficult to interpret.

The standard changes when an output can affect health, safety, mission continuity, legal rights, capital equipment or a scientific conclusion. In those settings, the organisation may need to know more than what the model predicts. It may need to establish which evidence supports the output, whether the operating conditions are familiar, which constraints apply, what could make the result invalid and who is accountable if it is wrong.

That is the boundary this article examines. It is not an argument against machine learning. It is an argument for choosing a modelling approach from the requirements of the decision rather than from the availability of a fashionable technique.

“Black box” describes an accountability problem

Black box should not be used as a synonym for every complex model. Opacity is relative to the question that must be answered.

A model is operationally black-box when the people responsible for using its output cannot inspect or test the aspects of its reasoning that matter to the decision. That can happen because the learned mapping is too complex to understand directly, because the data or model are proprietary, because documentation is inadequate or because a post-hoc explanation does not faithfully represent what the model used.

Conversely, a technically complex system can still expose meaningful constraints, uncertainty, provenance, failure conditions and validation evidence. A mathematically simple model can remain a black box if nobody can establish how it was trained, what its score means or where it is valid.

The issue is therefore not model size alone. It is whether the system supplies the evidence required for responsible action.

Prediction is not the same as a decision

A predictor estimates an output from an input. A decision adds objectives, constraints, consequences and a policy for acting under uncertainty.

Average predictive accuracy does not specify the cost of different errors. A false negative and a false positive may have radically different consequences. It does not show whether the model relies on an acceptable variable, whether a mandatory physical or procedural constraint is satisfied or whether the current case lies inside the conditions represented by the training data.

This distinction matters because a model can optimise the stated target while the target itself is wrong.

Obermeyer and colleagues examined a widely used health-management algorithm that used predicted healthcare cost as a proxy for health need. At the same risk score, Black patients were considerably sicker than White patients. The authors estimated that correcting the disparity would increase the proportion of Black patients receiving additional help from 17.7% to 46.5%. The predictive target—cost—was measurable, but it did not represent illness equitably because access to care and spending were unequal. Obermeyer et al., Science (2019)

That result does not show that black-box models are uniquely biased. A transparent rule can encode the same defective proxy. It shows why high-stakes modelling requires inspection of the objective, data-generating process and decision consequences, not only the score obtained against a convenient label.

High liability changes the burden of proof

When the cost of failure is high, the evidence package must be stronger than a benchmark result.

The system should specify its intended use, foreseeable misuse, material error types, validation population, operating limits and fallback behaviour. It should provide enough traceability to investigate a failure and enough monitoring to determine when the deployed environment no longer resembles the validated one. Human review is useful only when the reviewer has relevant information, sufficient time and real authority to reject the output.

This is a risk argument, not a demand for perfect prediction. No model is error-free, including an interpretable or mechanistic one. The requirement is that residual risk be measured against the consequence and that the system fail in a controlled way when its evidence is insufficient.

The published NIST AI Risk Management Framework 1.0 makes the same separation. It treats validity and reliability, safety, accountability and transparency, explainability and interpretability, and harmful-bias management as distinct properties of trustworthy AI. Its measurement guidance calls for generalisability limits to be documented and for systems to fail safely when operated beyond their knowledge limits. NIST AI RMF 1.0 (2023)

A single accuracy number cannot satisfy those requirements because it does not describe the system-level risk.

Sparse data limits what can be learned

Sparse data is not simply a smaller version of abundant data. It changes what can be identified and how confidently performance can be estimated.

Several problems appear at once:

• many different functions may fit the observed examples;
• rare but important failure modes may not occur in the sample;
• subgroup performance may be estimated from only a handful of cases;
• flexible model selection can adapt to noise; and
• the test set itself may be too small to support a narrow confidence interval.

Cross-validation does not create independent evidence. It reallocates the observations that already exist. If feature selection, parameter choices or model selection use information from evaluation folds, the reported performance can become optimistic.

Vabalas and colleagues demonstrated this in simulation. With Gaussian-noise features and no real signal, ordinary K-fold cross-validation performed after model development produced accuracies above the 50% chance level, with bias still detectable at a sample size of 1,000 in their setup. Nested cross-validation and a genuinely separated train/test design produced estimates close to chance. The result is about validation leakage and selection procedure, not a claim that K-fold cross-validation is always invalid. Vabalas et al., PLOS ONE (2019)

When observations are scarce, the model should be asked to make a correspondingly narrow claim. Useful responses include simplifying the hypothesis space, incorporating defensible domain constraints, reporting uncertainty intervals, designing new experiments and withholding decisions where evidence is inadequate. None of these actions manufactures information that the data do not contain.

Opacity becomes especially problematic here because a good fit may conceal which of many data-compatible relationships the system selected.

Regime change breaks historical relationships

Most supervised prediction assumes that the relationship learned from historical data remains sufficiently stable at deployment. Regime change violates that assumption.

A regime can change because a policy alters incentives, a manufacturing process is modified, a treatment standard changes, a sensor is replaced, a new failure mode appears, an adversary adapts or the environment moves beyond the conditions previously observed. Sometimes the input distribution changes. In more serious cases, the relationship between inputs and outcomes changes as well.

Historical correlation offers no automatic protection. A variable that was predictive because it tracked one operating process can stop being informative when the process changes. The model may continue to return precise-looking outputs because producing a number is not the same as recognising that its premise has failed.

The WILDS benchmark was designed around distribution shifts that occur in real applications. Its ten datasets include shifts across hospitals, camera locations, geography and time. Across every dataset in the original benchmark, standard training produced substantially lower out-of-distribution performance than in-distribution performance, and the gap remained under the evaluated robustness methods. Koh et al., ICML (2021)

This does not mean that every shift will cause failure or that black-box models cannot be made more robust. It means that random held-out data from the historical regime do not test a claim about a new regime. Validation should reproduce the actual boundary—future time periods, new sites, unseen equipment, altered policies or other conditions that define deployment.

Where regime changes are expected, the system also needs change detection, performance monitoring against delayed ground truth, a defined response to drift and an authority capable of suspending the model. Retraining is not itself a control unless the new data, validation threshold and release process are specified.

Out-of-distribution use is an unsupported query

Regime change concerns a shift in the world or process. Out-of-distribution use can occur even when the wider regime is stable: a single case may lie outside the combinations, ranges or populations represented in development data.

Examples include a battery operated beyond the tested temperature range, a drug combination absent from the evidence base, an aircraft manoeuvre outside the flight envelope used for validation or a patient population not represented in a clinical dataset. The input may look numerically ordinary while being scientifically or operationally novel.

Interpolation and extrapolation must therefore be defined in domain terms. Distance in a feature space is not necessarily distance in the physical, biological or causal process. Two records can be statistically close while differing in the variable that changes the governing relationship.

Confidence scores do not solve this automatically. Ovadia and colleagues benchmarked predictive uncertainty under progressively shifted data across image, text, advertising and genomics tasks. Accuracy and calibration degraded under shift, and standard post-hoc calibration was not sufficient across the evaluated conditions. Ovadia et al., NeurIPS (2019)

The correct control is an explicit operating domain: what has been tested, what has not, how novel conditions are detected, when the system abstains and what happens next. An out-of-distribution detector can support that policy, but its own detection limits must also be validated. It is not a certificate that every unfamiliar case will be recognised.

Explanation is a technical requirement, not a graphic

“Explainable” is incomplete unless the intended question is stated.

A user may need to know:

• which observations materially influenced this output;
• which rule, constraint or threshold controlled the decision;
• whether changing an input would change the result;
• whether the model's relationship is causal or merely predictive;
• why two apparently similar cases received different outputs; or
• what evidence would invalidate the conclusion.

These are not interchangeable explanations. A feature-attribution chart does not establish causality. A local approximation does not describe global behaviour. A plausible narrative does not prove that the original model used the stated reasoning.

Post-hoc tools can still be useful for debugging and exploration, but their faithfulness must be tested. Adebayo and colleagues applied model- and data-randomisation tests to saliency methods and found that some produced similar-looking explanations even when model parameters or training labels were randomised. Such a map could be visually persuasive without reflecting what the trained model had learned. Adebayo et al., NeurIPS (2018)

Where explanation is necessary for approval, diagnosis, scientific interpretation or failure investigation, it should be treated as a testable output. The required explanation type, intended audience, fidelity criterion and known limitations should be declared before deployment. If a faithful explanation cannot be produced at the required level, a more inherently interpretable model or a different system architecture may be the appropriate choice.

Interpretability does not guarantee truth. It makes assumptions and failure paths more available for scrutiny.

Scientific accountability requires more than forecasting

Science uses models for several different purposes: measurement, prediction, hypothesis generation, explanation and intervention. Strong performance in one role does not establish validity in another.

A black-box predictor may identify samples worth testing, detect structure in high-dimensional observations or serve as a strong empirical baseline. Those are legitimate scientific uses. Difficulty begins when predictive association is presented as evidence of mechanism, causation or a valid intervention outside the observed conditions.

Scientific accountability requires the claim to be falsifiable. The report should identify the target quantity, assumptions, data provenance, exclusions, validation boundary, uncertainty, alternative explanations and conditions under which the claim would fail. If known conservation laws, biological pathways, temporal ordering or operational constraints are relevant, the analysis should state whether and how they are respected. A model that cannot expose these relationships may still predict, but it cannot by itself justify a scientific explanation of the system.

The problem is intensified by underspecification. D'Amour and colleagues showed across computer vision, medical imaging, language, clinical risk prediction and genomics that a training pipeline can return multiple models with similarly strong in-distribution test performance but materially different behaviour in deployment domains. The test score did not identify which model had learned a relationship that would remain valid. D'Amour et al., Journal of Machine Learning Research (2022)

Reproducibility of code is not enough when many equally reproducible predictors support conflicting behaviour. Scientific use also needs a reason to prefer the relationship embodied by the chosen model.

When black-box prediction is appropriate

Black-box prediction can be the right tool when the system satisfies most of the following conditions:

• the task is prediction rather than causal explanation or intervention design;
• training data are sufficiently large and representative of the operating population;
• deployment conditions are stable or shifts can be tested directly;
• errors are low-consequence, reversible or independently caught;
• ground truth arrives quickly enough to monitor performance;
• the system can abstain or fall back safely;
• no governing requirement depends on tracing the internal rationale; and
• the predictive gain over a simpler alternative is material under the relevant validation design.

Even then, “black box” does not mean “unmanaged.” Data lineage, external validation, uncertainty, security, monitoring and change control remain system requirements.

The point is proportionality. An opaque image classifier that assists an expert in organising a reversible workflow presents a different evidence burden from an opaque system that initiates a treatment, changes an industrial control setting or makes an unreviewable eligibility decision.

A decision test before model selection

Before choosing a modelling approach, an organisation should be able to answer eight questions.

1. Consequence: What happens after each type of error, and can the action be reversed?
2. Evidence: Which populations, sites, regimes and edge cases are represented in the data?
3. Target: Does the training label measure the real objective, or only a convenient proxy?
4. Stability: Which relationships are expected to persist, and what could change them?
5. Explanation: Who needs an explanation, for what decision and at what level of fidelity?
6. Scope: How will the system recognise unsupported or out-of-distribution use?
7. Control: Who can override, suspend or retire the system, and on which evidence?
8. Accountability: What record will allow an independent reviewer to reconstruct the decision and its validation basis?

If these questions can be answered without inspecting internal reasoning, a black-box predictor may be defensible. If the decision depends on constraints, causal structure, sequence, failure modes or scientific explanation that the model cannot expose, predictive accuracy alone is the wrong acceptance criterion.

The alternative is not necessarily one particular model class. It may be a simpler interpretable predictor, a rule-governed system, a model grounded in known structure, a hybrid architecture, a staged experiment or a decision process that keeps critical authority with a qualified human. The correct choice follows from the evidence requirement.

The Mondren perspective

Mondren does not treat mechanistic software and AI as opposing categories. Pattern recognition, perception, language and automation are valuable where their assumptions match the problem. The boundary appears when a decision must remain grounded in physical, biological or operational structure that historical correlation alone does not establish.

Our public position is therefore methodological: begin with the system, the required outcome and the cost of being wrong. Determine what must remain valid under sparse evidence, changing regimes and novel conditions. Then select the least opaque approach capable of meeting the performance requirement and the most rigorous validation design the decision demands.

This article makes no claim that a Mondren model has been benchmarked against the cited systems, and it presents no proprietary architecture, feature construction, model parameters or implementation logic. The problem classes, evidence requirements and evaluation principles are public. Construction remains proprietary.

The commercial implication

Model procurement often begins with the wrong comparison: which vendor reports the highest accuracy?

A defensible procurement process begins with the intended decision. It specifies the cost of error, deployment population, operating range, explanation obligation, update process and exit condition. Vendors can then be asked for evidence that matches those requirements: external and temporal validation, subgroup results, uncertainty calibration, out-of-scope behaviour, audit records, monitoring thresholds and failure recovery.

This changes the commercial question from “How intelligent is the model?” to “What decision can this system support, under which conditions, with what evidence and accountability?”

Black-box prediction is not the wrong tool because it is complex. It is the wrong tool when the decision requires evidence that the system cannot provide.

Accuracy can establish that a model predicted a test set. It cannot, on its own, establish that the objective was correct, the operating regime will persist, an unfamiliar case has been recognised, an explanation is faithful or a scientific conclusion is justified. Those are separate claims. In high-consequence systems, they must be tested separately.