Across decision theory, validation, fairness, Bayesian modeling, and scientific computing, the recurring problems began after a model produced its predictions.
The model review that cannot end with model performance
A healthcare analytics team has finished the familiar parts of a model review. Its risk model discriminates reasonably well between patients who will and will not experience the outcome. Calibration plots look acceptable. Performance has been checked on a holdout set. Several candidate models are close enough that any one of them could probably support a deployment proposal.
The meeting is still nowhere near a decision.
The outcome label comes from diagnosis codes, so an undocumented case could mean absence of disease or absence of detection. The intervention team can contact only a fixed number of patients each week. Sensitivity differs across demographic groups, although their baseline risks also differ. The model will be moved to another hospital whose patient mix and care process are not the same. Some of its uncertainty estimates came from approximate computation whose speed was measured more carefully than its inferential effect.
None of these concerns is repaired by adding another model to the benchmark. The team has to understand how the outcome became observable, what action the score will trigger, how evidence transports, which approximations entered, and what uncertainty should reach the people making the decision.
That hypothetical meeting captures the pattern I kept finding in the sessions I selected at JSM 2026. As I described in the first essay, the itinerary was built near the edge of my existing competence. It was shaped by healthcare analytics, statistics, data science, and software rather than designed to represent the conference as a whole. Within that selected path, however, a striking amount of the technical work concerned the system around a prediction.
The talks reached the problem through different routes. Decision theory began with the action, while targeted validation began with the cost of obtaining trustworthy measurements. High-performance statistical computing examined representation, numerical precision, and hardware. Model criticism allowed for the possibility that every candidate was wrong. Together, they put pressure on a longer chain: measurement, representation, estimation, computation, validation, and decision.
A prediction inherits the decision that follows it
Suppose an organization has noisy estimates of the value of acting on many units: hospitals considered for intervention, neighborhoods considered for a program, or experiments considered for deployment. For unit $i$, let $\mu_i$ denote its unknown value and $K_i$ the cost of selecting it. The utility of a selected set $S$ can be written as
$$ U(S;\mu) = \frac{1}{n} \sum_{i\in S}(\mu_i-K_i). $$This expression makes the action visible. An estimation error matters most when it changes whether a unit crosses the selection boundary. A large error far from that boundary may have no operational effect, while a small error nearby can add or remove a hospital from the intervention list.
That was the problem in Timothy Sudijono’s talk, “Compound Selection Decisions: An Almost SURE Approach”. The corresponding manuscript develops an approximately unbiased estimate of selection welfare for choosing among candidate rules. The usual temptation would be to tune an estimator for mean-squared error and threshold the result afterward. Selection welfare gives different weight to errors because the threshold determines which errors change the action.
The practical lesson reaches beyond the particular estimator. If a hospital program will act on the top 30 facilities whose estimated opportunity exceeds an intervention cost, model tuning should reflect the consequences of selecting the wrong facilities. A modest reduction in average estimation error is less useful when it leaves the selection mistakes unchanged.
The clinical fairness session reached the same issue through a more familiar metric. Sensitivity, or the true positive rate, tells us what proportion of eventual cases cross a threshold. It does not combine the benefit of detecting a case with the burden of unnecessary intervention. In the subgroup-net-benefit talk by Jose Benitez-Aurioles, those consequences entered the criterion directly:
$$ \text{NB}_g(p_t) = \pi_g\text{TPR}_g(p_t) - (1-\pi_g)\text{FPR}_g(p_t) \frac{p_t}{1-p_t}. $$Here $\pi_g$ is outcome prevalence in group $g$, and $p_t$ is the threshold probability at which the benefit of acting is judged to outweigh the harm. The equation explains why equal sensitivity does not guarantee equal clinical benefit. Two groups can have the same true positive rate while differing in prevalence, false-positive burden, or the number of available treatment slots. Unequal sensitivities can also coexist with similar net benefit.
The paper behind the talk goes further by separating equality in model-generated benefit from improvement in final health equity. A program may create more benefit in a higher-risk, disadvantaged group and reduce an outcome gap even though a conventional parity metric becomes less equal. Under a capacity limit, the choice also becomes an allocation problem: equalizing one rate can redirect scarce intervention slots and change both total benefit and its distribution.
This does not make sensitivity irrelevant. Operational teams still need to know who is being missed. The point is that sensitivity answers a detection question, while net benefit answers a decision question under specified trade-offs. A model governance meeting needs both views when the intervention pathway makes both consequential.
The experimental-site-selection talk in the same decision-theory session supplied another version of this argument. A site with the largest expected treatment effect may provide little new information if the policy decision there is already clear. A site near a decision boundary, especially one whose result informs several related locations, can have greater value. The useful experiment is determined partly by the decisions the evidence may change.
These examples suggest a concrete ordering for applied work. Define the action and its constraints, state the utility or loss, and then choose the model or design criterion. When the order is reversed, the team may optimize a convenient intermediate score and discover only at deployment that it values the wrong errors.
The data available for analysis were produced by systems
In an electronic health record, an absent diagnosis code is easy to read as a negative label. For many diseases, that interpretation is too strong. The patient may have the disease without having received the test, specialist visit, or documentation needed to make it visible in the record.
Weidong Ma’s talk, “A Novel Framework for Addressing Disease Under-Diagnosis Using EHR Data”, separated disease status from documented diagnosis. Confirmed cases can often be treated as positives, while patients without documentation contain both true negatives and hidden cases. A classifier trained directly on the code learns the probability of documentation, which is jointly shaped by disease, care access, testing, utilization, and clinician behavior.
The corresponding Biometrics paper uses gold-standard ascertainment for a selected subset of patients without a documented diagnosis, then combines those validated observations with the much larger EHR sample. This allows the analysis to distinguish two questions: who likely has the disease, and among patients with disease, who becomes diagnosed.
That distinction changes both modeling and intervention. Suppose a health system uses the model to identify possible undiagnosed liver disease. A high score may justify additional review or testing; it is not itself a diagnosis. If diagnosis rates differ because one group receives less testing, a model trained on documentation may reproduce the detection process the program is supposed to correct. The team needs targeted adjudication, design-aware estimation, and a report that keeps predicted disease risk separate from observed care.
Targeted validation treats accurate measurement as a design resource. The gold-standard sample need not resemble a simple random sample when the selection probabilities are known and the final estimator accounts for them. In the same session, Sarah Lotspeich used the difference between straight-line and route-based distance to healthy food as a concrete measurement problem. A short line on a map can cross a river, highway, or rail corridor that makes the practical route much longer. If route queries are expensive, the useful question is which neighborhoods should receive them so that the final access or health-disparity estimate improves.
The choice depends on the estimand. Neighborhoods near an eligibility threshold may be informative for classification. Areas whose road layout makes proxy error likely may be informative for a regression correction. An equity-focused design may deliberately buy more accurate measurements in communities where uncertainty is greatest. The related paper shows why measurement design and policy targeting cannot be separated cleanly when the proxy helps determine who qualifies for intervention.
Selective labels create a harder version of the observation problem because outcomes are visible only after a prior decision. Credit repayment is observed for approved applicants. Job performance is observed for hired candidates. Some clinical outcomes are recorded only for patients who received a test or treatment. In Yiqi Liu’s talk on algorithmic frontiers with selective labels, the performance of alternative rules could not always be reduced to a single identified point. The public manuscript studies feasible loss sets and their frontiers under different assumptions about selection.
This changes what honest comparison looks like. When prior policy hides the counterfactual outcomes required to compare two algorithms, more aggressive modeling does not automatically recover them. Under weak assumptions, the defensible output may be an identified region: a set of performance combinations compatible with what was observed. Stronger assumptions can narrow that set, but the gain comes from added structure rather than from the data alone.
An applied team should therefore map the observation process before fitting the predictive model. Which variables are direct measurements? Which are proxies generated by care or policy? Which outcomes appear only after a prior action? Which records were selected for gold-standard review? The answers determine what can be estimated, what requires sensitivity analysis, and where additional information would be worth buying.
Approximation is part of the scientific claim
Large analyses rarely compute the ideal mathematical object without approximation. They use mini-batches, finite Monte Carlo samples, reduced precision, surrogate models, truncated bases, compressed arrays, or some combination. The important question is what inferential quantity these choices preserve.
Jonathan Huggins’s talk on robust and calibrated generalized Bayes at scale made this issue visible in stochastic-gradient computation. A schematic stochastic-gradient Langevin dynamics update is
$$ \theta_{t+1} = \theta_t + \frac{h}{2}\widehat g_t(\theta_t) + \sqrt{h}\,\xi_t $$The step size is $h$, $\widehat g_t$ is a gradient estimate from a mini-batch, and $\xi_t$ adds Gaussian noise. The algorithm replaces an expensive full-data gradient with a noisy estimate and then adds another source of noise intended to support sampling. Its stationary uncertainty depends on the interaction among step size, batch size, objective curvature, gradient noise, and model misspecification.
The inferential risk is easy to miss because an algorithm can look stable while targeting the wrong covariance. Parameter averages may be adequate even when interval widths are not. Recent work associated with the talk studies large-sample uncertainty for stochastic-gradient MCMC and subsampling MCMC for latent-variable models. Both reinforce a distinction that matters in practice: computational speed and uncertainty accuracy require separate evaluation.
Consider a large hierarchical health model fitted with mini-batches. If the final decision depends mainly on a population-average prediction, a particular approximation may be acceptable. If the report compares small subgroup effects or tail probabilities, the same tuning can distort the quantity that carries the decision. Benchmarking iterations per second does not resolve that difference.
The high-performance statistical computing session showed the same principle at a much larger physical scale. Yan Song’s talk on exascale climate emulators described stochastic generators for high-resolution global climate simulations. Spherical-harmonic representations move the spatial field into a basis aligned with the geometry of the globe. Low-rank structure, distributed computation, mixed precision, and accelerator-oriented software then make generation possible at a scale where a direct covariance calculation would be infeasible.
The scientific target is a distribution, not only a predicted mean. An emulator used for extreme-event research must preserve relevant tails, spatial dependence, temporal persistence, and derived climate summaries. A general acceptance criterion can be written as
$$ d_k\{g_k(Y),g_k(\widetilde Y)\} \le \varepsilon_k, \qquad k=1,\ldots,K, $$where $Y$ is simulator output, $\widetilde Y$ is its approximation, $g_k$ is a scientific summary, and $d_k$ measures discrepancy. The tolerances $\varepsilon_k$ express how much distortion the downstream use can accept. The stochastic-generator paper and its exascale extension make the statistical representation inseparable from the computing architecture.
The session’s talks on mixed precision and scientific compression widened the point. Low-order bits can contain spatial structure without affecting a scientifically meaningful summary. A low-precision calculation can produce a small parameter error that is harmless for prediction and unacceptable for a tail probability. Information-theoretic structure, numerical fidelity, and inferential value are related, but none substitutes automatically for another.
This gives teams a more useful way to budget approximation. Instead of asking whether compression is “accurate” or a sampler is “fast enough,” name the downstream quantities first. Check error in those units, record the numerical and hardware path that produced the result, and use higher precision or more computation where the target is sensitive. Approximation becomes reviewable once its allowed loss is attached to the scientific use.
Model criticism reaches beyond the fitted equation
Model selection often ends with an index: candidate 3 has the lowest average loss. That output hides two questions. How uncertain is the ranking, and is candidate 3 adequate for the intended use?
Jeffrey Miller’s talk, “Robust model selection using likelihood as data”, treated the per-observation losses across candidate models as multivariate data. For observation $i$ and model $k$, define
$$ L_{ik}=-\log p_k(Y_i). $$Every model scores the same observation, so the losses are paired. Their covariance contains information about how reliably the candidates can be distinguished. If two models fail on the same cases, a comparison that ignores this pairing discards part of the evidence. The preprint uses the joint loss vectors to estimate relative risks and uncertainty rather than returning an unsupported winner.
This admits outcomes that model-selection workflows often suppress. Two candidates may be statistically tied. Several may be practically equivalent. The best candidate may still be poor in absolute terms. Cross-fitting is needed when the models were estimated on the same observations used for evaluation. For a clinical team, a tie can justify choosing the simpler or easier-to-monitor model; poor absolute adequacy can stop deployment even when a relative winner exists.
Gemma Moran’s talk on aggregated posterior checks for generative modeling extended criticism into the generation pipeline. A variational autoencoder may regularize latent representations toward a prior without making draws from that prior adequate for generation. The relevant object is the implemented sequence that samples a latent value and decodes it. Held-out comparisons can test whether that sequence reproduces the target population, including rare groups and modes. A training penalty is evidence about optimization behavior; it is not a substitute for checking the generator people will actually use.
The Bayesian-calibration session supplied a more physical form of criticism. A common calibration model writes observations as
$$ y(x)=\eta(x,\theta)+\delta(x)+\epsilon, $$where $\eta(x,\theta)$ is simulator output at calibration parameter $\theta$, $\delta(x)$ is model discrepancy, and $\epsilon$ is observation error. If the discrepancy is highly flexible, many pairs of $\theta$ and $\delta$ can fit the same data. The posterior can contain long ridges or several modes because the observations do not uniquely separate parameter error from simulator inadequacy.
Devin Francom’s talk on the folk theorem of statistical computing in physical-model calibration treated slow mixing and sensitivity to initialization as diagnostic evidence. Sometimes the model should change: parameters can be reexpressed, discrepancy can be constrained by scientific knowledge, or phase and amplitude can be modeled separately. The related elastic-calibration work develops that last idea for functional responses whose timing and magnitude vary.
Other cases require stronger computation rather than a smoother story. Two physically different parameter regimes may produce nearly identical observables. A multimodal posterior then expresses genuine ambiguity. Replacing it with a unimodal approximation or reporting a posterior mean between modes would make the output easier to summarize while weakening its physical meaning.
The diagnostic order matters. Verify the implementation and units. Inspect the geometry from several starting points. Ask whether the dependence or modes are scientifically plausible. Revise the model when the revision has substantive support, and upgrade the computation when the complexity remains. Effective sample size is useful evidence, but it cannot decide whether ambiguity belongs to the algorithm, the model, or the science.
The recurring object was the inferential system
Return to the clinical model review from the opening. The team wants to identify patients for an outreach program with limited weekly capacity.
The diagnosis label first needs an observation model because documentation depends on testing and access. A targeted validation sample determines how well hidden cases can be distinguished from true negatives. Moving the model across hospitals requires evidence about which disease and documentation relationships transport. Approximate computation must preserve the subgroup quantities used in review. The threshold must reflect the benefit of outreach, the burden of unnecessary contact, and the capacity of the clinical team. Fairness assessment needs both operational metrics, such as who is missed, and decision metrics, such as how benefit is distributed. Model comparison must retain the possibility of a tie or inadequate absolute performance.
Each layer can change the final claim. Better disease estimation can shift who crosses the threshold. A new hospital’s testing process can make the transported score mean something different. A faster sampler can narrow uncertainty for computational rather than scientific reasons. Equal sensitivity can coexist with unequal benefit. A relative winner can fail every absolute check that matters for deployment.
This view assigns the model equation an important but bounded role. The analysis is the full path connecting institutional measurement, statistical assumptions, computation, validation, and action. Review has to follow that path because the result inherits every choice along it.
The sessions I attended do not establish that this is the only center of contemporary statistics. They do show why the field cannot be understood only through a catalog of predictive models. The most persistent questions in my itinerary concerned the claims and decisions built around those models.
That leaves the interpretive question for the final essay: as producing plausible predictive models becomes easier in many workflows, what distinctive contribution does statistical reasoning make to the systems in which those models operate?