Back to Research
  • Modeling
  • Blog
  • Code

sampleworks: a framework for generating and improving biomolecular ensembles with AI protein structure predictors

11 min read

Words by:

Smiling bearded man wearing round glasses and a collared shirt
Marcus CollinsSenior Machine Learning Scientist

Generating ensembles of structures using generative AI and experimental data.

Prism’s goal is to measure and study conformational ensembles of proteins–the set of structures between which proteins shift to achieve their function. To do that, we want to leverage the latest developments in AI and ML. Generative AI methods have revolutionized biomolecular structure prediction in the last several years. Models like AlphaFold3, Boltz-2, RosettaFold3, and Protenix demonstrate near atomic accuracy in their predictions. But that accuracy masks a problem: these structure predictors were largely trained on single, static structures. By contrast, the experiments used to determine structures in the Protein Data Bank (PDB) reflect conformational ensembles, so static PDB models don’t fully explain the underlying data. That observation led us to two questions:
  1. 1.
    What fundamental capacity do existing AI structure predictors have to generate a representative conformational ensemble of a macromolecule, like an enzyme whose function depends on its ability to shift between conformations?
  2. 2.
    How can we best adapt these predictive models to better generate ensembles? Are there particular methods that work better than others, and if so, why?
The most recent structure prediction models share a general architecture that allows us to “steer” their output to better fit experimental data, such as electron density derived from X-ray macromolecular crystallography. We hypothesize that steering AI structure predictors towards experimental data will allow us to build more accurate, biologically relevant ensemble models of biomolecules like proteins.
We’re not the first to attempt this, and while others have shown the method works in principle, significant gaps remain. Every published work until recently was bespoke: they each implemented one method for steering the AI structure predictor’s output (there are many) and one kind of data into one structure predictor. Further, the sets of example proteins studied were disjoint. It was impossible to judge which methods were best, or how well each method would generalize to other biomolecules.
To achieve our goal of using and benchmarking AI structure predictors to generate conformational ensembles that accurately reflect experimental data, we needed to address three challenges. We need to compare different structure predictors and steering methods to identify gaps and figure out what works best. We need to be able to easily steer based on any kind of experimental data or restraints, and combine these together as well. And we need to build a flexible and sustainable framework to work with these structure predictors, steering techniques, and data.
To do all that, we built sampleworks, our framework for controlled generation of macromolecular conformational ensembles. In the remainder of this post, I’ll share how we used sampleworks to compare AI structure predictors and steering methods, develop and incorporate new methods, and study conformational ensembles grounded in real experimental data. I’ll discuss evidence that current AI structure predictors have memorized their training data, limiting what they can generate even with guidance. Then I’ll wrap up by discussing what our results point to for the future.

sampleworks: a modular framework for steering AI models to fit experimental data with ensembles.

Sampleworks architecture iteratively updates input features using ensemble model loss before evaluating protein structures
The Sampleworks architecture, Figure 1 from https://thestacks.org/publications/sampleworks-release. Figure Credit: Karson Chrispens.
In just the last year or so, we’ve seen a small explosion of work leveraging AI structure predictors to better fit experimental data used to determine biomolecular structures, like X-ray crystallography or cryoEM. These predictors often use multiple sequence alignments as input, transform those into numerical “features”, and use those features to transform a random cloud of atoms iteratively into a valid protein structure–a process called “denoising”. As the denoising process is stochastic, it seems naturally suited to generating ensembles of structures.
These predictors can be “guided” or “steered” in several ways, e.g., by adjusting the steps of the denoising process, or directly optimizing the numerical representation of the protein, to fit experimental data. However, most AI structure predictors have neither a built-in mechanism for steering (a notable exception is Boltz-2), nor any built-in notion of ensembles. These predictors can produce many samples of the same protein, but those samples are not meaningfully related to each other. To address these gaps, previous implementations of experimental data guidance modified each structure predictor’s code directly, one by one.
sampleworks solves both of these problems for any structure predictor. The architecture is designed so that the guidance or steering methods sit outside the core AI models themselves, which live in “model wrappers”. Since sampleworks tracks the ensemble as a set of structures, that guidance naturally flows back to optimize ensemble properties, rather than individual conformations within it. sampleworks allows us to integrate nearly any structure predictor with a couple of days of effort. For example, we recently added Protpardelle-1c and are testing BioEmu2 now. While here I’ll discuss results with one kind of experimental data (electron density maps), we’ll soon release code to combine Bragg diffraction data and X-ray diffuse scattering, separately or together. Further, it is built to easily support additional experimental data with an existing forward model. Our package includes a geometry-restraint loss function that combines easily with experimental-data losses and serves as a template for implementing other restraints in the future. sampleworks therefore brings ensembles, experimental data, and guidance to any AI structure predictor, with minimal extra effort.
With sampleworks, we can now answer one of our key questions: what capacity do these AI structure predictors have to generate ensembles that reflect the observed data?

AI structure predictors appear to have memorized their training data.

The sampleworks framework allowed us to steer using four leading protein structure predictors: two versions of Boltz-2 (XRD and MD), Protenix, and RosettaFold3. All modeling can be done in one framework, allowing us to make direct comparisons between the predictors on the same test set of proteins, while holding other parameters constant. We selected 40 proteins from the PDB, each with multiple segments containing “altlocs”: alternative conformations of side chains or parts of the protein backbone. There are about 800 such segments in our 40 protein test set. To test the structure predictors’ ability to generate multiple conformations, we created synthetic electron density maps with 0.5 occupancy of each of two alternative conformations, A and B, and then used sampleworks to steer ensemble generation towards this density.
Figure 3 from https://thestacks.org/publications/sampleworks-release. cases where guidance is successful (A, PDB 6B8X), and where it is not (B, PDB 5MHX). Panel C shows the improvement in real-space correlation coefficient (RSCC) with guidance against the RSCC of the predictors without guidance, demonstrating the overall success of the method. Figure Credit: Karson Chrispens.

Without guidance, none of the models perform well over the whole test set, as shown in the figure above. With guidance, some structures respond well (we capture both conformations, e.g., 6B8X) but others do not (e.g. 5MHX). Why would guidance work well for one structure, but not another? We hypothesized that the training data contains multiple conformations for some protein sequences but not others. So we can plot the fraction of “B” conformations generated versus the fraction of “B” conformations found among similar sequences in the training data. As you can see below, none of the models can overcome their training data bias.
Four model plots show optimal guidance reduces training bias slopes compared with baseline
Figure 4b from https://thestacks.org/publications/sampleworks-release. Colored lines and points indicate the fraction of “B” conformers when the predictions are guided to a 50/50 mixture of A and B conformations. Points indicate one protein segment, and lines are averages over segments with the same fraction of B conformations in their training data cluster. Grey points and lines indicate model biases without guidance. Perfect guidance would result in a horizontal line at fraction in state B (predicted) = 0.5. Figure Credit: Karson Chrispens.
When we dig into examples like 6B8X, we see many sequence-similar proteins in the PDB in different conformations. In short, AI structure predictors still have a way to go before they will reliably generate ensembles for arbitrary proteins.

Are there better methods? Yes and no.

For the results above, we used “coordinate space guidance” (CSG) for steering, which works by directly manipulating the denoising process in structure prediction. However, others have suggested that other steering techniques may work better. sampleworks makes it easy to implement and test other approaches, independent of the AI structure predictor and experimental data. We have implemented an additional method called inference-time latent-space optimization (ITO). Comparing CSG and ITO, we found that, on average, neither method performs better than the other, but for individual proteins or residue segments, one method or the other can achieve much better fits to the data (see figure below). We have also implemented an alternate sampling strategy called Langevin sampling (LS), which is less efficient than our default sampler (called EDM), but theoretically achieves better results. We observe similar results with LS and EDM samplers. These observations highlight the need for our broad-coverage test set, rather than focusing on a few single examples.
Charts comparing RSCC distributions and coordinate guidance versus latent optimization for Protenix, RF3, and Boltz2
Figure from our upcoming GEMBIO contribution: across 791 small and large backbone shifts, and side-chain only shifts, between alternative conformations, the RSCC is comparable between two steering methods, ITO (aka latent optimization) on the vertical, and coordinate space guidance (aka classifier guidance) on the horizontal. But in individual cases, one method can outperform the other by a huge margin. Figure Credit: Feng Yu.

The future: diffuse X-ray scattering data, training the models directly on experimental data, and alignment.

The Prism project started out as DiffUSE, a reference to diffuse X-ray scattering, an experiment with unique potential to reveal conformational ensembles. The electron density determined by standard MX is an average over many different conformations, making it difficult to see how the pieces of a protein move in concert. Diffuse scattering is sensitive to those correlations, but it has historically been ignored in MX experiments because we didn’t have the tools to measure and model it. Recently, our Prism colleagues have made diffuse scattering measurable and reproducible and developed methods to model some contributions to this scattering. Prism is now working to build and test the “forward model” needed to guide sampleworks ensemble generation with this unique data.
Looking further ahead, it seems that we will need to improve ML structure predictors themselves. The current generation does not represent ensembles well. Predictions don’t match experimental data and, more systemically, they aren't built to numerically encode ensembles in a way that a machine learning algorithm can process. Future models will probably require a combination of “aligning” the internal numerical representations of proteins with other tasks than structure prediction, and training the models to predict the experimental data directly. sampleworks will make this straightforward, as we can already “backpropagate” the errors through the model and integrate multiple types of data. Instead of using those error signals to steer the structure predictors, we can use them to update the predictors' internal “weights” directly.
We will continue to add functionality that should help individual users reveal ensembles from their data more effectively. We anticipate that sampleworks will be production ready for large scale reprocessing of existing experimental data in Spring 2027. We would welcome your feedback and suggestions along the way.

Calling potential collaborators!

sampleworks is flexible enough to allow structural biologists to experiment with structure predictor guidance on their individual systems and bio-ML researchers to develop new features or perform more systematic analysis of their AI models. Our modular architecture means you can easily try many of the latest and best structure predictor methods in one place–with or without guidance. For example, perhaps you want to know if a model like BioEmu2 or Boltz-2 MD can generate novel conformations of a signaling protein you study, or you have electron density maps you think might be better fit with an ensemble.
sampleworks also makes it easy to develop and validate new ideas in biomolecular structure prediction and steering. Maybe you have an idea of how to build a better sampler, or an idea on how to steer the models differently. Rather than build everything yourself into just one structure predictor, sampleworks provides the infrastructure to build your idea out for any structure predictor, and make rigorous comparisons with other methods. And we have built sampleworks to be as flexible as possible in incorporating different types of data, so you can try out your favorite experimental data as well.
We are a small team, and so we have to prioritize our limited time carefully. But there are many experimental methods (HDX, NMR, cryoEM) that we want to incorporate and make widely available to the community. We won’t get to these in the next six months, but we want to start having conversations with the teams who work with this data and might want to make better use of it. sampleworks presents a unique platform to reach end users because we will be able to combine your data with others, and allow you to immediately try a wide range of ML-based approaches to modeling it. The start up cost is almost zero, and we already have benchmarks and evaluation code to start testing things out.
We’d love to hear your feedback and can help you get started using sampleworks to model your data. We invite bug reports and feature requests at the repo: https://github.com/prism-science/sampleworks.
Get in touch!

Credits

While I wrote this post, the work represented here is due to an excellent team. Karson Chrispens (UCSF) started work on sampleworks before I joined Astera and continues to actively develop it with us. Feng Yu and Suyi Chen built our ITO and LS implementations. Doris Mai has been involved in everything and is currently working on reciprocal space and diffuse X-ray scattering guidance. Michael Anzuoni has built all sorts of infrastructure and is currently integrating BioEmu2. Mike Wall is a constant help with all things X-ray. Stephanie Wankowicz has provided us with strong vision and leadership as Prism’s Scientific Program Director.

Share article: