Quantifying the relationship between a biomolecule’s genetic sequence and its biological function is a fundamental problem that addresses how complex molecules can evolve. Recent technologies have allowed the collection of large numbers of sequence-function pairs. However, exploiting this data for biophysical insight is challenging, as the data is a sparse sampling of the combinatorially large space of possible sequences. With statistical models inferred from these datasets, I show how a small number of intermediate molecular phenotypes can explain many aspects of sequence-function relationship. From a library of mutated promoter sequences and expression measurements in e.coli lac operon I infer detailed physical interactions between two regulatory proteins. Using a different heuristic approach, I infer fold stability from an thousands of mutated variants of green fluorescent protein, and show how enzymatic activity can be quantified in beta-lactamase. Finally, with a thermodynamic model I infer a detailed energy landscape of a small bacterial protein and separate the effects of mutations on binding and folding stability from a high quality experiment with 500k variants.