OpenAI’s BioBasicsBench exposes gaps in AI lab judgment

OpenAI has introduced BioBasicsBench, a benchmark developed with enzyme-design biotech company Ligo to test how AI agents interpret raw laboratory evidence. In results published November 6, GPT-6 Astra earned 66.7% of available credit at the xhigh reasoning setting, compared with 43.8% for GPT-5.6 Sol. OpenAI also documented failures that show why a higher score is not yet reliable laboratory judgment.

The benchmark contains 100 tasks across five areas: protein methods, imaging and cytometry, quantification and assays, nucleic acid methods, and molecular cloning. Scientists built the questions from their own experiments, using gel and blot images, colony plates, microscopy, sequencing files and instrument exports rather than only charts selected for publication. An agent may need to identify a mistake or recognize that the evidence cannot support a conclusion, not simply name what appears in an image.

Each task pairs its evidence with an expert-written grading rubric. OpenAI says the review process included a Ligo scientist who did not write the task, an external PhD-level wet-lab scientist and a final internal review. Answers receive credit for individual criteria, including conclusions, measurements, reasoning or recognition of insufficient evidence. Mean reward is a partial-credit score, not the percentage of experiments completed successfully or the share of tasks answered perfectly.

OpenAI evaluated seven models across 37 model-and-reasoning settings, with five attempts on each task at each setting, producing 18,500 results. That still means 100 distinct tasks, not 18,500 separate experiments. Its 95% confidence intervals resample whole tasks while keeping the five attempts on each task together.

At xhigh, the highest effort setting reported for all seven models, Astra’s 66.7% mean had a 95% confidence interval of 60.3%–73.1%. GPT-6.1 Sol scored 62.0%, GPT-5.6 Sol 43.8%, GPT-6 Luna 38.3%, GPT-5.5 35.4%, GPT-5.4 33.2% and GPT-5.2 31.2%. These are OpenAI’s November 6 measurements under that setting, not an independent ranking of every product carrying those model names. Our GPT-6 family explainer covers the distinction between Sol, Luna and Astra. How those models sit next to Claude and Gemini is covered in our AI models explained guide.

Astra generated an average of 4,704 output tokens per task, against 9,099 for GPT-5.6 Sol, in the same xhigh comparison. That describes output length, not total compute, elapsed time or the cost of running a lab workflow. OpenAI’s examples show Astra spotting an incorrectly labelled antibiotic control and recognizing a microscope depth scan that the older model interpreted as a biological change.

The failures are more consequential than a leaderboard alone suggests. In one colony-PCR example, Astra distinguished two groups of bands but let an assumption about the larger product override the controls, selecting the wrong colonies. In another, it discounted lighter vacuoles that contradicted its preferred claim that these cell compartments were generally darker. It noticed variation in focus but did not carry the contradictory observations through to its conclusion.

In a separate colony-counting example, Astra missed visible colonies, failed to distinguish close neighbours and marked empty agar. The net result was an approximately 12% overcount on that plate, relative to the benchmark reference. That is one illustrated error, not an overall error rate for the benchmark. OpenAI notes that such an overcount would inflate an estimate of DNA-introduction efficiency by the same proportion if plated volume and dilution were unchanged.

Kingy AI’s separate coverage likewise distinguishes partial-credit scores from dependable decisions and explicitly says it did not reproduce the results. External review of benchmark tasks is not an independent rerun of the model evaluation. OpenAI itself cautions that higher scores do not yet mean reliable lab judgment: the reported gains coexist with mistakes in controls, contradictory evidence and routine measurement.