Phenotyping faster

Each summer, student researchers tend the CFANS research plots in St. Paul, but keen observers may have noticed something else roaming the fields the past few years: a fully electric rover with a passing resemblance to a Volkswagon van. This rover, controlled by a graduate student with a remote control, was taking photos as it quickly drove through the field; photos which would train an artificial intelligence model to recognize the signs of scab—a fungal disease that infects wheat and barley.

Signs of scab

Scab, the common name of Fusarium head blight (FHB), affects kernels on the heads—or spikes—of the grains. Outbreaks of FHB can be devastating, cutting yield in half and contaminating the remaining crop with a mycotoxin, leaving it harmful if consumed by humans and animals. Plant breeders at the U of M, and across the country, are working hard to mitigate the disease by breeding varieties of wheat and barley that are resistant to FHB, but it is a slow and expensive process.

Brian Steffenson, PhD, is investigating the genetics of FHB resistance, and it takes a trained eye like his to accurately assess disease levels in the field. For these assessments, researchers count the number of kernels infected by the disease in individual spikes to determine the severity of FHB infection, a process called phenotyping. Obtaining robust data for genetic studies “requires the manual counting of infected kernels from thousands of spike samples,” according to Steffenson. And the process must be repeated throughout the growing season. “The onset of disease is really fast, and if you miss this really tight window, you could lose results for the whole summer,” explained Julian Cooper, a PhD candidate working on this problem.

The wheat and barley breeding programs at the University of Minnesota estimate they spend nearly $100,000 each year on field-based FHB phenotyping. “It takes a lot of people to do it, it’s tedious, and it’s just not something people love to do,” says Cory Hirsch, PhD, a plant pathologist who studies plant stress. To accelerate this process, an innovation in phenotyping was necessary. 

Speeding up the process 

Ce Yang, PhD, leads the Agricultural Robotics Lab at the University of Minnesota, which she describes as “a playground for agricultural engineers.” Steffenson reached out to Yang to brainstorm ways that technology could solve the tedious problem of phenotyping FHB by manually counting infected kernels. Creative solutions—including a robotic dog trained to roam the fields and capture photos—were proposed and piloted. Additionally, Yang’s group began developing computer models that could interpret those field photos and identify FHB-infected kernels on individual spikes of wheat and barley. But they still needed a way to speed up the fieldwork.

A breakthrough in the project occurred in 2020, when Yang saw a news story about a rover that was being used to rapidly inspect crops like soybeans and strawberries in the field; Yang saw potential to apply the rover to phenotyping FHB. Yang shared the story with Steffenson, who recalls, “I was so excited to see its extraordinary ability to capture detailed traits of plants growing in the field.”

Mineral, an Alphabet company that applies artificial intelligence to agriculture, was behind the design and development of the rover. Yang and Steffenson reached out to Elliott Grant, who was leading the rover project at Mineral, to discuss collaboration. “Brian and I initiated the conversation [about the Mineral rover], simply because its design and functionality would be the best fit for high-throughput disease phenotyping in barley and wheat,” said Yang. 

Wheels on the ground

The team at Mineral was interested in applying the rover—and their team's hardware-software integration expertise—to the problem of FHB. Funding from the US Wheat and Barley Scab Initiative, a research consortium whose goal is to “develop effective control measures that minimize the threat of FHB,” made it possible for the U to acquire a rover and begin adapting for use with FHB.

Hirsch took the lead on the project as principal investigator, with input from Yang, Steffenson, and other U of M researchers. The collaboration brought together U of M crop researchers and Mineral’s data scientists and engineers with expertise in image analysis and AI model development, all contributing to the common goal of detecting FHB more efficiently.

Mineral’s phenotyping rover is controlled by one person in the field, with a video-game style remote control; the rover is equipped with eight cameras that could be adjusted to the optimal height, angle, and resolution needed to take high resolution images of individual spikes of wheat and barley.

Throughout the past three summers, Hirsch and Cooper deployed the Mineral rover in the research fields of the Minnesota Agricultural Experiment Station in St. Paul and the Northwest Research and Outreach Center in Crookston to take thousands of photographs of wheat. They used those images to train a neural network to identify wheat spikes and quantify the level of FHB on each one. This process of “training the model” is similar to the way a novice is trained by an expert.

Breaking down this process into an algorithm took four separate models, each trained for a different aspect of automatically identifying FHB: detecting a spike of wheat in an image, determining if the image was high enough quality to analyze, segmenting the spike from the background, and identifying which kernels appear to be diseased. Then, researchers could quantify the overall severity of the disease within plots and across plots.

For comparison in this study, they also had five human “raters” performing the same task in the field. The researchers found that using the rover to capture images and then analyzing them with computational models was about five times faster than humans manually identifying FHB. Encouragingly, the computational models were able to perform this task with accuracy that was similar to highly trained humans.

Training a model to do the work of an experienced crop researcher was possible through the collaboration between academia and industry. “The Mineral support group was instrumental in developing models for identifying individual spikes within a row and parsing out the infected portions,” says Steffenson.

An app for scab 

The collaboration between the U of M and Mineral ended in 2023, and the rover drove off into the sunset. The U of M research team is preparing to publish the results of their study, and Mineral is moving on from the rover to other tools that can make the food system more productive and sustainable. “We've learned so much from [Mineral] about the hardware and software and the groundbreaking output that new technology can provide to improve all levels of our research,” said Cooper, who is preparing to defend his PhD.

Although the collaboration with Mineral has ended and the phenotyping rover is no longer roaming the fields on campus, the data collected during this project laid the groundwork for the next generation of rapid phenotyping technology. “The images the rover has collected, and the models that have been developed, will help us transition faster into a more community-accessible platform,” says Hirsch.

The lab is now pivoting the project to use the cameras found in cell phones, hoping to replicate what the rover was able to do in a way that is more portable and readily available. “We’re trying to use what we’ve learned to do the same type of imaging, just in a different manner,” says Hirsch. He and his lab plan to develop a rapid phenotyping platform to share widely with the plant breeding community. “We want to get this in the hands of more people,” says Hirsch.

The breakthrough of rapid phenotyping in the field means that breeders will have one more tool in the fight against scab. “The work being done on the University of Minnesota campus is being done for all of Minnesota—for growers, producers, researchers, and breeders across the state,” said Cooper.