Classification of daily crop phenology in PhenoCams using deep learning and hidden markov models

Authors: Taylor, Shawn; Browning, Dawn

Submitted to: Remote Sensing
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 1/4/2022
Publication Date: 1/9/2022
Citation: Taylor, S.D., Browning, D.M. 2022. Classification of daily crop phenology in PhenoCams using deep learning and hidden markov models. Remote Sensing. 14(2):286. https://doi.org/10.3390/rs14020286.
DOI: https://doi.org/10.3390/rs14020286

Interpretive Summary: Monitoring the progression of crop growth throughout the year is important for understanding the national food supply chain and advancing sustainable agriculture. This monitoring is traditionally done through a combination of in-person surveys and analyzing satellite imagery. There is an opportunity for more localized and automated monitoring via near surface cameras positioned at agricultural fields. These cameras are attached to a tower and take images throughout the day of the same section of field, and then upload the images to a central server. It’s difficult to turn daily images into usable information of crop phenology since most cameras are not scientific instruments, and experience a variety of weather conditions like cloudy or snowy days. In this study we developed a deep learning based workflow to process daily images from near surface cameras into a usable dataset of field status and crop phenology. The resulting data can be used to estimate current crop status at numerous locations across the United States, and can also be used as a reference to refine satellite based estimates of crop phenology.

Technical Abstract: Near surface cameras, such as those in the PhenoCam network, are a common source of ground truth data in modelling and remote sensing studies. Despite having locations across numerous agricultural sites, few studies have used near surface cameras to track the unique phenology of croplands. Due to management, crops do not have a natural vegetation cycle which many phenological extraction methods are based on. For example, a field may experience abrupt changes due to harvesting and tillage throughout the year. A single camera can also record several different plants due to crop rotations, fallow fields, and cover crops. Current methods to estimate phenology metrics from image time series compress all image information into a relative greenness metric, which discards a large amount of contextual information. This can include the type of crop present, whether snow or water is present on the field, the crop phenology, or whether a field lacking green plants consists of bare soil, fully senesced plants, or plant residue.  Here we developed a modelling workflow to create a daily time series of crop type and phenology.  while also accounting for other factors such as obstructed images and snow covered fields. We used a mainstream deep learning image classification model, VGG16. Deep learning classification models do not have a temporal component, so to account for temporal autocorrelation of the images our workflow incorporates a hidden markov model in the post-processing. The initial image classification model had out of sample F1 scores of 0.83-0.85, which improved to 0.86-0.91 after all post-processing steps. The resulting time series show the progression of crops from emergence to harvest, and can serve as a daily, local scale dataset of field states and phenological stages for agricultural research.