Semi-automated training of AI vision models

Authors: Pelletier, Mathew; Wanjura, John; Holt, Gregory

Submitted to: AgriEngineering
Publication Type: Peer Reviewed Journal
Publication Acceptance Date: 6/16/2025
Publication Date: 7/8/2025
Citation: Pelletier, M.G., Wanjura, J.D., Holt, G.A. 2025. Semi-automated training of AI vision models. AgriEngineering. 7:7. https://doi.org/10.3390/agriengineering7070225.
DOI: https://doi.org/10.3390/agriengineering7070225

Interpretive Summary: Artificial Intelligence (AI) is becoming more powerful, especially in understanding and recognizing images. But teaching these AI models to “see” like humans still has a big challenge: they need millions of pictures, each carefully labeled by experts, to learn from. This process is slow, expensive, and hard to get right—especially when the topic is complex and people may disagree on how to label the images. To make this easier, our research found a smarter way. We used two new general-purpose AI models that can look at an image and describe it in words (like writing a caption). Then, we built a simple extra tool that reads those captions and decides which category the image belongs to. This system worked surprisingly well! Even though it was a bit slow—about 10 seconds per image—it didn’t need thousands of labeled images to get started. And once it’s set up, it can label new images on its own. These auto-labeled images can then be used to train faster, more specialized AI models. In short, our method makes it much cheaper and easier to create image datasets for training AI. That means more industries—even those without large image collections—can now start using AI vision tools with much less effort.

Technical Abstract: AI models have recently exploded in utility; however, in the realm of Vision-AI models, they have a significant draw-back. State of the art models require millions of manually annotated images in which to train new models. For similar datasets, this can be somewhat alleviated by using a pre-trained model and performing transfer learning to adapt it to a new image dataset. However, even for transfer-learning, to obtain a robust model, and ensure that it isn’t keying off unappreciated false correlators in these images; transfer-learning training on them still requires 10,000’s of annotated images. And the image annotations must be done manually by a person, or team with tight coordination, with each person having expert knowledge to ensure each image is classed correctly. The more challenging the topic, the more difficult it is to get each team member to classify images in the same manner, which in turn leads to sub-optimal inference performance. To ad-dress this and lesson the adoption hurdle for using AI-Vision in new industry’s that do not have the benefit of millions of already identified images; this research developed a method that utilizes two of the new General-Purpose, GP, Vi-sion-Transformer-to-Caption, ViTC, models. It was found that with a minimal amount of training, limited to development of a post-use semantic-classifier, SC. The SC takes as an input the caption output from the GP models and makes a pre-diction as to the correct class. This approach was found to provide exemplary classification rates in our test cases. While this isn’t a high-speed approach, as the frame rates were slow when running the ViTC models, 10 seconds per image in-ference. This approach almost eliminates the need for extensive training, thereby reducing the cost and labor burden of having to have expert teams of people annotating these huge image datasets. That are required to train, test, and validate new AI-ViT models. With the GP+ViTC+SC approach, we found that only a minimal amount of pre-training is required for the SC, to adopt the model for the specific application. With the GP+ViTC+SC classifier, it can then be used to automatically annotate the huge image dataset that can then be used for training a new high-speed dedicated ViTC model, all nearly completely hands free. This approach is possible, as all of the heavy lifting for classifying the images, for annotation of image dataset, was done in the original development of the GP-ViTC models. Of note is that the creation of the post adoption SC requires far less training, so in combination this approach significantly reduces time, expense and labor in-volved in created new custom AI-Vision models. This report covers this new approach with a full case study that signif-icantly reduces the labor and expense involved in creating new image training sets, that can then be used at minimal cost to develop low-cost image datasets for effective semi-automated training of new AI models.