The Twisted Journey of a Computer Vision Project: From False Assumptions to Unconventional Revelations

In the field of computer vision, even the simplest projects can become unexpected odysseys. Case in point: a seemingly straightforward undertaking to build a model capable of identifying physical damage on laptops—a cracked screen, a missing key, a broken hinge—through photographic analysis. The nature of this challenge soon transcended its original simplicity, revealing itself as a complex puzzle demanding radical solutions. The unpredictable adversities experienced underlined the nuanced nature of marrying AI and hardware.

The resolution to the image quality issue led to a focus on mixing image resolutions during model training, based on studies revealing the impact of image resolution on deep learning models. While the consistency improved, the core issues of hallucinations and inadequate image management persisted.

In an experimental shift in strategy, the project incorporated image captioning with the text-only LLM method. This model first generated multiple captions for a single image, which a multimodal embedding model would then verify for image-text fit. The top-scoring captions would then go on to generate new captions in a series of iterative attempts to accurately map the image. Despite its innovative design, this method introduced new issues. The captions sometimes included imaginary damage, some issues were entirely overlooked, and the complexity of the system increased, once again leaving the project at a crossroads.

This project switched its gears to the unlikely tool of agentic frameworks—an example of the necessity for ingenuity when dealing with the paradoxical challenges of creating smart technology. Like any expedition into unknown territory, the difficulties faced by this endeavor underline the importance of resilience, experimentation, and unconventional thinking in shaping the technology of tomorrow.