How on-device intelligence is redefining industry standards and efficiency

In today’s increasingly connected world, Edge AI started as a thought process to address the challenges of data processing and transmission. With the growing relevance of the semiconductor industry, the need for efficient data management has become more prominent. Traditionally, this data would be sent to centralised cloud servers for processing, which is a transactionally heavy on communication channels and not cost effective. Edge AI can mitigate these issues by enabling data processing at or near the source—on the devices themselves. By processing data locally, Edge AI minimises reliance on cloud infrastructure, significantly reducing computing costs and energy consumption, making this technology important for today’s data-centric technological landscape and more environmentally sustainable.

A key component of this mechanism is also collective intelligence, which is facilitated through federated learning and meta-learning. Federated learning allows multiple devices to collaboratively improve a shared AI model without exchanging raw data, thereby preserving privacy while enhancing the model’s accuracy. Meta-learning, on the other hand, can enhance this framework by enabling devices to adapt their learning strategies based on collective experiences, effectively “learning to learn”. This dual-layered approach—immediate learning from edge devices combined with meta-learning through federated collaboration—creates a robust ecosystem that adds value to edge computing as a process. As we move towards ‘Sovereign AI’ where data is preferred to be local and compute engines only exchange meta information essential for learining, Edge AI is now more relevant to bring this transformation.

Challenges in Edge AI

One significant hurdle is the predominance of supervised learning models, which require extensive training before deployment, making immediate inference difficult. While unsupervised learning approaches exist, they often lack the accuracy needed for reliable decision-making.

Secondly, many AI models, including neural networks and deep learning architectures, tend to be very large and computationally intensive, often requiring several megabytes or even gigabytes of memory—far exceeding the capabilities of typical edge devices. Additionally, existing frameworks and tools for AI model generation are often not optimised for embedded platforms, as they rely on tensor data structures and GPU-based processing that are ill-suited for edge computing environments.

In response to these challenges, developments are underway for novel lightweight frameworks that mimic the essential properties of traditional models while employing simpler mathematical constructs to deliver adequate performance with reduced accuracy. As technology has evolved, advancements such as co-processors have enabled more complex processing capabilities at the edge, allowing for the deployment of even small language models (SLM) and other sophisticated applications. However, significant challenges remain, particularly when it comes to running AI models on microcontrollers with limited memory resources. Despite progress in techniques such as vectorisation, quantisation, and advanced hyperparameter search and tuning to optimise model sizes, there is still a need to fully leverage the benefits of Edge AI.

Key Breakthroughs

Since the advent of Edge AI, a significant progress has been the adoption of Apache TVM (Tensor Virtual Machine), an open-source deep learning compiler that facilitates the optimisation and interoperability of AI models across various hardware architectures. By allowing models generated through its framework to run on multiple processor families, TVM enhances flexibility and performance, making it easier for developers to deploy their applications efficiently.

Additionally, the emergence of new chipset companies focused on embedded platforms has been instrumental in addressing the challenges of fitting complex AI models into constrained hardware. Notably, NVIDIA’s tools, such as NVIDIA EGX, TensorRT and the Jetson Nano, Industrial grade IGX Orin enable comprehensive robotic frameworks to operate effectively at the edge, showcasing the potential for running sophisticated applications like motion planning and object recognition without relying heavily on cloud resources.

Furthermore, advancements in model quantisation techniques have paved the way for developing integer models, which optimise performance while reducing resource consumption. There are now more advanced methods like QLoRA, GPTQ which are more popular for quantizing LLMs to make it suitable for running on the edge as the LLM based applications are also finding there way on edge devices. These innovations collectively represent a significant leap forward in Edge AI, enabling more complex processing capabilities directly on edge devices. These factors define the breakthroughs in the edge computing process.

An Industry Perspective

As companies navigate the complexities of implementing data policies, they are focusing on faster pathways from data to insights, which is reshaping their standard operating procedures (SOPs). The rise of AI-centric decision-making processes, including human-in-the-loop systems, is enhancing operational efficiency and productivity, particularly in areas like quality inspections, process control, and automation. Moreover, advancements in neuromorphic computing are enabling ultra-low power processing at the edge, allowing for rapid decision-making in scenarios that require immediate feedback, such as sorting fruits or packaged goods.

As the technology matures and costs decrease, the potential for Edge AI to penetrate sectors like agriculture and fast-moving consumer goods (FMCG) will expand significantly. In agriculture, drones equipped with Edge AI technology can identify ripe fruits in real-time, optimising the harvesting process by ensuring that only the best quality produce is picked. This capability not only boosts productivity but also reduces waste by minimising the chances of overripe fruit being harvested. Similarly, in fast-moving consumer goods (FMCG) environments, such as bottling plants, Edge AI systems can inspect packaging at high speeds—scanning up to 200 bottles per minute—ensuring quality control while maintaining rapid output. This technology can achieve sub-millisecond response times when paired with specialised neuromorphic cameras, although these devices remain costly.

Achieving these efficiencies requires a holistic approach that optimises the entire data collection and processing pipeline. This includes selecting appropriate capture devices, ensuring optimal lighting conditions, and mitigating environmental factors that could degrade data quality, such as dust or corrosive gases in manufacturing settings. Edge AI implementations require consistency across the engineering development cycle, ensuring that investments yield a rapid return while minimising operational disruptions.

To maximise success, companies should target applications where AI can achieve accuracy rates above 95%, ensuring that they address high-impact problems. Identifying these “low-hanging fruits” is crucial for realising quick returns on investment. Implementing Edge AI necessitates a comprehensive understanding of the entire process flow—from data collection to model inference—while also considering environmental factors that could affect sensor performance. As organisations navigate this complex landscape, maintaining an open mindset and exercising patience will be essential for fully harnessing the transformative potential of Edge AI, ultimately leading to improved productivity and operational excellence.

Author: Biswajit Biswas, Chief Data Scientist, Tata Elxsi