The rapid emergence of IoT, simply put, is an outcome of the convergence of technologies, devices, and services. From RFID-like identification technologies a decade ago, to networked sensors and actuators today, innovation has been quite fast paced and widespread, but mostly focused on the specific areas of application (or within the domains at best). This has resulted in the proliferation of numerous options at every seam and every interface, without a proliferation of solutions that address data security, system interoperability, or system interconnectivity across domains. While there were reasons and patterns in the past for this diversity of solutions, things are changing for the better, with experts from academia, governments and industries emphasizing and pushing for standards in technology and governance to be defined and enforced.
The emergence of IoT technologies pose several interesting challenges to engineers and businesses alike:
- How to transmit data in a secure fashion which is also power efficient? Does it make sense to collect and store all the data?
- How to handle unreliable networks and breaks in data transmissions? What guarantees can be made about such data?
- How to manage the incredibly high volumes generated at such high frequencies of data collection? How to leverage this to generate maximum business value and at the lowest possible cost and in the shortest time?
Any organization that is looking to be a serious user of IoT technologies must work through many such challenges and put in place capabilities for building efficient and reliable data pipelines that run on distributed and scalable data platforms. Such organizations need to deploy data lakes and processing platforms with advanced analytics capabilities for generating deep business insights, for reacting to events with real-time automated responses. They also need to develop the ability to learn and evolve these practices and systems over time, while guaranteeing the confidentiality, privacy and governance for all data. It is at this intersection of capabilities where IoT and Big Data converge to provide real value for enterprises.
The first step in addressing such challenges faced by IoT users in organizations is standardization. The deployment of standardized approaches and frameworks is possible only to the extent that standards are developed and exist in the field, but the impact of lack of standards can be isolated by making use of useful and relevant abstractions – and that’s where reference architectures come in. By generalizing the domain, a reference architecture offers the most fundamental set of features – concepts, principles and relationships – that can provide a framework for any concrete architecture to be defined for a system in the domain. One such effort for IoT has been the IoT-A reference architecture.
In future posts we will elaborate the IoT-A architecture and provide guidelines to develop a concrete architecture with logical constructs of a system that combines IoT and Big Data technologies together.