Abstract—Aside from putting entirely new devices and system concepts into the market and considering “things” is an entirely ambiguous and unrestricting word, the Internet of Things requires making normally interoperable, integrated and tightly associated pieces work together in disintegrated, loosely associated couplings of capability. This is fine when the coupling is between well-known entities having loose timing constraints. The problem is entirely different for real-time embedded systems necessary for Industry 4.0. This paper reviews problems coupling real-time embedded elements together and extends these issues to coupling them over the Internet. Schemes such as MQTT, Python’s Twisted engine, Jini (now Apache River), Node.js, and PBOs are cataloged and pros and cons are exposed with the goal of motivating the development of an open standard and governing organization.
Introduction
If the Internet has enabled devices to easily communicate using a well-established communication standard, and if that standard is the harbinger of Industry 4.0 and the Internet of Things (IoT), why can’t we tie one IoT device to another without routing through someone’s proprietary cloud solution? Whereas we can connect any Bluetooth-based mouse to any laptop, I want to connect any well-characterized IoT device to any smartphone without using a proprietary solution. Both ends of the link should know how to talk with each other even when the devices are on different subnets on, or off, the same physical network and regardless of the physical media used – total interoperability. I’m reminded of universal remote controls. The idea is that you can replace all the remotes that came with your TV, stereo, home theater, Blu-ray player, etc. with one. The trouble is the functionality is never quite right. The truth is that the only thing universal is that all of them use infrared LEDs. Those remotes store everyone’s unique codes. It’s feasible all devices could use the same codes. Do we really need as many “increase volume” or “go up one channel” codes as there are television manufacturers?
This article was motivated by the lack of an open and broadly accepted standard and specification for how physical end devices on TCP/IP-based networks could interact in order to facilitate some means of assembly, possibly self-assembly, into more complex real-time systems. The goal is to start the development of, and highlight the opportunity for, a non-proprietary, open standard such that anyone, or any group, can implement services that allow devices from multiple manufacturers to interoperate without needing a “closed” piece.
Published just in time for this article, reference1 argues the opportunity is big for companies serving the Industrial Internet of Things (IIoT). The author of that paper claims they can sell more services and products, in part, by collaborating with other firms “to make customer data that they store and share on their devices more valuable. With the same products, they could offer customers new services and therefore broaden their customer bases.” Reference2 estimates market revenue “from integrating, storing, analyzing, and presenting Internet of Things (IoT) data” at $5.7B USD in 2015! That sounds like invasion of privacy and a marketing bonanza to me. In other words, it’s good business to keep things closed and the innuendo is that such device-to-device coupling needs to be handled by them – not you. This will happen if we do nothing. This scenario is the antithesis of my article. It is fine for such services to exist, but they shouldn’t be the only option. Apple and Weaved may offer solutions to those who want them to manage the details, and there are people who need this, but if you’re more of a Linux person, you may wish there was a common open standard available.
This article surveys samplings of research and frameworks on various technologies and techniques from a range of sources. I’ve picked technologies that intrigue me when reflecting on the goal. There’s far more to consider than I’ve outlined here. This is simply a start that others can build on. My article continues by explaining what I think is needed for communication and security. Then I proceed to what’s needed for real-time systems before I review some specific implementations that are close but do not have all the features we need.
Communication needs
Connectivity, device discovery and network awareness
The problems of interconnectivity are not new3. The Open Systems Interconnection model (OSI) was an effort to address the connectivity problem in the early 1980s; and it has served us well, but device discovery and network awareness problems remain unsettled.
The Association for Computing Machinery (ACM) has undertaken considerable research on this subject. The applications it considers are very broad, including everything from vehicular internetworking to stationary, network-connected storage devices4. The 2012 ACM conference held in Helsinki had presentations on fog networking5 and Princeton University is actively involved in its development. The name fog was chosen to connote the concept of a cloud at ground level – enabling much more extensive, lower level connectivity. This research, and that of software-defined networking (SDN)6, may well produce the framework we will eventually use but, while theoretical underpinnings are being developed, proprietary solutions continue to be introduced to the market. What’s available for the pragmatist?
The universal serial bus (USB) serves as inspiration to address discovery and awareness. It does this, in part, by virtue of well-specified “classes” of devices7. This allows nearly any device to be easily attached to nearly any host. Once a device is defined, its basic functionality can be developed for any platform’s exploitation.
A USB host takes the role of device coordinator. The IIoT services implied in1 are cloud services that essentially act like hosts, but requiring a host is problematic. It forced the USB Implementers Forum, Inc. to eventually introduce USB On-The-Go to allow USB devices “to communicate both with USB peripherals and directly with each other when a PC is not available”8. We need the same thing for the IoT: linkage that is device-to-device directly or device-to-device through a host or both.
Security
Issues and solutions to IoT security are easy to overlook. Advanced encryption standard (AES) 128 is the most popular solution used. It’s reasonable to believe that any serious deployment of IIoT will require encryption. Imagine a factory of IoT-coupled devices being inadvertently or maliciously attacked by a man-in-the-middle. Since most IoT devices will make their connections wirelessly (e.g., Bluetooth, Wi-Fi, cellular MODEM, LoRa), encryption and authentication must be part of a new standard. Reference 9 exposes some of the issues.
Real-time needs
Stewart described the needs of embedded real-time systems10. Industry 4.0 and the IIoT require some amount of real-time performance if they are going to control our machines and our factories.
Stewart, et al., created a framework to support component-based software in an abstraction called the Port Based Object (PBO)11. Their aim was to create reusable real-time software components. IoT devices can also be thought of as reusable software components. The reasoning goes if we adopt Stewart’s interface concepts to IoT, we’d have a framework out of which a Cyber Physical System (CPS) could be created. Because the PBO concept is mature, a lot can be learned by studying its adoption – and reasons for and against it. At the core of this concept is the need to address real-time performance. The following are the conditions that Stewart’s framework addresses.
Independent processes
Stewart argues in10 that embedded real-time systems contain independent processes that have a need to share relatively small amounts of data at relatively high frequencies. These frequencies are certainly higher than the Internet can handle. Even though today’s Internet bit rates are in hundreds of megabits per second, payload size and the requirement for handshake makes Internet-based communication impractical in most contemporary, embedded, real-time systems. Communication density will be compromised for real-time IoT devices. Perhaps communication payloads from multiple processes can be bundled to increase payload efficiency and minimize the number of transmission instances.
The processing burden of TCP/IP communications generally forces an application, that otherwise requires only an 8-bit host processor, into a 32-bit solution. Thankfully, a 32-bit device costs less than $0.50 USD in moderate quantity, so I expect IoT devices to contain 32-bit cores.
Bindings between Independent Processes
These independent processes must be bound together in order to determine output values from various inputs, and some outputs might be needed to process yet other outputs. Stewart argues these bindings are advantageously made to be dynamically reconfigurable and coupled together within specific time limits. For example, a device might come online, or be discovered, that is able to reduce the communication burden of an upstream device and thereby dynamically disconnect a downstream sink and reconnect it to a new upstream source. It would be beneficial to address this in our standard. Devices should provide the capability to parse, buffer, and distribute/redistribute data. This means we likely have to specify distributed functions and roles in addition to defining distributed device types. These functions and roles might be considered “virtual” or “abstract” IoT devices.
Initial USB-based development tools had a problem when the host was busy. They had no way to force the host to pay attention, with a result that they were intermittent and “buggy.” Only when those tools became smarter (e.g., more memory, less dependence on a remote CPU for timeliness of response) was this problem mitigated. IoT devices will have to be smart.
Multiple processing units
Using one or more central processing units (CPUs) to achieve performance goals is often used in real-time embedded systems. This can take the form of multiple microprocessors, programming logic chips, or both. There should be a many-to-many association of CPUs to IoT devices.
The coordination between multiple CPUs should allow each CPU to operate independently of other CPUs. Like processing threads in the CPUs of modern PCs, each available CPU could possibly help reduce communication bandwidth by condensing/extracting information from raw data, or they might provide some level of fault tolerance. Whatever the application, the system should be tolerant of different operating frequencies and different communication rates as Stewart cites so that multiple CPUs (nodes) are supported.
Sampling of available frameworks
I now catalog some frameworks that have received some notoriety and have some features applicable to our goal. I’ll explain why they were invented and comment on their fit. The survey consists of Jini, Node.js, Twisted, and ends with an exposé of MQTT which, by my reckoning, is becoming the defacto standard for the closed implementations.
Jini (now Apache River)12 13
Venerable Jini was developed by Sun in the late 1990s and is now the Apache River project. It’s based on Java. It was developed to allow modular services to connect themselves together over a network and thereby establish a so-called service-object-oriented architecture. An interesting feature of Jini was its ability for network entities (e.g., services) to discover themselves. This is done through a centralized broker (a lookup service) that all clients contact in order to learn about the others. Once discovered, the services can interact with each other. It’s reported the lookup service does not scale well to very large systems, and considering the IoT, we’re talking about potentially huge systems.
There were three general goals of Jini: (1) A client should not need to know where a particular service is located, (2) The client should not fail if the service is unavailable or becomes unavailable during execution, and (3) Clients and services should be proactive in detecting failure conditions. These are worthy goals for our standard.
Since Jini is based on Java, it has a certain heritage with embedded devices, as Java was originally envisioned as a platform designed for small, embedded systems in electronic appliances like set-top boxes. So far, so good, but Java is the language of IT departments. It is not a platform for small embedded systems today unless those systems are single board computers (SBCs) like the BeagleBone Black and Raspberry Pi. Nonetheless, object awareness and discovery are nice features to be in our specification, and the concept of a centralized broker, as an optional component, is a handy thing. As I mentioned, the closed systems that exist today are essentially centralized brokers. I see value in this but only as an option.
Node.js14
Node.js was created in 2009 to allow web servers to reach out to clients on their own accord. Not bad; our embedded IoT devices can be thought of as little web servers, so this might fit our scenario.
The node.js website says, “Node.js® uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices.” This sounds like the perfect platform. However, node.js is built on Chrome’s JavaScript runtime engine, and “real-time” in this context is web session-based – far different from real-time in our context of embedded. Think of the data rate required for an IoT device that tallies cycles as a packaging machine completes them. These machines package goods at the rate of thousands per minute. Node.js was not made to support the level of real-time performance that many IoT devices will need.
Further, if used as is, the IoT device would require at least a JavaScript engine. Whereas JavaScript engines are normally built into web servers like Apache, there are other choices such as V815 and TraceMonkey16, but none of them are known for speed. This means node.js is likely too bloated and too high level for use in IoT devices. This platform might be suitable on SBCs or, better yet, whole PCs. As is, node.js doesn’t appeal to me as the platform for our standard.
Twisted13
Twisted is written in Python and was made for developing internet applications, specifically games. It’s an event-driven, network programming framework. Developers write callback routines that are called by the framework. It was created to allow game developers to embed Internet-based communications into their games. TCP and UDP services and a Unix socket model of interfacing are core to this framework.
An advantage to Twisted, since it’s based in Python, is that its underlying engine, the Python engine, is written in C, among other languages. This is a major advantage in getting it to work on an embedded device, since nearly all embedded devices are developed in C. CPython17 is the mainline Python engine, but engines such as PyMite18 exist that run on 8-bit devices with as little as 64 KB nonvolatile memory and 4 KB of RAM. This isn’t to say an 8-bit device should be the target of an IoT device. It’s mentioned only to point out how tight a Python engine can be. Due to the burdens of TCP/IP as mentioned, IoT devices are expected to embed 32-bit cores. Twisted is looking like a worthy framework for our standard.
Python can also be embedded within products to provide end-users a built-in programming interface19. Twisted is looking even better. Programmable IoT devices might now be programmable in Python by end users. Nice!
The callback routines required by Twisted are called “deferreds.” These are asynchronous to all other processing. This requires a different way of thinking for it to work for real-time processing. Whereas high priority tasks might normally be placed in an interrupt service routine, asynchronous call-backs require thinking of some alternative approaches to interrupt routines such as semaphore-based gating. The end result might be only near-real time performance – a condition where occasional misses of real time events are tolerated.
Twisted is made to be implemented at either or both ends of a client-server connection. Since the mechanism can send any information using the TCP/IP suite of services, it remains a good candidate.
However, Twisted does not include USB style class-type definitions. Whereas Twisted is easy to use for any ad hoc application or product that has a TCP/IP-based interface, it requires custom code for a device to tie into an application. Twisted needs a device-type specification like USB On-The-Go, so that well known devices can be folded into the app without new development. This must exist at either end, or both ends, of a link.
OASIS Message Queue Telemetry Transport (MQTT)20
MQTT is a light-weight, publish/subscribe messaging protocol. It was originally created by IBM and Arcom but became an open standard of OASIS in 2014. The mission of OASIS is to drive open standards development, convergence and adoption for global information communication.
MQTT is described as extremely simple and lightweight and is designed for constrained devices and low-bandwidth, high-latency or unreliable networks. Its aim is to minimize network bandwidth and device resource requirements without compromising reliability and to assure message delivery. It’s been touted as ideal for the IoT, and it is popular.
As a protocol, it has to be implemented to become a complete framework. The Eclipse Paho project is such an implementation. It supports MQTT in a variety of languages including C/C++, Java, JavaScript, and Python21. Most implementations are for the client side, but server side implementations are also available.
The server is called a “message broker” in MQTT. Unlike Twisted, whereby end-devices can talk directly to each other, MQTT generally involves this middleman message broker.
Mosquitto is a Python implementation of an MQTT broker, and Apache ActiveMQ is another. ActiveMQ is in Java with bindings to many other languages. There are still other message brokers, like RabbitMQ and ZeroMQ. Many other brokers exist to support various other IT-originated protocols.
As mentioned earlier, many proprietary IoT solutions are picking MQTT for messaging. The issue is the lack of standard for what data to communicate from what IoT devices. The MQTT implementations that exist are compatible only with themselves. The protocol allows messages to be communicated, but only those IoT devices that were created for those messages know what they mean. The messages are gibberish to any other device that might be able to use the information. The same is true of Twisted. It would be perfectly fine to use Twisted with MQTT as a protocol, but there needs to be a message content standard for each type of IoT function.
Other research and protocols
There is truly a maze of choices for protocols and frameworks to consider; most are academic. The Center for Intelligent Maintenance Systems is very active in the area of Cyber Physical Systems (CPS)22. Dr. Jay Lee is the Director of the center and leads the development of a concept called the 5C architecture23 24 25. 5C stands for connection, conversion, cyber, cognition, and configuration. This is a general framework that could have application to our effort; as there are systems currently deployed that use this scheme.
Still other protocols are in reference26. It mentions Advanced Message Queuing Protocol (AMQP) and Simple/Streaming Text Oriented Messaging Protocol (STOMP), and there are popular data formats like XML and JSON. There is a lot of material that can be considered, but at the same time, there’s enough to get started on a standard and a development framework today.
Examples of proprietary Solutions
Wunder Bar is an IoT starter kit that was sponsored by Conrad, created by relayr and designed and produced by Mikroelektronika27. It consists of IoT modules for sensing light, temperature, acceleration, infrared, sound, humidity, and more. These modules, in turn, communicate with a main module over a Bluetooth connection. The main module has no sensors. Its job is to communicate with the relayr-provided cloud service through an MQTT-based communication scheme. From the cloud service, users can access the data from those modules on their Android- or iOS-based smartphones or PCs. It’s a clever package and well thought out. Without providing the hardware, Weaved offers equivalent cloud services, but they are not compatible with relayr. Even if they use MQTT, they can’t be interchanged.
The relayr cloud software is closed source, but the main module’s code should be available as open source by Embedded World 2015. When this happens, users should be able to create their own cloud solution. Regardless, the issue remains that there isn’t an open platform ready for use. Any custom solution will have to rewrite existing functionality to accommodate a custom or an open cloud.
Another solution coming soon is from Adafruit28. The introduction of the blog post announcing it says, “Here at Adafruit, we sell all of these amazing components, but we couldn’t find a good way to interact with them over the internet. There are certainly a lot of great services out there for datalogging, or communicating with your microcontroller over the web, but these services are either too complicated to get started, or they aren’t particularly fun to use.” I couldn’t agree more.
Consuming and displaying real-time data with an easy-to-manipulate dashboard is the point of Adafruit’s offering. They implement 2-way interaction via a RESTful API, and some level of MQTT is supported. The framework is built with node.js (already mentioned), Ruby on Rails, Postgresql, Redis, and Memcached.
Here, in Adafruit, is a company committed to open source. Yet, they are having to roll their own solution, because one is not already available that is open.
Organizations working on open solutions for the IoT
The topic of IoT is definitely hot. It’s interesting to watch the keynote address from the 2015 Consumer Electronics Show that Adafruit has archived for us29. Samsung promises to be open – sounds good.
I mentioned that reference1 was published just in time for this article. As I was finishing this article, another article was just published. This newest article mentions companies and organizations forming alliances to work on the problem of establishing standards and collaborative opportunities30.
One of them is the “Internet of Things Consortium” (IoTC). The concern I mentioned, that the IoT is becoming a marketing bonanza, appears to be well-founded. Their homepage says “IoTC Research Lab conducts research projects focused on understanding consumer purchase intent, usage patterns and awareness of IoT products.” The organizations listed on their website have deep pockets but I don’t think this will play for users of Industry 4.0 and IIoT devices. The IoTC is a consumer-focused entity.
Another organization cited is the Allseen Alliance31. Even though its website looks like it is a marketing firm, it has serious technical content for developers. Their “AllJoyn” code framework is open-source but only supports C, C++, Objective-C, and Java languages.
These entities motivate me to predict there will no single standard for the IoT. I doubt there will be one for the IIoT and Industry 4.0. I leave the debate to the intellectuals as to whether or not this should be the case.
Conclusion
There are a lot of technologies, but no one has threaded together an open set that addresses our needs. The solutions that exist in the market are limited, closed, or established for the exploitation of special interest groups. Furthermore, like ham radio enthusiasts, this is the group capable of performing a lot of experimentation that is needed for a grass roots standard to take hold. I like the idea of there being a competition of ideas, but I want them to be open and not big enterprise sponsored.
As unrealistic as this may be, I want an open standard that allows me to:
- Connect IoT devices via TCP/IP-based protocols, Wi-Fi, Ethernet, Bluetooth, USB, etc. with and without a central server
- Reuse or incorporate software with minimal development or refactoring
- Incorporate devices that are developed and deployed by different parties through self-aware or auto-discovered capability and interfaces
- Program in C, or C++, and Python
- Implement my own cloud service
I think this can be provided with:
- Python and C/C++
- MQTT
- Modification of Twisted to serve as this standard’s code framework
- Incorporation of USB-style defined-classes
- An independent organization that maintains the:
- Standard
- Framework documentation and code as downloadable modules so that well-characterized devices require no new code development
- Acceptance and registration of IoT defined classes
- Role as the message broker to facilitate device discovery
Acknowledgment
I thank Travis Dazell of Digi-Key for calling my attention to Fog Networking.
References
- B. Fischer, "How High Tech Can Pursue Industrial Internet of Things Market," 30 December 2014. Online. Accessed 6 January 2015.
- Business Wire, "Market for IoT Analytics to Reach US$5.7 Billion in 2015, with Startups Driving the Innovation, According to ABI Resear," Business Wire, 13 January 2015. Online. Accessed 15 January 2015.
- S. P. Kumar and C.-Y. Chong, "Sensor Networks: Evolution, Opportunities, and Challenges," PROCEEDINGS OF THE IEEE, vol. 91, no. 8, pp. 1247-1256, 2003.
- F. Bonomi, "Connected vehicles, the internet of things, and fog computing," in VANET 2011, Las Vegas, USA, 2011.
- F. Bonomi, R. Milito, J. Zhu and S. Addepalli, "Fog Computing and Its Role in the Internet of Things," in Proceedings of the first edition of the MCC workshop on Mobile cloud computing, Helsinki, 2012.
- D. Kreutz, F. Ramos, P. Verissimo, C. Rothenberg, S. Azodolmolky and S. Uhlig, "Software-Defined Networking: A Comprehinsive Survey," Proceedings of the IEEE, vol. 103, no. 1, pp. 14-76, 2015.
- USB.org, "USB Class Codes," Online. Accessed 6 January 2015.
- USB.org, "USB On-The-Go and Embedded Host," Online. Accessed 10 January 2015.
- B. Boldt, "Is the Internet of Things just a toy?," 2 January 2015. Online. Accessed 6 January 2015.
- D. B. Stewart, "Software components for real time," Embedded Systems Programming, pp. 100-138, December 2000.
- D. B. Stewart, R. A. Volpe and P. K. Khosla, "Design of dynamically reconfigurable real-time software using port-based objects," IEEE Transactions on Software Engineering, vol. 23, no. 12, pp. 759-776, 1997.
- S. Oaks and H. Wong, JINI in a Nutshell, Sebastopol: O'Reilly, 2000.
- Twisted Matrix, "Twisted Matrix Labs," Online. Accessed 8 January 2015.
- node.js, "node.js," Online. Accessed 8 January 2015.
- Wikipedia, "V8 (JavaScript engine)," Online. Accessed 9 January 2015.
- Wikipedia, "SpiderMonkey (software)," Online. Accessed 9 1 2015.
- python.org, "EmbeddedPython," Online. Accessed 9 January 2015.
- python.org, "PyMite," Online. Accessed 9 January 2015.
- Python community, "5. Embedding Python in Another Application," Online. Accessed 8 January 2015.
- MQTT community, "MQTT," Online. Accessed 8 January 2015.
- eclipse.org, "paho," eclipse.org, Online. Accessed 12 January 2015.
- "Center for Intelligent Maintenance Systems," Online. Accessed 13 January 2015.
- J. Lee, B. Bagheri and H.-A. Kao, "A Cyber-Physical Systems architecture for Industry 4.0-based manufacturing systems," Manufacturing Letters, pp. 18-23, January 2015.
- Center for Intelligent Maintenance Systems, "IMS_CPS," Center for Intelligent Maintenance Systems, Online. Accessed 13 January 2015.
- "Cyber-physical system," wikipedia.org, Online. Accessed 13 January 2015.
- vFabric Team, "Choosing Your Messaging Protocol: AMQP, MQTT, or STOMP," VMware, 19 February 2013. Online. Accessed 12 January 2015.
- Mikroelektronika, "Wunder Bar," Mikroelektronika, Online. Accessed 13 January 2015.
- adadfruit.com, "Coming Soon: Adafruit IO," adadfruit.com, Online. Accessed 13 January 2015.
- adafruit, "Samsung promises to be open for IoT components and devices @Samsungtweets #samsung #Iot #CES #ces2015," adafruit.com, 6 January 2015. Online. Accessed 16 January 2015.
- B. Cha, "A Beginner’s Guide to Understanding the Internet of Things," recode.net, 15 January 2015. Online. Accessed 16 January 2015.
- "Allseen Alliance," Online. Accessed 18 January 2015.
