Red Hat Research Quarterly

Can streaming data and machine learning build better communities?

Red Hat Research Quarterly

Can streaming data and machine learning build better communities?

about the author

Jim Craig

Jim Craig is a product manager in Red Hat’s global public sector team. He helps define solutions to customer challenges using Red Hat technology and services as well as services from the partner community.

Article featured in

An open source powered smart village project underway at the Red Hat Collaboratory may have the potential to change the world—or at least a town near you.

Joint logo of Red Hat and Boston University

For as long as I can remember—and after almost 40 years in the IT industry, that’s quite a while now—every year for the last 20 years or so has been “the year of the smart city.’’ It’s like when groupware was new and promised the paperless office, prompting the amusing response, “We are more likely to see the paperless toilet before the paperless office.”  

Fortunately, only one of these predictions has recently come to pass, and there are now numerous examples of smart cities, from Malaga to Montevideo. These smart cities are increasingly driven by open source, open standards, open data, and open APIs that, when combined to create a platform, make the design and deployment so much easier and quicker.

That is the foundation of a project now in process, “Creating a global open research platform to better understand social sustainability using data from a real-life smart village.” In December 2021, the project was one of the recipients of the inaugural Red Hat Collaboratory Research Incubation Awards, awarded to projects that take advantage of the collaboration between Boston University and Red Hat Research. But this story begins much earlier.


Early in my tenure with Red Hat, I was introduced to FIWARE, a not-for-profit foundation based in Berlin, Germany, that had developed an open source platform for smart solutions. Originally funded by investment from the European Union, and now through membership fees and engaging in public sector funded projects, FIWARE developed a context broker able to ingest data in many different formats, from many different sources. After normalization, this data is available for developers to build applications to deliver value to citizens, city officials, and their partners and stakeholders.  

With over 400 members at the time of writing, FIWARE has curated a framework of partners able to deliver data across a range of domains, from cities to utilities, manufacturing, and agrifood, with more being added all the time. Bringing this data together can have a powerful impact. For example, a street lamp and a bus have two very different contexts, yet if the street lamp “knows” where the bus is, it can switch on as the bus arrives, allowing passengers to see better as they alight. The lamppost could provide direction information, local news and events, and a WiFi hotspot, remaining illuminated until all the passengers have left its coverage zone, switching off to save energy. Figure 1 indicates the extensibility of the FIWARE architecture.

Figure 1. An example of the possible data collection and analytics sources for a smart village using the FIWARE context broker. Image source: FIWARE

Smarta Byar and Sensative 

During a webinar on smart cities hosted by Red Hat with OpenForum Europe in March 2021, Peter Geršak, State Secretary for Public Administration of the Government of Slovenia, said something that really struck a chord. Mr. Geršak pointed out that Slovenia has only one large city; it is a mostly rural country. He wanted to ensure that rural communities would benefit from these new developments and therefore used the term smart communities, which I like and have since adopted. After all, communities of all sizes can be more sustainable and efficient for their citizens, not just cities.

About a week later, completely by coincidence, Jan Malmgren contacted me with an idea to scale up the work he has been doing in Veberöd, Sweden, called Smarta Byar, which translates to smart villages in English. The village of Veberöd is in the Lund municipality in southern Sweden; it has a population of just over 5,000 people and is expected to almost double in the next few years. Malmgren, a resident of Veberöd since his childhood, decided to use technology to improve the lives of his fellow villagers. 

A digital version of a drone photo of the village of Veberöd
Figure 2. Veberöd high-level digital twin. Image source: Jan Malmgren

Armed with a drone camera, Malmgren took around 50,000 aerial photographs of the village. The local university at Lund used a supercomputer to stitch them together (Figure 2) and create a 3D model of the village (Figure 3).

Zoomed in version of the village digital twin, now in 3D, with text and labels marking 20 different ways that monitoring and AI analytics can support city functions, including traffic, health, water, emergency, doors (locked or unlocked).
Figure 3. Veberöd 3D digital twin. Image source: Jan Malmgren

To take his idea to the next step, Malmgren needed some Internet of Things (IoT) devices, a platform to capture and make sense of the data, and—most importantly—some smart people to make it all work. Enter local FIWARE partner Sensative, based in Lund. Sensative developed Yggio (pronounced ig-yu), a digitalization infrastructure management system (DiMS). Yggio is an open, unifying, and massively scalable IoT platform. 

At the time of writing, the Unity game engine is the platform for the digital twin, which is being replaced by a platform from another Swedish organization, Smart Visualizer. (Smart Visualizer is based on Unity.) 

Early use cases

With its rural location, the Veberöd area is frequented by summertime walkers who also like to barbecue. Using humidity and temperature-monitoring sensors, the Veberöd municipality can determine which barbecue areas are in use. Veberöd also lies in the primary agricultural region of Sweden. To ensure that pumps delivering water to cattle troughs are working, humidity sensors were installed and connected to Sensative’s Yggio platform to detect any lack of water.

In collaboration with Telia, another IoT operator in the region, and through NB-IoT, bikes are tracked in the village. If someone moves a bike without the owner’s phone being geolocated nearby, the system sends an alarm to the owner and the police. The bike is also GPS-tracked so that it can be more easily recovered. School gates are monitored for whether they are open or closed to help with child safety. Finally, water quality and use are monitored in collaboration with the local water utility company VA Syd. 

Scalability through data science

Almost immediately after meeting Jan Malmgren, I shared the project idea with my colleague Alexandra Machado, who leads Red Hat’s Social Innovation Program. The Social Innovation Program works with not-for-profits to help them realize a benefit to their operations using Red Hat open source technology. Past engagements have included the World Health Organization, Greenpeace, UNICEF, and Curriki. Machado suggested I approach the Red Hat Research team and submit a proposal for a Collaboratory Research Incubation Award with Boston University. Adding Boston University (BU) to the ongoing collaboration between Red Hat and Smarta Byar has provided a real injection of energy, intellectual rigor, and a focus on new use cases.

Distinguished Professor of Engineering at BU Christos G. Cassandras, our Principal Investigator (PI), helped with our successful bid for the incubation award. Cassandras is also head of the Division of Systems Engineering and Professor of Electrical and Computer Engineering at the Center for Information and Systems Engineering (CISE) at BU, and he brought with him a talented team of academics and students to work on the next phase of the project, including co-PIs Vasiliki Kalavri, John Liagouris, and Mayank Varia. I asked Cassandras how the project would scale up and out and what part data science plays in scaling. 

Professor Cassandras explained:

“Can we transfer a smart city app in Veberöd to a large city? I believe this is what the Red Hat OpenShift Container Platform can accomplish.”

“There are two aspects of scalability that apply to our project and the question of how we can take what we develop from a small village like Veberöd to large mega-cities worldwide. First, there’s the software platform side of scalability. Can we transfer a smart city app in Veberöd to a large city? I believe this is what the Red Hat OpenShift Container Platform can accomplish. For example, suppose our primary use case at the end of this project were an adaptive traffic light control (TLC) app for the main—and possibly only—major intersection in Veberöd. This would automatically adapt green-red cycle times based on the traffic sensed, including the arrival of pedestrians. 

“Now, suppose we want the same app to scale up to a big city intersection with multiple traffic flows and much higher traffic volumes. If our software platform is properly designed, any tweaking we do in Veberöd (where we can continue to study the app’s operation with the luxury of less traffic) can be readily transferred to the big city traffic light even though its operating conditions are seemingly very different.

“The other aspect is the system side of scalability. This aspect is a bit more subtle, but one I can better explain in terms of data science, and potentially one with broader implications for modern technology. Take the same example of an adaptive TLC app first developed in Veberöd’s lone main intersection. In a large smart city, we first want to scale our ability for adaptive TLC to multiple intersections interconnected in a large network. The ability to do that would allow traffic lights to communicate with each other—and with smart vehicles—and potentially create green waves, or cascading green lights that keep traffic flowing continuously. 

“As a result, vehicles seldom stop, thus drastically reducing congestion, energy consumption, and, in the case of fossil-fueled vehicles, toxic gas emissions. The difficulty with this is the curse of dimensionality, a term coined by Richard Bellman, the inventor of the dynamic programming optimization algorithm. In other words, the complexity of extending what works in one node of a network to multiple nodes increases exponentially with the size of the network. Things just don’t scale using one particular approach, but there is an alternative.

“However, the curse of dimensionality starts with the premise that the algorithms we develop with data science are based on data collected through the traditional time-driven paradigm: With every tick of some underlying clock, the algorithm is updated. This approach is extremely computationally expensive and often unnecessary, since the specific information an algorithm needs to operate may not have changed from one clock tick to the next. Not to mention that if the algorithm operates in an energy-constrained wireless environment, communication with every clock tick wastes battery life (especially crucial for IoT devices!). And, from a security standpoint, every clock tick becomes an opportunity for data to be jammed or stolen.

“Fortunately, we now have the new event-driven paradigm at our disposal: the algorithm is based on events of interest to its specific operation, so it is triggered only when relevant data are detected. In this mode of operation, computations are fewer, communication is drastically reduced, and security is tighter. There are fewer instances of information exchange, and while clock ticks are predictable to a malicious agent, events are generally not predictable. In the adaptive TLC example, the algorithm that adjusts green-red cycles does not need to be updated in a time-driven fashion. The only events that matter are the IoT sensors’ detections of vehicles or pedestrians—and, of course, the light-changing events, which we control directly.

In this mode of operation, computations are fewer, communication is drastically reduced, and security is tighter.

“How does this affect scalability as defined above, extending from one node in a network to multiple potentially interacting nodes? If a data-driven algorithm operates in an event-driven manner, it scales with the size of the events involved, not the size of the network. Going back to the TLC example: in a single intersection, the TLC algorithm is triggered only when one of a small number of possible events occurs. Therefore, as nodes are added, the additional TLC algorithms also depend on a small number of events that occur at the associated intersection. 

“If we want to extend our TLC app from a single intersection in Veberöd to hundreds of intersections in Stockholm or Boston, this effort is scalable—that is, it increases linearly—with the number of events at each intersection. If it scaled with the number of nodes in the network, the effort would increase exponentially.”

At this stage of the project, there are so many interesting and innovative ideas flying around, it’s like being a kid in a candy store. One of these ideas is what Malmgren has termed social sustainability: checking on the health and mental well-being of his fellow villagers using a simple app combined with machine learning and AI models to understand if there is a correlation between water consumption, happiness, and well-being. 

Or could it be speeding cars, light pollution, or air quality impacting the well-being of the villagers? Another exciting example discussed recently is reducing or possibly eliminating cars from the village through the use of on-demand, autonomous electric vehicles.

Whatever use case(s) we arrive at, there is confidence it will have been determined with the highest level of citizen engagement, intellectual rigor, and eye to the future. Malmgren is very clear in his goal: He would like this smart village platform to go global, being adapted, tweaked, and tuned for different environments and use cases. Stay tuned to Red Hat Research to see what happens next!

See Robert Spal’s article “How open data standards make Brno a better city” in this issue for an example of smart community concepts in action.

More like this