Building better communities with streaming data and machine learning using FIWARE

Jan 9, 2023 | Blog, North America

The latest standards powering our open source smart village project underway at the Red Hat Collaboratory

By Chris Tate, Principal Consultant, Red Hat

Do you believe that software can help communities and build a better world, but find it challenging to get started? I don’t like to wait for the right software to come along—instead, I find folks to work with to build it. Early last year, I joined a team of researchers from Boston University, Red Hat’s Global Social Innovation Program, and Smarta Byar (an organization specializing in Smart Villages) to create a global open research platform where researchers can collaborate to explore social sustainability. 

Using real data from the small town of Veberöd, Sweden, the project “Creating a global open research platform to better understand social sustainability using data from a real-life smart village” has achieved visible success toward the goal of building an open source technological infrastructure. Researchers can collaborate on this platform effectively to study mobility, public health, sustainability, and more to better understand connections between well-being and eco-smart cities. The project was one of the recipients of the Red Hat Collaboratory at Boston University’s inaugural Red Hat Collaboratory Research Incubation Awards, a program to provide funding and support for projects that leverage collaboration opportunities between Boston University and Red Hat. Red Hat Research leads the Collaboratory partnership within Red Hat, from cultivating collaboration opportunities to overseeing program administration.

Smarta Byar is continuously deploying new smart sensors in Veberöd, and Jan Malmgren, the CEO of Smarta Byar, helped us integrate with IoT nodes in the village. New integration with FIWARE technology allowed us to implement their Smart Data Models, aligning us with the rest of the open source smart solution community. The online platform allows us to share our results and has accomplished significant goals for the project. According to Christos Cassandras, the Distinguished Professor of Engineering at Boston University, “Not only were the goals of Phase 1 met, but they were exceeded by a large margin.”

Since the project started a year ago, research, development, deployment, and standardization in this partnership have moved the project forward in support of its goals to:

  • Design and expand the current technological infrastructure
  • Design and create the data visualization strategy and front-end application
  • Expand on the strategy of an open, scalable, secure, and reliable platform for researchers

Caption: Smart Cities project team members from Boston University and Red Hat met with FIWARE at Boston University to plan the project’s next steps: building the actual Smart Traffic Light, integrating with FIWARE, and bringing the solution to the edge. 

Designing the Smart Traffic Light algorithms at Boston University

Since the project launched, graduate students from Boston University have developed simulations of a traffic stop in our test village of Veberöd. I built a scalable online platform to import the students’ simulations and translate the data into maps and dashboards. Students have done an incredible job simulating traffic at the first potential intersection in Veberöd that requires a traffic light. 

They used an open source application called SUMO to pick a location on a map and loaded all the road data into a directory of files for the simulation. The road data did not have all the details needed to simulate pedestrian traffic, so they used the SUMO tools to add missing sidewalks. The default traffic light algorithm proved inefficient, so they put their advanced scientific and technical backgrounds to work to develop a new traffic pattern that optimized the traffic flow in all directions for both vehicles and people traffic. This video of the new, efficient simulation shows the movement of cars and pedestrians regulated by a Smart Traffic Light. Orange dots represent cars, pedestrians are purple dots, and the traffic light changes between green and red over time. 

Building a scalable application on Red Hat OpenShift

To share the project online and provide role-based access control to the traffic and IoT data, I developed the software we architected at the beginning of the project. Our new OpenAPIs for smart cities interact with external services to gather live data, interact with open source applications to simulate traffic, and help university students study smart city indicators. 

The site, which you can find running live in production at smartvillage.computate.org, is an innovative project running in the cloud with enterprise standards. Because it runs on Red Hat OpenShift, it features the best in security, high availability, scalability, updates, storage, and more. The site is deployed on the Operate First environment. Operate First is an open source community cloud where maintainers use environments to develop, deploy, and operate their services in an open fashion on Red Hat OpenShift. 

As a new open source project growing in maturity, we are integrating our project with the Open Source Software Foundation standards for obtaining a Best Practices Badge. We are building a community around smart cities that can thrive by observing all the standards in our project around documentation, change control, reporting, quality, security, and analysis. I’m pleased to announce that our latest stable version 1.0.1 of the smartabyar-smartvillage project is now available in the following community locations: 

Building Smart Cities technology together with FIWARE

In September 2022, the Smart Cities project team members from Boston University and Red Hat met with Juanjo Hierro, the CTO of FIWARE, at Boston University to discuss the project’s next steps. Since then, we have been building compatibility into our Smarta Byar Smart Village Project with FIWARE and the FIWARE community. 

Caption: Chris Tate presenting our collaboration and steps to become a FIWARE Innovation Hub for Smart Villages at the 2022 FIWARE Summit in Gran Canaria, Spain. 

Starting with the FIWARE Generic Enabler General Requirements, we have successfully integrated the following components of FIWARE.

Deployment of an NGSI-LD Context Broker on OpenShift

Earlier in 2022, the Red Hat Social Innovation Program brought the talented Red Hat Open Innovation Labs together with FIWARE for a six-week residency and deployed a comprehensive FIWARE ecosystem on OpenShift for the first time. (Read the press release: FIWARE powers eco-smart cities in collaboration with the Red Hat Social Innovation Program). Since that residency, I have had the pleasure of meeting participants from the Open Innovation Labs team, as well as the collaborators from the FIWARE and HOPU teams at the FIWARE Summit in Gran Canaria, Spain. I was thrilled to hear their story and learn about the GitOps they have shared with the open source community to deploy the Context Broker on OpenShift. 

After coming home from the conference, I couldn’t wait to try it out on my own OpenShift Local environment. Building on the great work already done during the residency, I was able to take even more GitOps capabilities I learned working with Red Hat Research engineers and contribute to the FIWARE community. My pull request was recently merged upstream to add Kubernetes Native Configuration Management (kustomize.io) scripts and make deploying a FIWARE Context Broker even easier in your own OpenShift environment. 

Integration with an NGSI-LD Context Broker

Our application now publishes data into the Orion Context Broker with Linked Data Extensions, an application supporting the NGSI-LD standard for the open exchange of structured data for smart cities. This is an exciting opportunity that enables our application to join the rest of the vast FIWARE Marketplace. I learned about the FIWARE Marketplace and the strong community of FIWARE Innovation Hubs and partners while attending the FIWARE Summit. Building with the FIWARE Context Broker means that we can combine our data with the rest of the FIWARE Marketplace applications to provide new ways to visualize our data. 

Building with FIWARE Smart Data Models

For anyone not yet familiar with the API of a FIWARE Context Broker, the best way I found to understand it better was by studying the FIWARE Smart Data Models. Models are available in a dozen current domains, including smart cities, agrifood, water, energy, environment, sensoring, aeronautics, destination, robotics, health, manufacturing, and cross-sector applications. If you are building smart solutions, you are likely to find that your data matches one of the colossal 795 defined data models combined in all of these domains! The FIWARE Context Broker supports all of these data models, which is why our project is now powered by a FIWARE Context Broker. 
Now that we have aligned ourselves with the latest standards in open source IoT technology and the cloud, we want to bring this simulated technology into the physical world. In December we submitted a renewal proposal to the Red Hat Collaboratory for our project “Creating a global open research platform to better understand social sustainability using data from a real-life smart village.” Looking ahead, together with Smarta Byar, we plan to extend our technology and traffic light algorithms all the way to the edge, with an actual smart traffic light in the village of Veberöd. We want to prove that open source technology and open collaboration can bring about real positive change and increase happiness in the village. Stay tuned for more updates on the Smarta Byar Smart Village project!

Related Stories

Intern spotlight: Eric Munson builds guitars and Unikernel Linux

Intern spotlight: Eric Munson builds guitars and Unikernel Linux

PhD interns at Red Hat Research’s partner universities play a pivotal role in bringing together the cutting-edge thinking of research institutions with the real-world expertise of industry. The PhD program enables long-term research partnerships that provide greater...

Correctness in distributed systems: the case of jgroups-raft

Correctness in distributed systems: the case of jgroups-raft

By José Bolina Building distributed systems is complex work, but strong primitives with well-defined guarantees and an expected behavior can make it easier. With stronger guarantees in primitives come strong safety and correctness verification requirements. In some...

Hackathons power open source technology and innovative research

Hackathons power open source technology and innovative research

By Chris Tate, Principal Software Engineer, Red Hat Christopher Tate is a lead software engineer for logging, metrics, alerts, and AI/ML smart data research projects in the New England Research Cloud (NERC) environment. He is also the creator of the Smart Village...

Intern Spotlight: Christina Xu, Red Hat Research Boston

Intern Spotlight: Christina Xu, Red Hat Research Boston

At Red Hat Research, we hire creative, passionate students ready to work and learn with a global leader in open source solutions. Our interns bring fresh ideas and new connections to challenging problems in the open source community, unlocking their own potential...

Intern Spotlight: Jake Correnti, Red Hat Research Boston

Intern Spotlight: Jake Correnti, Red Hat Research Boston

At Red Hat Research, we hire creative, passionate students ready to work and learn with a global leader in open source solutions. Our interns bring fresh ideas and new connections to challenging problems in the open source community, unlocking their own potential...

Getting started with data science and machine learning

Getting started with data science and machine learning

Data science has exploded in popularity (and sometimes, hype) in recent years. This has led to an increased interest in learning the subject. With so many possible directions, it can be hard to know where to start. This blog post is here to help.