RHRQ asked Brno research manager Matej Hrušovský and Red Hat quality assurance engineer Pavel Tišnovský to talk with long-time collaborator Tomáš Černý, a native of the Czech Republic now teaching at Baylor University in Waco, Texas.

Prof. Černý was in Brno recently as part of his highly successful student research initiative, which brings Baylor students to the Czech Republic to work with university researchers and Red Hat engineers, a project funded in part by the National Science Foundation. They discuss this project and his research on developing static analysis tools for microservices.

Matej Hrušovský: Let’s begin by introducing you. Tell us about your position at Baylor University.

Tomáš Černý: This is my sixth year at Baylor. It’s the hardest, because I have my tenure review. I am also teaching software engineering, and I’m active in software engineering research and the software engineering doctoral program. Currently, we are looking into code analysis of microservices and cloud-native systems.

Baylor is one of the best schools in Texas. It is an interesting school of about 20,000 students that recently joined the R1 tier (a US classification meaning “very high research activity”—Ed.), so we have a lot of funds coming into research. We’re a small computer science department; we have less than 20 faculty members. Our graduate program has grown significantly in the last few years, and we now offer an online master’s program. The research spans from software engineering, of course, to machine learning, data science, and cybersecurity, so there are significant investments in those areas.

Matej Hrušovský: Before Baylor, you were at the Czech Technical University (Prague) in the faculty of electrical engineering. Can you tell us a bit more about your history, like how we met and how your first cooperation with Red Hat started?

Tomáš Černý: Some of my students at CTU reached out to me and said, “They teach such cool classes in Brno with Red Hat. We would really like to have those classes as well.” So it was a student initiative. I reached out to Jiří Pechanec (an engineer in Brno-Ed.), who was teaching that course, and asked, “Would it be possible to do something similar in Prague?” Then the wheels started to spin, we initiated multiple classes, and it was amazing. The students really loved it. Then we started doing other things: the grant we got with colleagues in TAČR (Technology Agency of the Czech Republic) and the Red Hat Lab in Prague. But about that time, there was a new position open at Baylor in software engineering.

And since I got my master’s from Baylor University, I reached out, and they were as excited as I was. So I ended up here, but the collaboration with Red Hat continued, and I’m extremely happy about that. At this point, we have supported nearly 30 students who were able to do research with Red Hat over the past four years, and they’ve had a life-changing experience. Anytime I reach out to one of those students, they are responsive, they are excited, and they don’t regret doing research with us.

Matej Hrušovský: I remember when you moved to Baylor, you immediately reached out and wanted to collaborate with Red Hat. At the same time, I’m sure Red Hat was not your only opportunity for industrial cooperation. Is there a reason you chose Red Hat?

Tomáš Černý: I think your culture is unique—I don’t think any other company has that. Presenting open source software to funding agencies and universities, versus proprietary closed software, makes a significant difference. Open source research creates a direct path for the student to make a contribution. If they’re working on research and their contribution is to a company product, no one can ever see that. Especially for students who are maturing and learning how the industry works, this is a fantastic experience no other company can offer.

Matej Hrušovský: Eventually, you were even able to get a National Science Foundation Grant in cooperation with Red Hat as an industrial partner. Can you tell us how that began? 

Open source research creates a direct path for the student to make a contribution.

Tomáš Černý: I still have warm feelings for the Czech Republic and for Red Hat as well. So I thought, let’s take the students from Texas to the Czech Republic to do research. We reached out to Red Hat, and now we are in the fourth year of this project. We have 40 publications out of this collaboration, which is a giant number, and I have broadened my current research around all that. I’m really amazed by what has happened.

Matej Hrušovský: What was the first experience like? I know you spent one month in Prague at CTU and the second month at Red Hat in Brno. 

Tomáš Černý: It was great. When we were in Prague, one student said, “Every building here looks important.“ You can imagine how the buildings differ in Texas. You rarely have a building more than 100 years old. The students were very impressed. Then, of course, they learned to do research, and their research was very successful. Very quickly, we got papers published based on prototyping and addressing practical problems. 

Of course, there were some challenges in getting used to research, and we had to learn so much the first year. Now when students come with zero experience, we know exactly where to start. The work must be precise; we have to do a sufficient survey of the literature and the prototypes and directions for a given research problem. We also need to describe the problem very well. 

Now, after COVID, we want to collaborate with Masaryk University (MU) as well. There are many universities in the Czech Republic, but MU is a great partner of Red Hat. We could offer a double-degree program with the university where students would stay two years in the Czech Republic and two years in the United States or vice versa. They would have two mentors, and they could research practical topics that Red Hat could transfer to practice. Especially being a software engineer, where research is not the core of our work, that’s exactly where Red Hat fits in very well.

Matej Hrušovský: What has it been like to combine the academic side at CTU with the applied side at Red Hat? 

Tomáš Černý: Having the perspective of academia plus an industrial partner is very important, especially for inexperienced students. They might question whether these things are purely academic or also practical. Would the students do the research without this program? The difficulty is that they don’t know how some simple things work, and we can teach them that. They also don’t have the motivation without the industry perspective, so they might be thinking, “Is this even necessary? Is this even useful?” When they see someone from an industry who is actually interested in the topic, they don’t question anymore. 

Matej Hrušovský: Do you have any particular projects already planned out for the students? Are there any technologies in mind? And I would very much love it if Pavel, our engineer, chimed in with his more technical questions at this point.

Tomáš Černý: This year, the topics are related to architectural visualization and description languages. One very exciting thing is architectural degradation, or architectural decay. We build systems that work very well, but over time they suddenly stop working as anticipated. How do we know what went wrong? This is a very challenging question. Maybe you’ve heard about technical debt. We make small contributions to a project by offering new features, but we are building a debt by implementing it quickly. Eventually, we have to pay off this debt with interest when something slows down, breaks down, or is inefficient concerning maintenance.

Having the perspective of academia plus an industrial partner is very important, especially for inexperienced students.

The question we want to answer this year is not necessarily how we can solve this, but how we can visualize the changes in the system. If we can see the system from both the dynamic and the static perspective as it is now, is it very different from the previous version? Can we highlight the changes between the versions?

There are three aspects to this problem. There’s the architectural description language that maintains what should be preserved within the system when it evolves. There’s the visualization of the dynamic and static perspectives of a system. I know that Pavel’s interest is in seeing how the data flows through a log pipeline. Finally, my interest is how we can visualize that the system has changed and observe the impact of the change. If something changed recently, can we observe the impact on other dependent systems? We also want to trace the consequent changes, just like we do with a Git tree.

Pavel Tišnovský: So we as DevOps engineers need to comprehend what’s going on because the world of architectures based on microservices is very, very large. From my experience, almost nobody from the team understands all the consequences when something gets changed. It’s vital to us to say, “When we change this configuration, this path, or how the data flows, what will be affected by it?” 

Tomáš Černý: Yes, and this is typically done by dynamic analysis and by tracing. Dynamic analysis is certainly a great direction that everyone has taken so far, but when you apply dynamic analysis, you apply it when the system is in production. Isn’t it too late if your customer is testing something you should be aware of?

So, in our long-term project goals, we are also looking into static analysis. This is challenging because you have systems coming from Python, Go, Java, and C++, and you have to combine the knowledge within them. We address this by using intermediate representation. There is one great approach that does a similar thing, LLVM, but the problem is that it’s a compiled-level intermediate representation. We want to recognize at a high level the components being used in the system. We cannot compile the code because we would lose this information. Still, we would like to have this intermediate representation of the component level from different languages. If you look at microservices, they do recognize components. Who would build microservices without components, and why?

If a component is an element we recognize, and the intermediate representation model of the system is based on components, then we don’t care about the languages. We can start to reason about those systems, whether it’s architectural reasoning, security reasoning, privacy reasoning, and so on. Yes, we face many challenges, but there are also many things already answered from that perspective. 

Pavel Tišnovský: Right now, it seems the world is moving towards microservices. My feeling is that this trend arose from the pains of monolithic architectures. But is there any strong scientific background on it? For example, we have some scientific background for compilers. There is some scientific background for functional programming or object-oriented programming. Would it be possible to have something for the microservices, mesh-based world?

Tomáš Černý: You are asking a great question. Right now, microservices in cloud-native systems are very much driven by the industry. Industry is much faster in implementing things than academia. In the past, we were building monolithic systems, and we have been very comfortable with that. But with microservices, we have to think in a decentralized mindset, so we lose the system-centric view. We need to understand the holistic system perspective so we can establish some reasoning on whether an aspect of the system has the right quality or not. 

Over the years, we have established evaluation criteria that allow users of monolithic systems to say, “Is this the right approach to build a system?” For a distributed system, we have to develop new metrics and patterns, and we need the mechanisms to detect these. If suddenly we are operating at multiple codebases, the problem is significantly harder. Can we do this with dynamic analysis? Currently, we can detect cyclic dependency, coupling or structural coupling, and similar measures. But if we have access to the code and get the entire perspective before something gets published, we certainly have much stronger quality assurance. 

The development process with decentralized systems is complicated because you have independent teams. They make their own decisions, and something that doesn’t cause a problem in their microservice might have a significant impact on the overall system. Right now, we deploy it and wait for DevOps to see it and say, “Something went wrong. We have to fix it.” That’s not the right approach. The most efficient approach is to get feedback immediately: “We have committed something that will have some implications.” 

Right now, those tools are not there. We need those tools because we would like to have the comfort of building decentralized systems with a monolithic-like view, and there is nothing like that at this point. To get there, we need to determine what a system-centric view of a decentralized system looks like, so we can assess the implications and impacts of a codebase change within one microservice on the overall system.

Pavel Tišnovský: What about security? Will it be possible in the future to have some theory on how to make systems that are secure? Or rather, to have some patterns or rules, not just about what not to make, but how to make the system reliable and secure enough?

We need to determine what a system-centric view of a decentralized system looks like, so we can assess the implications and impacts of a codebase change within one microservice on the overall system.

Tomáš Černý: First of all, there are three perspectives. The systems have to be deployed somewhere. We have the operating systems and the containers, and there is research on how easily, if one container is compromised, it can spread to the other containers. 

Then we can look at the security within the implemented system. Since it’s decentralized, we have custom roles for role-based security in one system part and different ones in another part. Do we enforce it in a similar way across the holistic system? Right now, we depend on code analysis done manually, and no one is going to check this regularly, right? The system evolves very quickly, and one of your microservices can end up having a restriction, but another completely avoids the restriction. You don’t even know what data you are sending out; maybe those data are private.

Then there is an entirely different area of research: secure by design. But are developers experts in security? No. Are they going to understand the policies and the guidelines for security? No. So what do they end up with? They end up with whatever they have been doing until now. We need a tool that will indicate, “Hey, something is wrong.” The problem again is the heterogeneity of those systems.

So for all these reasons, we need new tools that will tell us what our system looks like. Are those tools going to be based on dynamic analysis with tracing because it’s comfortable and easy to add? I don’t know. I think we should look for static analysis to answer those questions. Are there any tools that could do that? No, but I believe there should be, because look how many problems it would solve.

Matej Hrušovský: I think you’ve answered all our questions. Is there anything that you wish we had asked that we didn’t?

Tomáš Černý: No, but I’ll add that I’m excited to work with you guys! There is a great opportunity to look into the static analysis we are currently researching. I talked at a Europe RIG meeting about static analysis used for microservices, which I hope motivates others to get their hands on it and contribute to it. 

So if you are an academician, reach out. If you are a technical person, reach out. You can either test our tools or share with us the challenges you are facing but don’t have the resources to solve, and we can develop a prototype. We cannot promise a fully working tool, but a prototype proof of concept could solve some of your problems. Then it could have the momentum of the community to spin off a community-based open source project.

Matej Hrušovský: That is the hope of everything Red Hat Research is doing: connecting the right people and creating opportunities for people to work together on interesting research that may be useful elsewhere. Thank you very much for finding the time to talk with us.