Is it possible to add a Backdoor entry into OpenStack?

During the The New Stack Analyst Podcast on OpenStack Contribution Trends, @VirtualTal of Adallom raised an interesting question about one of the contributing companies who were blacklisted by many governments. His concern was the possibility of this contributor adding backdoor entry code into OpenStack code. Though I addressed that briefly, I couldn’t explain in detail since the focus of Podcast was on contribution trends.

@ReneBuest had also had a tweet recently highlighting OpenStack contributions from NSA and his later tweets showed his possible concern of adding backdoor entry into OpenStack.

This post addresses these concerns (short answer is NO!), by explaining the OpenStack Code Submission Process. Here is a good link on OpenStack Code submission workflow which uses Git, Jenkins and Gerrit. Of course, you know where the OpenStack source code repository is!

OpenStack Code Submission Process

OpenStack has more than 2000 contributors, 100 contributing companies and more than 10 official projects, few incubated projects and more supporting projects and all such madness is self-regulated :).

Let’s say Alison, a developer, wants to submit code changes. Let us assume she’s has got her development environment setup (if not, I would recommend her to setup instructions here).

Now that she has code changes ready, here are the steps she takes to submit those changes to the project repository she is working on.

  1. Now that Alison has code changes, she runs the unit tests locallt, gets ready to ‘commit’ her code
  2. She commits her code (by running git commit)
  3. She requests for code review (by running git review).
  4. At the same time, the changes are queued to be tested against Jenkins based functional gating tests. Success or failure is provided as a ‘review’ feedback with +1 or -1 vote.

Her code changes will now show up now at the review queue, awaiting code reviews. At this point, any other developer can review this code patch, and gives a +1, 0, or -1 vote, indicating approve/ neutral/ disapprove. If the developer is also one of ‘core reviewer’ she can also give a +2 vote (explaining why +2, instead of regular +1).

5. Once the patch passes the tests AND gets a +2 review from one or more core developers, Alison can request for merging the code upstream. The gate tests are run again, and on passing, code changes will be merged upstream.

If there are any merge conflicts at this point, or gate tests fail, Alison needs to fix them and submit again.

OpenStack Review Process

Any OpenStack developer can review and assign a +1/0/-1 score/ vote according to her approval/ neutral/ disapproval ratings. There are minimum recommendations on what to look for while reviewing the code. As you can see from there, each project has a review bar to meet. This is to ensure that spell-check reviewers don’t get much of a say during code review 🙂

Only when the code path has got +2 votes from one or more active core developer, it is ready to be merged. Here, the code reviews from core developers are trusted, and they have a large responsibility in ensuring high quality code gets merged.

This is why, a code patch could remain un-merged even after getting several +1 votes and passing the gate tests. This is because, each code patch needs to get a +2 vote from CORE reviewers in order to get merged.

Who is a Core Developer?

A core developer is a one who has submitted enough HIGH Quality code and done enough HIGH quality code reviews that the community trusts her code reviews. And in order to become a core developer, she has to meet requirements and has to be approved by at least 5 existing core developers (or majority in case of less than 5 core developers already in that project).

OpenStack, being developer driven, runs through collective participation. One establishes her credibility through active and high quality participation. Once established, the community trusts her capabilities.

Now back to the original question:

Can Backdoor Entry be added?

As you can see, the code to be submitted is reviewed by peers and core developers. If someone has malicious intent to push such code, it has to really escape the attention of other developers and gets approval from at least 1 core developer who needs to vote +2. Such +2 votes are scrutinized more, so it really needs approval from 2 core developers.

It is highly unlikely that 2 core developers would approve malicious code with the intentions of pushing backdoor entry in to OpenStack.

Here is why it is not likely:

  • If one has malicious intents to begin with, they wouldn’t have become core developer. They wouldn’t have passed approval process.
  • Let us assume one suddenly developed such intents (cinematic albeit) after becoming a core developer. Even in this case, there are enough checks in place ,  other core developers will override such code approvals.

So it is not really possible.

Can this really happen at all?

This could happen if a project has ALL core developers who has the intent to do such things. That means, it can’t happen in any of the existing OpenStack projects.

Someone has to start a new project with the sole intent of pushing such code, and be able to run that by all of their resources.

And that has better chances of happening in some other community/ project, not OpenStack!

Cloud Don Take

While it is a valid concerns for customers, they need to understand that no existing software is going to be 100% fool-proof, and OpenStack is no exception. However, you have excellent guidelines and success stories in implementing secure cloud using OpenStack. One can confidently use OpenStack to implement a secure, scalable cloud.

Customers need to look in to the complete picture – accompanying hardware, their security practices, processes etc. Analysts need to explain customers the entire picture, rather than pandering to incorrect FUDs/ incomplete questions like ‘Is OpenStack secure?

5 thoughts on “Is it possible to add a Backdoor entry into OpenStack?

  • The irony of Huawei being blacklisted aside, being dismissive of this (very real, I would say) threat is dangerous, particularly given your argument depends on trusting individuals, and assumes that “many eyes make bugs shallow” (refer to Heartbleed for a counterexample). In any case, we know that attackers only ever work alone, right?

    • Sam, I understand your objection, yet I honestly have a hard time imagining someone investing the social and intellectual capital in becoming a core developer of OpenStack only to throw it all away in order to compromise one of the providers running OpenStack.

      You’d have to not only acquire the skills required, but also spend so much time putting them to work on OpenStack that you’re granted core status. All that to get *one* chance of sneaking a backdoor into OpenStack so that you *maybe* can steal someone’s data and never be able to show your face again?

      Heartbleed was — to the best of our knowledge — unintentional, so has little relevance here, afaics?

      If you want to get a backdoor into OpenStack, I think a much more real target is the OpenStack review infrastructure itself. We have top people keeping it safe, and injecting malicious code into a git branch would most certainly be discovered quite quickly (anyone trying to pull from that branch would get an error about revision history not matching what they have locally), but compared to your malicious core developer theory, it’s a much more likely attack vector, IMO.

      However, the whole talk about injecting backdoors seems like a red herring. If I wanted to compromise an OpenStack cloud, I’d probably just spend a bit of time finding existing security holes and use those. I’d be surprised if there weren’t a bunch.

  • Sam, thanks for the comment. We could definitely do more, one such effort is trying to add automated tests that scan for possible anti-security patterns (just getting started). My point is, it is not correct to say that anyone can add malicious code to OpenStack just because it is open source. And more possible place is not core OpenStack code, but may be appliance they are wrapping it around. Finally, ‘this’ threat is true for ‘any’ software, not just OpenStack.

  • I like the way Sriram framed the landscape and didn’t notice as dismissive a tone as maybe you did @samj but I know @sriramhere pretty well. I think what’s most interesting about the article is what happens now. The gentleman @ReneBuest who posted NSA on twitter is in France. The gentleman from Equinix is in Switzerland (resides at the moment) and I am pretty sure Siren is not a Massachusetts resident. I think it’s interesting the question and implications are raised mostly by European folks who are rightfully more (…) than North American cloudies on this topic. What is much more interesting is what our dialogue can uncover potentially. I am very interested in loading this structure /process up and seeing where it may fail so it can be buttressed. Please can you guys elaborate on other attack vectors and approaches to skirt the process? I think the most obvious one that is plausible is that “someone hijacks a core developers account…” as Sriram says “This could happen if a project has ALL core developers who have the intent to do such things.” but wouldn’t the community review process catch such Trojan attacks? More importantly what are the most plausible entry points for threats and likely the weakest aspects of the OpenStack review process that could be addressed? Finally are we following best practices vis a vis Linux community process? Better or worse? What are the best opportunities and improvements to make to strengthen the bulwark(s) and proactively defend?

  • Sam/ Soren/ Josh:

    Thanks for the comments and feedback. Apologize if the tone was dismissive. This blog post was about the possibility of a code contributor messing with OpenStack code base intentionally. This doesn’t mean a hacker couldn’t break into an OpenStack cloud deployment or there are no security bugs in OpenStack :).

    Also, code reviews are not the only checks the community does. We have a work group called OpenStack Security Group, that actively triages security bugs, publishes Security Advisories/ Notes. They are the ones who published the OpenStack Security Guide (https://wiki.openstack.org/wiki/Security/Projects). There is also a small/ protected Vulnerability Management Team which deals with vulnerability disclosure and handling. There are also some early stage efforts such as Threat Modeling/ Fuzz Testing etc.

    You can find more details here: https://launchpad.net/~openstack-ossg or https://wiki.openstack.org/wiki/Security/Projects.

Leave a Reply

Your email address will not be published. Required fields are marked *