In this article we will discuss the log4j incident, why people are worried about
the open source software (OSS) supply chain, and how to work towards fixing it.
The spark: Log4Shell
Last week (Dec 9th) a major vulnerability was discovered in an open source
logging project for Java called log4j. The
vulnerability called Log4Shell would allow anyone to remotely run arbitrary code
if they sent a message in the right format to the server. This is one of the
worst attacks your system can be susceptible to and if you are interested in the
technical details of the problem, here is an
overview.
The attack surface of Log4Shell is
staggering,
Amazon, Apple, Google, and the Apache Server are
affected; it can almost not
get bigger than this. We will see the real fallout of Log4Shell in the next
weeks and months as right now servers worldwide are being scanned and prodded
for this vulnerability.
Since there have
been
many
supply
chain
attacks
recently,
the whole conundrum sparked a debate in the OSS and infosec community: Many
believe that the OSS ecosystem is
broken,
maintainers need to become more professional and make OSS maintainer
a real job. Some argued
that in this case the problem was not that maintainers were unpaid, burnt out,
and taken advantage of, but more how this particular feature was implemented in
log4j (Note:
Maintainer burnout
is still a
real and
significant
problem for
security). Others
insisted that open source is
not broken - society and
capitalism are the real culprits and everyone involved in OSS
knows what
they are getting into.
Open source as a model of distribution, development, or business is not a model
of either a dystopian nightmare or an utopian dream. Every project is different
and there are no silver bullet solutions to sustainability.
Open source maintainer as a real job
It is a real problem that software engineers maintaining critical software
infrastructure used by governments and corporations worth billions are not able
to make a living off of it. Maintainers often can only work on OSS in their free
time. This is fine for a pet project, but critical infrastructure projects, such
as logj4, should be more resilient. People who are well off enough or receive
enough donations to be able to work on their projects full-time are likely a
tiny fraction of all open source maintainers.
In a perfect world, everyone who is maintaining such an important piece of code
can do it full time and with adequate compensation. But this is not a perfect
world. The best we can do is work on securing each link in the chain.
Sponsorships
GitHub sponsorships and
Open Collective are a good start, but not enough
to sustain infrastructure development. For example, the Ory ecosystem (most
notably Ory Hydra) - used by billion-dollar companies and securing >30 billion
requests per month - has received 22k $ on Open Collective over the last six
years. That is not a small amount compared to what most other OSS projects
receive. Still, if split between the two original core maintainers
(@aeneasr and
@zepatrik) it would amount to about 150$/month
over the years, which is an absurd amount for a full-time maintainer that
requires a deep level of expertise in security, cryptography and web
infrastructure - not counting the additional maintainers that have been added to
the project since its inception.
Towards sustainable open source maintainership
Making a living off open source software and being able to work full time on it
is a dream for many maintainers. At Ory, we are working hard to make this dream
come true. All our open source packages (visit this page for
a full overview) are now led by maintainers
paid full time for their work.
Here are three practical steps that every OSS maintainer can take if they would
want to professionalize their project:
Reach out to your network of contributors, maintainers, and software engineers.
See if anyone using your software at a business can make a sponsorship happen -
much is possible when you are asking the right people.
This sounds scary, but it will be much easier (or rather possible at all) to
collect funds from BigCorp if you are an LLC.
The trick is that you can easily incorporate a pass-through US LLC and open a
business account for it even if you're not a US citizen.
(source)
Create a GitHub/GitLab organization for the project to make it more resilient
(multiple code owners). Set up a landing page with clear links to your
sponsorship channels and contact information.
This only scratches the surface of what is required to make OSS development
sustainable. At Ory, we build a commercial service
on top of our OSS work. This creates a positive feedback loop: As everyone is
using the same base Ory services, improvements on the commercial Ory Network are
based on improvements to Ory Open Source, while contributions from the OSS
community benefit users of the Ory Network in the same way.
What about dependencies?
Dependencies play a major role in the saga of the log4j vulnerability and
security complications in general. It is mind-boggling how big dependency trees
can get, in many cases, people had no idea they were even running log4j between
the thousands of dependencies in their stack.
Ory depends on many software packages (e.g. see the dependency list of Ory
Kratos here), so it is also
in our and our users best interest to ensure a secure and hardened OSS supply
chain. Ory uses automated tooling in the CI pipeline to
scan docker images
and npm-packages for vulnerabilities as well as carrying out regular independent
security audits of our libraries and dependencies. A
"Software Bill Of Materials"
can help as well, watch out for this topic in an upcoming blog post.
Conclusion
Is the path we chose at Ory the definite and only way to build and sustain open
soure software?
Probably not. For many projects a professional commercial structure would be
overkill and many maintainers - for good reason - don't want to deal with the
administrative, legal, and other matters that come with running a professional
business. There are options for OSS maintainers to make a living off their
craft, many more than there used to be just a few years ago. Big companies often
want to support and fund the open source software their business runs on. The
structures and frameworks for them to do this efficiently are still emerging,
but we are confident that the future of software lies in OSS.
Open source isn’t broken. It’s working exactly as intended, and it’s by far
the most powerful force in the technology world, and it will outlive any of
the corporations so many people bend over backward to please today.
(source)
Fund open source software
If you want to support Ory Open Source, find
us on Open Collective or better yet
sign up for the Ory Network and get immediate
value for your support.
