IT and software development
Swipe, like, share, repeat. Going viral isn’t an exact science—but at BuzzFeed, quizzes and political exposés alike can make it around the world with just a few clicks. For hundreds of millions of users, the leading digital media publisher is the first stop whether you need a break from work or the latest breaking news.
Managing a cross-platform news and entertainment network of articles, lists, quizzes, videos, original series, lifestyle brands, and world-class reporting requires a flexible infrastructure, one which can support multiple deployments per day. Behind BuzzFeed.com is a distributed team of developers and engineers located from California to London to Argentina, with tech users, IT teams, and BuzzFeed News reporters building and running code.
At the start, these teams deployed manually, pushing code to production themselves. The number of clicks quickly multiplied—and not just on the website. Engineers were spending more and more time on frustrating, manual tasks, and less on building tools that allowed them to focus on critical code. “The mindset we carry within our team is that we always want to automate ourselves into a better job,” explained Director of Engineering Andrew Mulholland. “We want to make sure that the task we’re doing manually today becomes mostly automated. We focus on building tools that serve as an enabler for developers.”
With an automation-first approach, Mulholland turned to solutions already in BuzzFeed’s tech stack. GitHub Enterprise had been used within the company by developers and journalists alike for years, but quickly became the foundation for a modern development workflow built on collaboration and monorepos. “One of the great things about BuzzFeed’s culture is that, if you see something broken, and you offer up a pull request to fix it, people will be happy to review the code and approve it,” he said. “Monorepos make that even easier. You have access to all code by default and this serves as an enabler for learning.”
Beyond the tech team, users across the organization have access, including some of the Buzzfeed News team. The same transparency that applies to the company’s code applies to their reporting as well: data journalists publish their data to a separate, public GitHub repository, whether it’s Jupyter Notebooks or stand-alone code. “We want to try to show our work as much as possible and not take readers’ trust for granted,” said Mulholland. “When we’ve done analysis for a story, we make the tools that we used for that analysis available so people can run and verify it if they want to.”
With GitHub, this process of open code by default is seamless—anyone within BuzzFeed’s GitHub organization can open a feature request in another service. Known as innersourcing, Mulholland explained how code reuse and collaboration has allowed BuzzFeed developers to contribute to improving BuzzFeed’s infrastructure. For example, when an engineer needed custom autoscaling, the infrastructure team wasn’t available to develop it. But since the developer was able to access the team’s code, they were able to self-service and suggest the code changes themselves. “They came to us with some requests for improvements, and we said that if they helped, we’d be happy to accept that code. So they added the code in, and it’s now widely used across dozens of different services.”
Every second counts in an outage—especially for one of the world’s most-visited websites. In one instance, a service outage occurred while the original development team wasn’t available; however the problem was able to be quickly resolved. “Because the application was in a monorepo in GitHub and we have a high degree of standardization between our services, one of our infrastructure engineers was familiar enough to make changes and solve the issue.” The next day, the original team approved and merged the changes.
One of the most important roles for the infrastructure team at BuzzFeed is serving its internal developers. And local development environments are key to that. “Being able to have developers bring up a service locally on their machines is crucial to ensuring a quick service when they run it,” Mulholland said.
BuzzFeed created libraries and templates for BuzzFeed’s supported languages—Python, Golang, and Node—to enable developers to quickly start building locally. To save even more time, the team used GitHub as a core part of their highly-automated CI/CD workflows. Now, developers can immediately package and push code. “As soon as your branch is pushed up to GitHub, that will kick off a build in our CI system. Then once it has been reviewed and merged, it will be automatically deployed to production.”
By moving from manual deployments to automation with GitHub, BuzzFeed’s teams deploy 30 percent more frequently, sometimes up to 300 times per day. Now there’s more bandwidth for other top priorities—including security and open source.
Open source is important to BuzzFeed; as a user of many open source technologies the company wanted to give back to the community too—and so in 2018 opened sourced its single sign-on (SSO) solution. Before sharing the SSO project as open source on GitHub, the team worked to improve the readability and structure of its code. “When you’re under pressure to get things done, there’s an element of technical debt that creeps up,” Mulholland explained. “We had our SSO project internal for a year before making it open source. Since making it open source meant making the code public, it forced us to raise the bar and fix our code internally, and created a great learning opportunity for our developers.”
Participating in open source brings better code—and credibility. “A lot of people obviously know us for our buzzy content, quizzes, massive food brand Tasty, and BuzzFeed News, but now people understand that, ‘Oh yeah, there’s good software engineering happening there as well.’ That’s been very helpful when we’ve been hiring.”
By making the code to one of BuzzFeed’s most critical projects open source, it’s also made it eligible for the company’s bug bounty program. “With SSO being open source it’s easier for researchers to find issues—and that’s a good thing,” said Mulholland. “We then encourage the use of our bug bounty program to ensure we can reward researchers for responsible disclosure as we triage and remediate it.”
Initially, the bug bounty program was a struggle. As he explained, the all-in approach created another backlog: “In theory everyone was responsible for the triage of tickets, but by having everyone responsible, it meant that really no one was responsible. Often non-critical reports took a long time to be triaged.”
Having worked at companies with large security organizations, Mulholland knew a siloed approach wasn’t in step with BuzzFeed’s values. “It creates a very much us-versus-them mentality, and that doesn’t fit well how we do things. We wanted to create a more collaborative, inclusive approach.” So instead they built a security program that was uniquely BuzzFeed, coming up with a rotation plan that assigned an engineer from a different team each week to be on triage. “Our mindset was that by sharing the burden across the tech organization, it wouldn’t be too much of a burden to any individual or any one team.” It worked—the rotation meant that someone was always responsible and the meantime to triage and bounty dropped from weeks to days.
The improved process also brought in more bug reports, which Mulholland welcomed: “Suddenly we were getting way more reports because security researchers got the signal that we care about security, and we were paying bounties quickly. They realized it was worth their while, so they spent more time looking at BuzzFeed.” Mulholland and his team even began working with researchers periodically, inviting them to preview new features for potential vulnerabilities.
In Mulholland’s view, using GitHub has introduced more secure features into BuzzFeed’s infrastructure, along with some much-needed changes. The team just recently migrated BuzzFeed.com article pages onto a new React-based front end. “GitHub played a key role in that, with its collaborative features being a boon to development,” he noted. “By moving to the React front end, it’s more modular. It’s going to enable us to work to evolve the user experience to make it easier to find the content that you want to read, and easier for us to enable new features to put around the reading experience.”
After all, from security to CI/CD to personality quizzes, it’s about connection—and finding creative solutions that keep everyone moving forward. “GitHub has become so essential to the developer experience,” Mulholland said. “If we can keep enabling our engineers like we did using continuous deployments with GitHub, it saves our team a considerable amount of time. And those time savings are a boost to productivity.”
Start collaborating with your team on GitHub
Want to use GitHub on your own? Check out our plans for individuals