LinkedIn today announced it is swapping out its datacenters for the public cloud. The Microsoft-owned company is moving its infrastructure to Azure as part of a multi-year migration. We sat down with Mohak Shroff, LinkedIn’s SVP of engineering, to discuss one of the biggest technological transformations in the company’s history.
Microsoft announced its $26.2 billion acquisition of LinkedIn in June 2016 and closed the deal in December 2016. LinkedIn moving to Microsoft Azure always seemed inevitable. Was this part of the plan all along, part of the discussion during the acquisition talks?
“It was not,” Shroff told VentureBeat. “The integration has really been a remarkable success, primarily along one dimension, where Microsoft has really allowed us independence in decision making. And this, I think, stands as a clear example of that independence. They have offered us whatever resources we have needed, offered to work with us if we make the decision to move, but the decision has always been ours.” The Azure sales team apparently wasn’t even pitching or wooing LinkedIn. “We’ve had conversations with leaders on the Azure team. They asked questions like, ‘What would it take for LinkedIn to consider Azure?’ But were we being pitched? No.”
LinkedIn plans to migrate its 645 million members over several years so as not to compromise the site’s accessibility, reliability, and performance. “We think probably at least three years till we’re done, possibly longer than that,” Shroff confirmed. “It will be a gradual migration. We’ll see increasing workloads on Azure over time, with a pretty significant inflection point, about a year and a half, two years out from now. And then kind of an accelerated migration post that.”
Why Azure and why now?
LinkedIn has evaluated moving to the public cloud “many times” and for many years the team “felt like the time wasn’t right to make a move,” Shroff explained. The last evaluation started four to six months ago. LinkedIn looked at the public cloud landscape, its capabilities, and if it were to migrate, how it would do so. A couple of weeks ago, the team finally pulled the trigger.
Shroff now believes that the public cloud has hit an inflection point. “For a while we’ve seen a certainty to the public cloud. A certainty to Azure and its ilk on the future. They’re just trends you see in terms of the level of investment, innovation, and scale that come with the way public clouds have evolved. That makes it clear that that is the way of the future, and we want to be part of the future.”
The inflection point Shroff is referring to primarily concerns the capabilities available to LinkedIn. “Let’s say that, 10 years from now we’re finally on Azure. Is now the right time to start? So that when we’re on Azure, it’s ready for us in terms of the kinds of capabilities we would not want to build ourselves but want to access to on the public cloud. And this latest evaluation really came back with a ringing endorsement for now is the time.”
Existing Azure features
LinkedIn is already leveraging Azure. In addition to a few experiments, LinkedIn uses Microsoft’s public cloud in three notable ways:
- Azure Media Services to accelerate LinkedIn video post-delivery.
- Microsoft Text Analytics API, an Azure Cognitive Service, to improve machine translation in LinkedIn’s Feed.
- Content Moderator, a Microsoft Cognitive Service running on Azure, to help keep inappropriate content off LinkedIn.
In other words, LinkedIn relies on Azure for near-line or offline capabilities. Data flows through Azure for processing, understanding, and transforming, and then returns to the site. But requests that hit LinkedIn.com don’t go through Azure yet.
Still, these “less critical” aspects (data processing, big data management, AI, and ML capabilities), were enough to eventually sell LinkedIn on a migration. Take the first use case.
“As video has become a key part of LinkedIn, we’ve been leveraging the video capabilities on Azure to power that. We could choose to build that ourselves,” said Shroff. “But we look at the capabilities there and we say, ‘Well, this is world class. Why would we build this?’ And there are increasingly things of that sort, where it just sort of starts to make sense. Looking out into the future, we can build every one of these things ourselves, but it creates more things to migrate in the future. We feel like there’s an inevitability to the public cloud. It is the clear future of sites like ours. The migration is about balancing your ability to build things now versus the cost to migrate later. And now feels like the perfect time and the perfect balance of that.”
These offerings are available at massive scale on a system like Azure. Rather than build them out, which is increasingly more expensive and complex, LinkedIn is choosing to invest that effort into migrating.
Azure alternatives? Nope.
If Azure wasn’t an inevitability, does that mean LinkedIn took a look at other public clouds? “We’ve considered them in the past. In this instance, we did not,” Shroff said. “The primary reason for that is, we think of ourselves as a fairly complex use case for the public cloud, and have some custom needs.” Shroff insists that Microsoft played no role in LinkedIn’s choice. Whether that’s true or not, Azure stands to benefit beyond just gaining a new customer.
“One of the biggest values for us with using Azure has been the ability for people to lean in and really collaborate on building key capabilities for us,” said Shroff. “It’s really a two-way street in terms of the value there. As we are going to be migrating to the public cloud with migrating to Azure, some of the capabilities that the Azure team have built for us essentially will be made available to their entire customer base.”
All in all, Shroff sees three benefits of moving to Azure. The first is the growing number of Azure regions becoming available to LinkedIn. The site will become “more proximate” to its members.
The second is the elastic nature of the public cloud. LinkedIn has fairly consistent traffic patterns, but there are sometimes needs for bursts, especially with offline processing. The team is especially looking forward Azure’s data analytics, AI, and ML capabilities, plus the ability to leverage GPUs.
The third benefit is the ability to focus on building product, securing the site, and leveraging at-scale systems on the Azure stack over time. LinkedIn hopes that moving to Azure will free its engineers to build customer-facing functionality.
LinkedIn’s datacenters and employees
In addition to its corporate datacenter in California, LinkedIn has four leased colo space datacenters. Three are in the U.S. (Texas, Virginia, and Oregon), and one is in Singapore. That’s a lot of custom hardware and software that the company is about to give up.
“The exact end state for those datacenters is unclear,” said Shroff. “In some cases, we will vacate them. In some cases, we may find new uses for those datacenters. All of our activities that currently happen in our own native datacenters will migrate into the Azure public cloud.”
LinkedIn also isn’t expecting layoffs as a result of the move.
“We have our own datacenters. We have people who work in those datacenters. For the foreseeable future, we need these people,” said Shroff. “In fact, our scale is actually growing and we will be investing more into those same datacenters. The Azure migration doesn’t really reduce our dependence on those [employees] for quite some time yet. Post that, we anticipate being able to find roles. We’ll work with every one of these individuals to figure out what the right thing is for them. Either internally or as part of Microsoft or elsewhere. But it’s a long-term horizon.”
Staff aside, LinkedIn won’t save on other costs either. This is an investment play, first and foremost. “All blended together, we’re looking at this as a mostly cost neutral move for us,” Shroff added. “Resourcing, your point about people, as well as our infrastructure, our runtime costs, everything together. Moving to Azure is essentially cost neutral for us.”
A “cost neutral” move that LinkedIn hopes will result in a big payoff.