Story image

Mercedes F1 head of IT discusses data storage – and the woes of backup

19 Jul 2018

Formula One (F1) is like nothing else in the sporting sphere, a brutal sport that leaves no prisoners.

It’s an increasingly demanding technical and human challenge that combines cutting-edge technologies and innovation, high performance management, and elite teamwork.

Recently I was invited to take a tour of the Mercedes-AMG Petronas’ team's stunning garage and trackside facilities at the British Grand Prix at Silverstone.

Mercedes-AMG Petronas is a team made up of nearly 1,500 employees within two technology campuses to design, develop, manufacture, and race the cars.

There’s no doubt that data is one of the most critical assets a business has, with the effective backup, management and leveraging of data directly leading to competitive advantage in practically every industry in the world.

Astonishingly, each car from the team has more than 250 sensors that generate up to 500GB every Grand Prix weekend, while the Brackley technology campus averages 45TB every week in a 97% virtualised estate.

It’s almost like a military operation, where teams have to constantly move and establish bases on trackside, construct data centres for collection and analysis, before packing up and moving onto the next base to do it all over again.

And this is where Mercedes-AMG Petronas found themselves in a bit of a pickle – specifically, the issue of backup.

Up until recently, its backup system was unwieldy, unreliable, and slow – and head of IT Matt Harris who has been with the team for 20 years in various IT roles had had enough.

“Over the years we've organically changed substantially. When I first joined the team, we had a small amount of Linux systems, with most of the systems being Windows. Back then that meant we had various different backup solutions to serve the client,” says Harris.

“About seven or eight years ago we got down to one system that could backup both environments. However, with this solution we had one of my team essentially changing tapes or managing backups full-time. Of course, the more clever the person is the better they could make our backup system work, but then you're wasting some of the most talented resources by giving them the most mundane job to do."

Harris says it was a job that no one wanted to do, and hence it was a job that wasn’t being done well.

“At the time around 20 percent of our backups were actually successful. You're talking about a part of IT that no one wants to care about. It’s probably the least sexy part of IT you could possibly think of because you don't care about it until something goes wrong,” says Harris.

“Buying storage was never a hard conversation, but it was always difficult to justify the extra to protect it. We wanted to look at this slightly differently to enable us to recover data at varying levels of recovery points, without something that required management all day, because it's just painful. A bit like what we had done with our storage a number of years ago, we decided to take a step back and determine that we couldn't keep doing this, our team is not big enough to manage backup. We need to make it something that just happens.”

In September 2017 Mercedes-AMG Petronas announced that it had chosen a new partner to solve its woes, but this came after a lengthy evaluation process.

“We spent around 6-9 months learning what was out there. A massive turning point for us was we met Rubrik, first off on a Webex conversation and we were so blown away with the conversation we had with them that we really didn't look at anything else from that point onwards. We quickly learnt with Rubrik that there was so much that we could do other than just business continuity solutions,” says Harris.

“We looked at seven different solutions, I don't want to start naming other products but you can think of the very well-known traditional backups. With Rubrik it was a different conversation and it just felt comfortable. At the end of the day whoever we chose had to be a part of our team and they couldn’t be a sort of arm that we would have to ring up with a difficult relationship.”

Harris says the team’s Pure Storage servers integrate with Rubrik nicely.

“The ability to do some very clever stuff around backup workloads where we can perform a backup with absolutely no hit on the server. Because Rubrik orchestrates a snapshot, backs it up, mounts it and then does the inline backup offline, it provides us with a point in time backup that we know is absolutely sweet with zero hit on the system so we haven't got that eight hour window where our servers are limping with the load of backing up,” says Harris.

“From the word go when we set it up there has probably been around an hour's worth of training for my team. It's very simple and there's not a lot that you need to understand. Every year the data that is produced in Formula 1 increases significantly, so we're just trying to make sure that in the future we can manage that growth of data.”

I then asked Harris if the team had ever experienced a major data outage – which made him wince.

“I don’t think I’ll ever forget this for as long as I live if I’m honest. In 2008 someone turned around and deleted the wrong volume because back then we had two volumes, one labelled Backup and the other labelled Live. And on that day they were actually named wrongly. You simply can't make that kind of mistake in an everyday business, and you definitely can’t in an F1 team,” says Harris.

“We had an Oracle database and a file backup completely out of sync so we had an Oracle database pointing at data that wasn't there, and data that was different to what the Oracle database was pointing at, and data that was here but not even in the database as someone had deleted it. It was just a nightmare. In those days we were fortunate that we did have backups, but they weren't in sync. The snapshot technology that we have now enables us to protect the Oracle database and the file systems and get them back into service promptly.”

How Dell EMC and NVIDIA aim to simplify the AI data centre
Businesses are realising they need AI at scale, and so enterprise IT teams are increasingly inserting themselves into their company’s AI agenda. 
Orange Belgium opens 1,000 sqm Antwerp data centre
It consists of more than 500 high-density 52 unit racks, installed on the equivalent of 12 tennis courts.
Time to build tech on the automobile, not the horse and cart
Nutanix’s Jeff Smith believes one of the core problems of businesses struggling to digitally ‘transform’ lies in the infrastructure they use, the data centre.
Cloud providers increasingly jumping into gaming market
Aa number of major cloud service providers are uniquely placed to capitalise on the lucrative cloud gaming market.
Intel building US’s first exascale supercomputer
Intel and the Department of Energy are building potentially the world’s first exascale supercomputer, capable of a quintillion calculations per second.
NVIDIA announces enterprise servers optimised for data science
“The rapid adoption of T4 on the world’s most popular business servers signals the start of a new era in enterprise computing."
Unencrypted Gearbest database leaves over 1.5mil shoppers’ records exposed
Depending on the countries and information requirements, the data could give hackers access to online government portals, banking apps, and health insurance records.
Storage is all the rage, and SmartNICs are the key
Mellanox’s Kevin Deierling shares the results from a new survey that identifies the key role of the network in boosting data centre performance.