Story image

De-duplication technology poised to take over storage market

01 Oct 2009

Dr. Kai Li, a professor at Stanford University and co-founder of Data Domain, was recently in the country to give a   guest lecture at AUT about de-duplication technology and the massive potential that can be harnessed by reducing   redundant data.

Li said the technology will replace tapes and that data de-duplication will do for the storage industry what the iPod did  for the music industry.

“As humans we just don’t like tapes,” he said. “That’s why we replaced music cassettes with the iPod, we’re replacing  VCRs with DVRs, and now we’re replacing backup tapes with deduplication storage.”

Unlike backup tapes, which take a long time to access the stored data, de-duplication technology can access it quickly  and, if the software has strong architecture, data can be recovered reliably. Li said that with manual recovery of tapes,  the probability of a successful recovery is 90% because 10% of the time the tape can’t be found or read back. Another area where data deduplication presents opportunities is within archival storage, especially with emails, which typically have a lot of duplicate copies stored, so the compression rate is typically five to one.

“De-duplication can be done in multiple locations, it can be done at the source, or at the server, but in that kind of  situation [archiving] de-duplication can help reduce network traffic in addition to reducing the space and power in the  data centre,” he said.

Like with any technology, Li said there will always be those late to catch on, but like the iPod, the technology is set to  revolutionise the storage market.

“Any revolutionary replacement will take time. There are always people who are very eager to replace, there are  always people who are resisting changes,” he said. “So this change will take time, but it is clear, I think in the industry,  and in the marketplace, this is the right thing to do.”

But just as all mp3 players are not the same, Li said that as de-duplication becomes a ‘hot product’, adopters should  be wary of the number of products in the marketplace by testing the actual compression ratio before committing to any  one technology or product.

Li said: “You can call one thing data deduplication, but a good de-duplication technology will give you a much higher compression ratio, which is what you want".

Intel building US’s first exascale supercomputer
Intel and the Department of Energy are building potentially the world’s first exascale supercomputer, capable of a quintillion calculations per second.
NVIDIA announces enterprise servers optimised for data science
“The rapid adoption of T4 on the world’s most popular business servers signals the start of a new era in enterprise computing."
Unencrypted Gearbest database leaves over 1.5mil shoppers’ records exposed
Depending on the countries and information requirements, the data could give hackers access to online government portals, banking apps, and health insurance records.
Storage is all the rage, and SmartNICs are the key
Mellanox’s Kevin Deierling shares the results from a new survey that identifies the key role of the network in boosting data centre performance.
Opinion: Moving applications between cloud and data centre
OpsRamp's Bhanu Singh discusses the process of moving legacy systems and applications to the cloud, as well as pitfalls to avoid.
Global server market maintains healthy growth in Q4 2018
New data from Gartner reveals that while there was growth in the market as a whole, some of the big vendors actually declined.
Cloud application attacks in Q1 up by 65% - Proofpoint
Proofpoint found that the education sector was the most targeted of both brute-force and sophisticated phishing attempts.
Huawei to deploy Open Rack in all its public cloud data centres
Tech giant Huawei has unveiled plans to adopt Open Rack proposed by the Open Compute Project in its new public cloud data centres across the globe.