Hadoop might be the next big thing on Wall Street.
Source: flickr user matze_ott
I wrote last week about the firestorm of M&A action in the analytics space, but acquisitions alone don’t tell the whole story of why this space is so hot. If anything, they’re merely the effects of a groundswell of customer interest in the market. The results of two surveys released this week help fill in the details — especially why it pays for vendors to establish a foothold in the analytics market. For vendors utilizing Hadoop, the promise might be even sweeter: It’s a hot technology with relatively few competitors and much room for improvement.
Let’s start with demand for analytics. Where better to do so than in the filthy-rich, technology-savvy financial services market? An InformationWeek survey commissioned by SAS and Platform Computing shows that both buy-side and sell-side firms are struggling with too much data, inflexible analytics software and hard-to-understand results. Two-thirds of respondents are uncertain if their current infrastructures can keep up with exploding data growth. Tack on that the majority of all firms are struggling with integrating silos of departmental data, and that a quarter of midsize firms are experiencing significant issues with compute capacity. You suddenly have a lot of firms in need of better IT solutions.
The results of these unmet needs are actually quite troubling. As the survey notes, inadequate risk analysis was a factor in the recent economic meltdown, yet a majority of firms surveyed admitted to performing key analyses either never or only in an ad hoc manner. The good news is that, in the next year, many plan to increase their focus on risk management across the board. To help address analytical needs, more than half of all respondents are considering investments in straight compute solutions like grids, GPUs and cloud computing. An equal number (actually, a greater number among buy-side firms) are eyeing up new business analytics tools.
Enter Hadoop. The use cases for Hadoop in financial services are widely recognized; leading-edge firms already are deploying it. Cloudera cites financial services as one of two vertical markets adopting Hadoop most aggressively, with primary use cases being fraud detection, complex ETL and risk analysis. (Hadoop is not a real-time solution; we’re talking batch analysis here.) At Cloudera’s upcoming Hadoop World event, in fact, one of the first presenters will be Bank of America’s managing director for big data and analytics. A year ago, already, Vertica indicated that roughly 10 percent of its customers were in production with Hadoop — a trend spearheaded by its financial services customers. On the infrastructure level, storing and processing data with Hadoop is also relatively inexpensive because it’s ideally suited to run on clusters of commodity servers.
But Hadoop isn’t perfect. As I highlighted in a GigaOM article this week, a recent survey commissioned by Hadoop-centric startup Karmasphere indicates that although developers working with Hadoop love the product, there are several big obstacles to using it. The top three concerns: “steep learning curve,” “hiring qualified people” and “availability of appropriate products and tools.” Products aiming to make Hadoop easier to deploy and use — like those offered by Cloudera, Karmasphere, Datameer and IBM — directly address all three of these concerns.
It doesn’t take a genius to connect the dots. Financial services firms (along with companies in many other businesses) are facing increased pressure to analyze their data, and they’re increasingly turning to Hadoop to help them do it. In fact, they’re looking to place greater emphasis on the types of analysis that Hadoop can help with, which means spending more money. Although the number of BI and data warehousing vendors supporting Hadoop in some form is relatively large, the number of vendors peddling Hadoop-based solutions is small. If they’re not already, those currently operating should start experiencing significant customer traction. But given the ripeness of the commercial Hadoop market, something tells me they won’t have the market to themselves for long.
- The Incredible Growing Commercial Hadoop Market
- How Regulated Industries Can Move Toward the Cloud
- Report: NoSQL Databases – Providing Extreme Scale and Flexibility