Unlock new insights from machine-generated data in real-time

Brian Taylor
Updated

 Image: Splunk

splunk-010615.jpg

"Machine-generated data," said Splunk VP of Business Analytics Tapan Bhatt in a recent Q&A, "is one of the fastest growing and complex areas of big data. It's also one of the most valuable, containing a definitive record of all user transactions, customer behavior, machine behavior, security threats, fraudulent activity, and more."

Machine data is growing exponentially. Bhatt remarked that the "question then is what to do with this data; how to mine it effectively and in a timely manner to deliver new insights for the business that were never possible before." He added that what "IT leaders should explore is how to combine new sources of data with existing sources to discover new insights that prevent cybersecurity attacks, reduce fraud, or improve customer experience."

Based in San Francisco, Splunk is a leading provider of operational intelligence software used to monitor, report, and analyze real-time machine data. "Operational Intelligence," said Bhatt, "turns machine data into valuable insights no matter what business you're in."

Operational intelligence (OI) is defined on Wikipedia as a "category of real-time dynamic, business analytics that delivers visibility and insight into data, streaming events and business operations. Operational Intelligence solutions run queries against streaming data feeds and event data to deliver real-time analytic results."

In this email Q&A with TechRepublic, Bhatt also discussed trends in big data analytics, OI capabilities in the enterprise, and the use of OI in cybersecurity.

TechRepublic: What changes in machine-generated data do IT decision makers need to be looking for over the next several years?

Tapan Bhatt: Machine data is going to explode at an exponential rate, which is a change that IT decision makers should take as given. The question then is what to do with this data; how to mine it effectively and in a timely manner to deliver new insights for the business that were never possible before. The other change that IT leaders should explore is how to combine new sources of data with existing sources to discover new insights that prevent cybersecurity attacks, reduce fraud, or improve customer experience.

IT needs to reinvent itself as an organization that is innovative and central to a business gaining a competitive advantage.

TechRepublic: What are the most significant trends in your competitive space?

Tapan Bhatt: The rapid pace of digitization -- online, mobile, cloud, connected devices -- is fundamentally transforming the need for data analytics. Business needs rely on new data sources, such as machine-generated data vs. traditional structured data. Insight into such data in real time is what gives companies the competitive edge they need to succeed.

bhatttapan-05.jpg
bhatttapan-05.jpg

Splunk VP Tapan Bhatt

 Image courtesy of Splunk

bhatttapan-05.jpg

The nexus of cheap storage and computing is enabling organizations to harness insights from data that were never possible before. It is also democratizing access to data -- enabling more users across the organization to gain valuable insights from it.

Traditional approaches, however, are hindering organizations to unlock insights from data. While traditional approaches are great for uncovering what's known, they fail to discover new insights. Another shortcoming is around latency, too much time lapses between data and insight. Lastly, since most of the data today is unstructured machine data and is generated outside an organization, it is not easy for them to capture it and make it available for analysis readily and easily.

Modern and unique approaches to analysis are enabling users to ask any question of the data and do analysis at the speed of their thought, rather than having to think of all the questions upfront. Insights delivered an hour late are missed opportunities and so more and more organizations are looking to gain insights from their data in real time to be more nimble, agile, and responsive.

TechRepublic: What are the key elements of OI capability for an enterprise?

Tapan Bhatt: Operational Intelligence gives you a real-time understanding of what's happening across your IT systems and technology infrastructure so you can make informed decisions. It covers four key areas:

  • Search + Investigation: Find and fix problems, correlate events across multiple data sources, and automatically detect patterns across massive sets of data.

  • Proactive Monitoring: Monitor systems in real time to identify issues, problems, and attacks before they impact your customers, services, and revenue.

  • Operational Visibility: Gain end-to-end visibility across your operations and break down silos across your infrastructure.

  • Business Insights: Make better-informed business decisions by understanding trends, patterns, and gaining operational intelligence from machine data.

TechRepublic: Given the heightened concerns over security from the news in 2014, how can an OI approach assist an organization with cybersecurity?

Tapan Bhatt: In today's connected world, all data is security-relevant. Pre-filtering data or limiting security insights to a pre-defined set of correlations is a recipe for disaster.

An Operational Intelligence approach is the foundation for big data security solutions that can adapt to advanced threats and changing business demands. Simple monitoring of traditional security events is no longer enough. Security practitioners need broader insights from new data sources generated at massive scale across IT, the business, and in the cloud. Operational Intelligence can assist in staying ahead of external attacks, malicious insiders, and costly fraud demands, continuous security and compliance monitoring, fast incident response, and the ability to detect and respond to known, unknown, and advanced threats.

TechRepublic: How would you define the Splunk approach to OI? What differentiates your technology?

Tapan Bhatt: Operational Intelligence focuses on machine-generated data in real time. Splunk customers are able to examine their data as it's produced to discover things you never thought were possible. With a lot of relational analytics platforms, you have to define what you're looking for beforehand and specifically know the kind of answer you want back from the data before you ask the question. The problem here is you don't always know the exact question. So Splunk's take on Operational Intelligence is having the ability to explore this machine data in real time -- and ask any question you need to ask across use cases, such as IT Operations, Security, Business Analytics, and Informatics.

From a technology standpoint, here are some key differentiators:

  • Focus on machine-generated big data: Massive volume, high velocity, and diversity.

  • Universal indexing: The ability to ingest and index any kind of machine data regardless of volume, variety, or velocity.

  • Search: Exploratory analytics using search as the core means to gain new insights from data.

  • Schema at Read: Rather than relying to pre-defined schema, creating schema at read -- this allows users to ask any question of the data.

  • Real-time: Insights into data in real-time -- eliminating the latency between data generation and data insight.

TechRepublic: Let's say I am the CEO of a technology company with several hundred employees. Once I implement Splunk solutions, what can my team accomplish that was not possible before?

Tapan Bhatt: Upon implementation of Splunk, multiple teams across your company will be gaining and delivering insights across a variety of use cases:

  • Your application developers will be able to develop and rollout new application and services across mobile and cloud faster.

  • Your application managers will be able to improve the uptime and performance on your online and mobile application -- leading to a better customer experience.

  • Your security team will be able to improve overall security posture and ensure you are better protecting your customers and your business against cybersecurity threats.

  • Your fraud teams will be able to identify and mitigate fraud -- potentially saving the company millions of dollars.

  • Your customer experience team will able to look at customer interactions across channels to drive higher conversions and drive revenues.

  • Your Advanced Technology teams will be able to mine data from new connected sensors and devices.

TechRepublic: As an add-on to that question -- how can Splunk improve my company's security posture, and how can I measure ROI from the implementation?

Tapan Bhatt: IT security teams need a fast way to search through massive amounts of their machine data to turn it into useful reports and visualizations, and to identify threats / measure risk. Correlating this data and detecting threats in real-time requires analytics, so Splunk is ideal for this use case.

We have over 2,500 security customers for a wide range of use cases, including log management, incident investigation and response, forensics, security and compliance reporting, fraud detection, and real-time detection of known and unknown threats. From an ROI perspective, a breach can cost a company millions of dollars, so the return from just security alone is sizable.

Also read

Note: TechRepublic and ZDNet are CBS Interactive properties.