Elastic unveils Streams & DiskBBQ to transform AI log & data search

Elastic has launched two products aimed at improving how engineering and AI teams search, analyse, and manage large data sets. The company's releases, Streams and DiskBBQ, are designed to address challenges in handling logs and enabling efficient vector search at scale.

AI log management

Streams is an AI-powered solution built to automate the processing and analysis of logs for Site Reliability Engineers (SREs). Streams automatically partitions and extracts key information from unstructured logs and identifies critical issues such as errors and anomalies. This capability is intended to facilitate quicker incident investigation and resolution without demanding extensive manual effort from engineers.

The product addresses an industry-wide issue where SREs often sift through dashboards and alerts that pin down what and where failures occur but leave the why unclear. Traditional methods leave engineers with either the option of building complex custom pipelines or risking gaps in visibility by ignoring less-structured logs.

By integrating into the Elasticsearch platform, Streams combines AI parsing with adaptive handling for new log formats. The system flags errors such as out-of-memory events and internal failures, providing actionable points for investigation before service disruptions escalate.

"For too long, SREs have been forced to treat logs as a noisy, expensive last resort for investigations. Teams hunt through dashboards for what is broken, while the actual why is buried. Streams make logs your most valuable asset. It automatically finds the signal in the noise, surfacing critical events from any log source. This gives SREs time back, allowing them to move from symptom to solution in minutes," said Ken Exner, Chief Product Officer, Elastic.

Streams is available in Elasticsearch version 9.2 and as a serverless option. It is designed to ingest logs from any source, highlight significant events for prioritised response, and manage data in a way that aims to reduce both complexity and cost of operations.

Vector search enhancement

Elastic has also introduced DiskBBQ, a new vector search algorithm for use in Elasticsearch. DiskBBQ is presented as an alternative to commonly used vector search techniques such as Hierarchical Navigable Small Worlds (HNSW). HNSW relies on keeping all vector data in memory, which can become costly as data sizes grow.

DiskBBQ compresses and clusters vectors, allowing data to be retrieved from disk rather than memory, which helps to lower RAM usage and maintain predictable performance. The company asserts that this reduces infrastructure costs while supporting large-scale vector search operations.

Ajay Nair, General Manager, Platform at Elastic, stated, "As AI applications scale, traditional vector storage formats force them to choose between slow indexing or significant infrastructure costs required to overcome memory limitations. DiskBBQ is a smarter, more scalable approach to high-performance vector search on very large datasets that accelerates both indexing and retrieval."

Elastic reports that benchmarking has shown DiskBBQ achieves stable search latencies, around 15 milliseconds, using as little as 100 MB of memory. The algorithm's performance scales with additional memory but avoids the latency spikes typical of in-memory approaches. DiskBBQ allows Elasticsearch to index and search much larger volumes of data, restricted primarily by processor and disk capacities, rather than memory.

The solution is available in technical preview on Elasticsearch Serverless, offering customers the option to evaluate the technology for large-scale search requirements.