Log management cost for 100 GB/day

Log management cost variation at the same volume reflects fundamental architectural choices each vendor has made for log search. The choices are largely invisible to the customer until the invoice arrives, but they explain the 50x cost spread between cheapest and most expensive at 100 GB per day.

Splunk pioneered the full-text inverted-index approach to log search in the early 2000s, and the architectural choice has defined the platform's pricing ever since. Every log line is parsed at index time, every word becomes an entry in the inverted index, and search queries run against the index rather than the raw log data. The trade-off is fast free-text search at the cost of expensive index storage. Splunk's legacy per-GB ingest pricing of $150 plus per gigabyte per day was justified by this architectural choice. Workload pricing has narrowed the cost gap but the underlying architecture remains.

Datadog inherited a similar architectural model for log indexing and exposed it through separate billing meters: ingest at $0.10 per gigabyte (cheap, because no indexing happens) plus indexing at $1.70 per million events (expensive, because the inverted index is built and maintained). Most customers want logs searchable, so the combined cost is what matters; the separate meters create the impression of cheap log management until the indexing line item appears on the invoice.

Loki took the opposite architectural choice in 2018. Rather than indexing log content, Loki indexes only labels (the metadata attached to each log stream: service name, host, severity). Search queries are scoped by label first, then scan the raw log data within the matching streams. This trades search performance (slower for free-text queries on large datasets) for storage cost (5 to 10 times cheaper than full-text indexing). For operational logs where queries are typically scoped by service or host, the trade-off is favourable.

New Relic uses a hybrid approach with a single ingest meter for all telemetry types. Logs benefit from automatic parsing into structured fields without explicit per-event indexing charges, which makes the cost predictable and competitive with Loki at typical mid-market volumes. The trade-off is less control over which logs are searchable and which are archived.

Three log pipeline products dominate the source-side log reduction market. Cribl Stream, the most established commercial offering, transforms log data at the agent before it reaches any observability backend. Cribl reduces ingestion by 30 to 60 percent on most workloads through a combination of filtering, sampling, redaction, and routing. The Cribl pipeline costs roughly $2 to $5 per gigabyte processed at typical pricing, which means a 100 GB per day deployment pays Cribl $6,000 to $15,000 per month for the pipeline. The Cribl cost makes economic sense only when the downstream backend savings exceed the pipeline cost; for Datadog or Splunk customers this is usually true with substantial margin.

Splunk Edge Processor is Splunk's native equivalent, introduced in 2023. It performs the same source-side transformations as Cribl but is bundled into Splunk subscriptions. For Splunk-only customers, Edge Processor is the obvious choice; the additional integration value is real. Customers running multi-vendor pipelines (Splunk plus Datadog plus Loki) tend to prefer Cribl because it is vendor-agnostic.

Vector, the open-source alternative from Datadog (acquired Timber.io in 2021), runs as a self-hosted log shipper with similar transformation capabilities to Cribl. Vector is genuinely free at the licence level but requires platform engineering capacity to operate at scale. For teams comfortable with self-hosted observability infrastructure, Vector is the cost-leading option for log pipeline transformation. For teams that want managed pipeline infrastructure, Cribl or Edge Processor are easier paths.

The economic case for pipeline tooling is strongest for Datadog customers with heavy log indexing and Splunk customers with legacy ingest pricing. For Loki and New Relic customers at 100 GB per day, the savings from pipeline transformation rarely exceed the pipeline cost; the underlying log management is already cheap enough that source-side reduction is not worth the additional moving part.