Log management cost for 1 TB/day

The single most consequential cost-management practice at 1 TB per day log scale is auditing log volume by source and severity, then filtering aggressively at the source before logs reach the observability backend. The audit typically reveals two structural facts that drive the optimisation. First, 5 to 15 percent of log sources produce 60 to 80 percent of total volume; a small number of services, applications, or infrastructure components dominate the bill. Second, DEBUG and INFO severity logs contribute 70 to 90 percent of total volume but provide marginal incremental value for incident response or operational analysis.

Once the audit is done, the optimisation is straightforward. Drop DEBUG and INFO logs at the application or log shipper. Sample high-volume access logs at 10 to 25 percent. Route audit-only logs (compliance retention with no operational use) to S3 or equivalent object storage rather than the observability backend. Implement these three transformations consistently across the top 10 to 20 log sources, and 1 TB per day typically becomes 250 to 500 GB per day with no compromise to operational visibility.

The economic impact at 1 TB per day scale is dramatic. Reducing volume from 1 TB per day to 350 GB per day (a 65 percent reduction) cuts the Datadog full-indexing bill from $200,000 per month to $70,000 per month, the Splunk workload bill from $60,000 per month to $25,000 per month, and the Loki bill from $13,000 per month to $5,000 per month. The annual savings at the Datadog scale alone are $1.5 million; at the Splunk scale, $400,000. The audit-and-filter exercise typically pays back in the first month and continues to compound across the contract term.

The most common failure mode is shipping the audit but not following through on the filtering implementation. Auditing produces a clear picture of where the volume comes from; following through requires coordination with the application teams that own the logging configurations, which often takes longer than expected. Plan for 2 to 4 quarters of structured rollout across the top 10 to 20 log sources, with quarterly progress measured in volume reduction and dollar savings.

Many enterprises at 1 TB per day log scale operate two distinct log management platforms in production: a cheap operational log platform (Loki, self-hosted or hosted via Grafana Cloud) for application and infrastructure logs, and a mature SIEM platform (Splunk Enterprise Security, less commonly Elastic Security or Microsoft Sentinel) for security and compliance data. The split is economically efficient and operationally clean.

The economic case is straightforward. Operational logs are typically queried by label (which service, which host, which severity) rather than by free-text content. Loki's label-indexed approach is structurally cheaper for this access pattern; the search latency trade-off is acceptable for operational debugging. Security logs are queried by free-text content (which IPs accessed which resources, which user-agents triggered which patterns, which file hashes were observed) where Splunk's full-text indexing is operationally superior despite the cost premium. Running both platforms allocates the right tool to the right workload.

The operational case is equally important. Operational logs are owned by the SRE or platform engineering team, queried during incident response, and retained for 30 to 90 days. Security logs are owned by the security operations team, queried during threat hunting and compliance investigation, and retained for 12 to 36 months for regulatory compliance. The two teams have different query patterns, different retention requirements, and different operational rhythms. Running them on the same platform forces compromises that often serve neither well.

The cost arithmetic for the split-platform strategy at 1 TB per day total volume typically looks like this. Operational logs contribute 80 to 90 percent of total volume (800 to 900 GB per day) and run on Loki at $9,000 to $13,000 per month. Security logs contribute 10 to 20 percent of total volume (100 to 200 GB per day) and run on Splunk Enterprise Security at $20,000 to $40,000 per month. Total combined cost is $29,000 to $53,000 per month, meaningfully cheaper than either Datadog full-indexing or Splunk-everything at the same total volume.