In this episode of Detection at Scale, Dave Harreld, Global Head of Cybersecurity GTM at Databricks, shares his unique perspective on transforming security operations through advanced data analytics. With over 25 years in security spanning practitioner and vendor roles—from accidental CISO beginnings with PCI compliance to leadership positions at Splunk, Google, and now Databricks—Dave brings deep insights into how data lake architectures are reshaping the SOC landscape.
The conversation explores Databricks' vision for security analytics, emphasizing the separation of storage and compute to give organizations true data ownership while leveraging open formats like Iceberg and Delta. Dave discusses the evolution from traditional SIEM platforms to more flexible, horizontally scalable analytics engines that handle everything from SQL queries to graph analysis and AI-powered investigations. The discussion culminates in practical advice about implementing focused AI agents that augment rather than replace human analysts, addressing the reality that 80-90% of security work could be enhanced through intelligent automation.
Key Takeaways
Data Ownership Changes Everything: Databricks' separation of storage and compute allows organizations to store security data in their own S3 buckets while using advanced analytics engines, finally addressing the cost structure concerns that have plagued security teams for decades.
Focused AI Agents Outperform Monolithic Solutions: Rather than building one "super agent" to replace analysts, successful implementations use multiple focused agents for tasks like enrichment, context gathering, and initial triage—each operating within well-defined boundaries.
The 80-90% Automation Reality: Most security work involves breadth rather than depth, requiring knowledge across cloud infrastructure, attacker techniques, SIEM operations, and log analysis—areas where AI agents excel at augmenting human capabilities rather than replacing human judgment.
Hallucinations Are an Engineering Problem: Rather than viewing AI hallucinations as insurmountable obstacles, they should be treated as prompt engineering challenges that can be minimized through better context provision and more specific instructions.
Mission-Driven Career Longevity: After 30+ years in security, Dave emphasizes finding your personal mission that transcends specific companies or projects, staying current with AI developments, learning through building, and maintaining hobbies completely disconnected from technology.
