In the latest episode of Detection at Scale, I sat down with Ryan Glynn, Staff Security Engineer on the Detection Response Team at Compass. Ryan brings hands-on experience building custom machine learning models for security automation, having developed a phishing classification system that reduced on-call burden by 95% while processing 400+ emails daily. His background spans both traditional detection engineering and practical ML implementation, positioning him to discuss the intersection of deterministic security controls and AI-powered analysis.
Our conversation explores Ryan’s philosophy on where LLMs excel in security operations—particularly their strength in semantic understanding and intent classification—versus where traditional deterministic models remain superior. Ryan’s practical experience building custom ML models for phishing automation, combined with his evaluation of commercial AI SOC products, provides a grounded perspective on AI adoption. His emphasis on explainability, the importance of tuning at the detection layer rather than the analysis layer, and the need for human-in-the-loop validation provides concrete guidance for security teams navigating AI agents while building sustainable automation.
Topics Covered
LLMs for Semantic Understanding Over Decision-Making: Ryan argues the biggest strength of language models is natural language processing for documentation and intent classification, not making binary malicious/benign determinations, where deterministic models prove more reliable and explainable.
Using LLMs as Feature Generators for Deterministic Models: Rather than having LLMs make security decisions directly, Ryan uses them to generate binary feature flags that analyze email context (tone, product selling, aggression) and feed them into more reliable traditional ML models.
The 95% On-Call Reduction Through Custom ML: Ryan’s phishing automation model processes 400+ daily reported emails, handling classification (phishing/benign/spam), automated response (quarantine/release), and reducing analyst burden while maintaining high accuracy through company-specific training.
Agent SOC Limitations and Hallucinations: Ryan’s evaluation of commercial AI SOC products revealed gaps in which agents claim to perform analysis steps they didn’t actually execute, and make false statements such as “user never authenticated from this IP” when logs show otherwise.
Tuning at the Detection Layer, Not Analysis Layer: Why applying AI-powered allow-listing and tuning at the analysis layer across multiple detections is more dangerous than tuning individual detection rules, as blanket AI rules can inadvertently suppress legitimate alerts.
SOAR Integration for Contextual Flexibility: How language models can make SOAR workflows less brittle by handling ambiguous cases like determining if a reported email is actually a “forward of a forward” versus a legitimate report, routing appropriately for manual or automated triage.
The Challenge of Context Management: The difficulty of documenting business partner relationships, third-party integrations, and legitimate, unusual behaviors in ways that both humans and AI systems can reliably access during incident analysis.
Useful vs. Noise Alert Tagging: Why binary alert classification (useful/noise) with subcategories provides better feedback loops for AI systems than ambiguous “true positive/false positive” labels, enabling pattern matching and detection tuning over time.
The Importance of Analytical Skills for Detection Engineers: Ryan emphasizes that detection engineers need data science and analytical tool experience beyond security knowledge, recommending that everyone build at least one decision tree model to understand ML effectiveness and limitations.
Prompt Injection Risks and Documentation Poisoning: How malicious actors can manipulate LLM responses through confluence page ranking or documentation injection, similar to 1990s Google SEO spam, creating attack vectors for autonomous security systems.
The transformation Ryan describes aligns with Panther's approach to AI—leveraging language models' semantic understanding strengths for alert analysis and investigation, while maintaining deterministic detection logic and human validation. By automating the pattern matching and initial triage that LLMs excel at, security teams can focus on the custom model building, detection tuning, and strategic security decisions that Ryan emphasized as critical for reducing analyst burnout. Learn more about Panther AI and how we're building systems that combine AI efficiency with detection engineering rigor.
