Published 2024-06-29
Keywords
- AI-driven security, cloud incident detection, AWS CloudWatch, VPC Flow Logs, machine learning, XGBoost, anomaly detection, cloud-native monitoring, SageMaker, cybersecurity automation, explainable AI, network anomaly analytics, autonomous threat detection, serverless architecture, real-time cloud defense.
Abstract
The present study investigates the development and evaluation of an AI-driven incident detection framework leveraging AWS CloudWatch metrics and VPC Flow Logs for autonomous, real-time cloud security monitoring. Traditional rule-based monitoring systems in AWS, such as static CloudWatch alarms and signature-dependent GuardDuty alerts, are limited in detecting sophisticated or evolving threats. This research addresses that gap by implementing a machine learning–based detection pipeline within the AWS ecosystem, utilizing SageMaker, Lambda, and SNS to enable scalable and adaptive threat identification.
A hybrid dataset of over 2.8 million records combining system and network telemetry was collected under both normal and simulated attack conditions, including DDoS floods, SSH brute-force attempts, port scanning, and data exfiltration. After rigorous preprocessing and feature engineering, four AI models Isolation Forest, Deep Autoencoder, Random Forest, and XGBoost were trained and benchmarked against traditional baselines. Among them, XGBoost achieved the highest performance with 98.3% accuracy, 0.96 F1-score, and an average detection latency of 2.1 seconds, outperforming CloudWatch and GuardDuty by significant margins. The false positive rate was reduced by over 75%, while detection reliability and adaptability improved substantially.
Feature importance analysis using SHAP interpretability revealed that traffic volume, flow duration, and destination entropy were dominant predictors of anomalies, providing transparency and analyst trust in AI-driven decisions. The system achieved operational scalability at an average cost of USD 120 per month, proving its economic viability for enterprises.
The findings confirm that integrating AI models with AWS-native observability services enables proactive, interpretable, and cost-efficient incident detection, marking a paradigm shift toward autonomous cloud security operations. The study establishes a replicable blueprint for AI-augmented cloud defense systems capable of learning and adapting to dynamic threat landscapes in real time.