DETECTING RUNTIME ATTACKS

2025-06-12 11:44:44

This project, titled “Runtime Software Attacks,” was a semester-long project focused on detecting buffer overflow attacks, such as Return-Oriented Programming (ROP). These attacks are particularly challenging because they manipulate a program’s execution without modifying the binary code itself. The central research question was: “How can we implement and detect runtime attacks that manipulate execution flow without altering the source code of the binary iteself?”.

Image shows the dashboard for the systemcall detection monitoring system.

Key Project Components

Vulnerable Binary: A 32-bit Linux binary was developed with deliberate vulnerabilities, specifically a format string vulnerability and a stack-based buffer overflow. The format string vulnerability was exploited to leak memory addresses, which in turn helped bypass a stack canary. This vulnerable binary served as the testbed for implementing and evaluating the detection frameworks.
Attack Implementation: The attacks were crafted using Pwntools. The project successfully executed two types of attacks to gain a shell:
- Shellcode injection: This was used on a version of the binary with an executable stack.
- ROP chain: This was used on a version of the binary with a non-executable stack.
Detection Frameworks: Two distinct frameworks were designed and implemented to test different detection strategies:
- System Call Anomaly Detection: This flexible approach monitored a program’s system call behavior to identify and flag unusual patterns.
- Control Flow Validation: This deterministic model enforced a strict, predefined Control Flow Graph (CFG) to catch any deviations from expected execution paths.

Image shows the differnt CFG’sproduced from normal usage, shellcode injection and ROP

Everything connected

Architecture of the whole system.

Results & Conclusions

The project concluded that both detection methods could successfully identify runtime attacks. The system call anomaly-based method was noted for its flexibility, while the deterministic CFG-based approach provided a stronger, more certain level of enforcement. The study highlights the critical role of real-time monitoring in modern cybersecurity defenses. Future work could explore the use of machine learning to enhance anomaly detection and the integration of context-specific configurations and honeypots to improve threat detection.

Demonstration

Source code

The source code is available at github.com https://github.com/bummie/aaugrader