Modern malware disguises itself so well that scanning for known signatures often fails. TransCall instead watches what a program does, the sequence of requests it makes to the operating system, to spot malicious behaviour, even from malware families it has never seen.
The problem
Polymorphism, obfuscation and environment-aware evasion have weakened traditional signature-based detection. Behaviour, revealed through system call sequences, shows a program’s intent, but earlier machine learning and RNN methods struggle with long sequences and new threats.
What the research did
We built TransCall, a streamlined Transformer framework that uses multi-head self-attention to capture relationships across long streams of system calls. It is designed for real-time use in endpoint security.
What we found
TransCall outperformed Random Forest, LSTM and GRU baselines. In a standard test it reached an F1-score of 0.8571 and AUC of 0.8743. In a tough zero-day test, where whole malware families were left out of training, it achieved an F1-score of 0.9286. Visualising attention weights also showed which system call patterns signalled harm.
Who benefits
- Endpoint security vendors
- Enterprises facing novel malware
- Security researchers
Limits and what comes next
Tested on UNM-style system call datasets; modern operating systems and real-world software mixes are more varied.
The team
This research was led by Abdullah Al Siam with Moutaz Alazab, Areej Obeidat, me and Somaya Ali Al-Maadeed (Qatar University).
Frequently asked questions
What are system calls?
Requests a program makes to the operating system, such as opening files or connecting to the network.
What is zero-day malware?
New malware that security tools have not seen before.
How well does TransCall detect it?
It reached an F1-score of 0.9286 on malware families excluded from training.