TransCall: Catching Brand-New Malware by Watching How Programs Behave

By Nuruzzaman Faruqui

·

Modern malware disguises itself so well that scanning for known signatures often fails. TransCall instead watches what a program does, the sequence of requests it makes to the operating system, to spot malicious behaviour, even from malware families it has never seen.

0.9286F1-score on unseen malware families
0.8743AUC in standard tests
TransformerAI architecture

The problem

Polymorphism, obfuscation and environment-aware evasion have weakened traditional signature-based detection. Behaviour, revealed through system call sequences, shows a program’s intent, but earlier machine learning and RNN methods struggle with long sequences and new threats.

What the research did

We built TransCall, a streamlined Transformer framework that uses multi-head self-attention to capture relationships across long streams of system calls. It is designed for real-time use in endpoint security.

What we found

TransCall outperformed Random Forest, LSTM and GRU baselines. In a standard test it reached an F1-score of 0.8571 and AUC of 0.8743. In a tough zero-day test, where whole malware families were left out of training, it achieved an F1-score of 0.9286. Visualising attention weights also showed which system call patterns signalled harm.

Who benefits

  • Endpoint security vendors
  • Enterprises facing novel malware
  • Security researchers

Limits and what comes next

Tested on UNM-style system call datasets; modern operating systems and real-world software mixes are more varied.

The team

This research was led by Abdullah Al Siam with Moutaz Alazab, Areej Obeidat, me and Somaya Ali Al-Maadeed (Qatar University).

Frequently asked questions

What are system calls?

Requests a program makes to the operating system, such as opening files or connecting to the network.

What is zero-day malware?

New malware that security tools have not seen before.

How well does TransCall detect it?

It reached an F1-score of 0.9286 on malware families excluded from training.

About the author

Nuruzzaman Faruqui is an award-winning AI educator and researcher who has taught artificial intelligence and machine learning at university level since 2018. A two-time winner of the Excellence in Teaching Award (2023, 2024) and winner of the Best Researcher Award (2024) at Daffodil International University, he is now a doctoral researcher in AI at Monash University, Australia. He has published 53 papers and six books, and writes these articles to make AI research easy for everyone to understand. His teaching · About Zaman · Get in touch

Keep reading