How Antivirus Software Works: Malware Detection Methods

Debates about whether antivirus software is necessary or completely useless have been ongoing since the very first antivirus applications appeared. Just as long-lasting is the never-ending battle between malware creators and security software developers: one side constantly invents new detection algorithms, while the other tries to bypass them at all costs. So, how do modern antivirus programs work, and what methods do cybercriminals use to fight against them? That’s what this article is about.

How Do Antivirus Companies Update Their Databases?

In the context of modern security technologies, the term “antivirus” is more of a legacy than an accurate description. Classic file-infecting viruses—malicious programs that can infect executable files or dynamic libraries and spread without user involvement—are now extremely rare. The vast majority of malware found “in the wild” today are Trojans, which cannot infect files or self-replicate. Worms are encountered less often; these programs can copy themselves to removable drives or network disks, spread via networks or email, but cannot infect files. All other traditional malware categories differ mainly in their feature sets, but architecturally, they can be grouped into these three types.

How do malware samples end up in virus labs? Antivirus companies traditionally have several channels for receiving new samples. The first is online services like VirusTotal, where any anonymous user can check a file against dozens of popular antivirus engines. Every uploaded sample, regardless of the scan result, is automatically sent to vendors for further analysis.

Obviously, these resources send a huge amount of junk to virus labs, including harmless text files and images, so incoming files are filtered by specialized bots before being processed further. Smaller companies also use these services to save on maintaining their own virus labs. They simply copy detections from others, which sometimes leads to epic failures—like when a vendor jokingly or mistakenly flags a component of such an antivirus as infected, causing it to quarantine its own library and crash, much to the amusement of competitors and frustration of users.

The second channel is direct submissions: suspicious files that users send to the virus lab via the antivirus company’s website, support requests, or by uploading from quarantine. The third channel is honeypots—virtual servers with open ports and default logins like root/root, set up as bait for malware authors, who then upload their creations, thinking the admins are incompetent. The fourth method is database sharing between vendors, but due to increased competition and a shrinking market, such cooperation has nearly disappeared in recent years.

Once a sample reaches the virus lab, it’s sorted by file type and analyzed by automated tools that can determine a verdict based on formal or technical characteristics, such as the packer used. Only if the automated systems can’t identify the malware is it passed on to virus analysts for manual or instrumental analysis.

Anatomy of Antivirus Software

Antivirus programs from different vendors include varying numbers of components, and even the same company may release several versions with different modules for different market segments. For example, some antivirus solutions include parental controls to restrict minors’ access to certain websites or limit their computer usage time, while others do not. Typically, modern antivirus applications include the following functional modules:

• Antivirus scanner – A utility that searches for malware on disks and in device memory, either on demand or on a schedule.
• Resident monitor – Monitors the system in real time and blocks attempts to download or launch malicious programs on the protected computer.
• Firewall – Monitors current connections, analyzes incoming and outgoing traffic, and checks the source and destination addresses of every data packet. It acts as a filter, controlling the flow of information between the local computer and the internet, serving as a protective barrier.
• Web antivirus – Prevents access to dangerous resources that distribute malware, phishing, or scam sites, using a special database or rating system.
• Email antivirus – Checks email attachments and/or links for safety.
• Anti-rootkit module – Designed to combat rootkits (malware that hides its presence in the system).
• Proactive protection module – Ensures the integrity of critical system data and prevents dangerous program actions.
• Update module – Keeps antivirus modules and virus databases up to date.
• Quarantine – A secure, centralized storage for suspicious (or definitely infected) files and applications until a final verdict is reached.

Depending on the version and purpose, antivirus software may also include other modules, such as centralized administration or remote management tools.

Signature-Based Detection

Modern antivirus programs use several detection methods, often in combination. The primary method is signature-based detection.

This method relies on creating “signatures”—unique digital identifiers for files, consisting of a specific set of bytes derived from the file’s content. A signature is like a file’s fingerprint: it allows for unambiguous identification. File hashes, such as SHA-1 or SHA-256, work similarly, using one-way mathematical functions to generate a unique hexadecimal string from a file’s contents. However, a virus signature is more complex and includes additional unique file characteristics.

Signatures are collected into a data block called a virus database. These databases are regularly updated to include signatures of new threats discovered since the last update.

The antivirus scans files stored on disks (or downloaded from the internet) and compares them to the signatures in its database. If there’s a match, the file is considered malicious. This method has a significant drawback: if an attacker changes even a few bytes in a file, its signature changes. Until the new sample is analyzed and its signature added to the database, the antivirus cannot detect or neutralize the threat.

Behavioral Analysis

In addition to signature detection, most modern antivirus programs use behavioral analysis mechanisms. Behavioral analysis is a type of probabilistic analysis—antivirus software monitors application behavior and, if it seems suspicious, blocks the potentially dangerous program.

One safe way to analyze application behavior is to run it in a “sandbox”—a protected, isolated virtual container that prevents access to the OS and file system. If the program’s behavior raises suspicion (for example, it injects code, modifies the boot record, or changes executable file structures), it may be flagged as potentially dangerous or malicious.

Heuristic Analysis

Heuristic analysis is another type of probabilistic malware analysis, based on logical algorithms that can identify and neutralize potentially dangerous applications. Heuristic analysis helps when threats can’t be detected by signatures.

In simple terms, each function a program can perform in the OS is assigned a “danger rating.” Some actions are considered less risky, others more so. If the sum of a program’s actions exceeds a certain “safety threshold,” it’s flagged as potentially malicious.

For example, if a program runs in the background, has no graphical interface, polls remote servers, and then tries to download and launch something, it’s likely a Trojan downloader—or maybe just a Chrome updater. This highlights the main weakness of heuristic analysis: a high chance of false positives.

Another heuristic method is code emulation. The antivirus loads the suspicious application into its own buffer, breaks down the code into instructions, and executes them sequentially, checking the results.

Heuristic analysis aims to detect threats unknown to the antivirus—those whose signatures aren’t yet in the database. However, even if a new threat is detected, it can’t always be “cured” immediately. Often, users must wait for a database update with specific removal algorithms for that malware.

Proactive Protection (HIPS)

Proactive antivirus protection (HIPS—Host-based Intrusion Prevention System) is also a form of behavior-based defense. The antivirus monitors running applications and informs the user about their actions. The user decides whether to allow or block a program’s actions—this is the classic HIPS model. There’s also an expert mode, where the antivirus automatically blocks certain actions based on built-in rules and permissions. Users can add programs to an exclusion list, allowing all or selected actions in the protected system.

Methods for Evading Antivirus Detection

Unfortunately, the battle between malware authors and antivirus developers is ongoing: the former constantly invent new ways to bypass protection, while the latter improve detection algorithms. Here are the main techniques used by malware creators:

Repacking

The most common method for bypassing signature detection. As mentioned, a signature is like a file’s fingerprint and is unique to each file. If even minor changes are made, the antivirus can’t recognize it by signature, and the file won’t be detected until it reaches a research lab.

The simplest way to change a file’s structure without altering its functionality is to use a software packer. Packers compress the application’s contents and add code needed for unpacking and execution. Some also include encryption features, making analysis harder. Attackers exploit this by repeatedly repacking files, changing their signatures each time and making them “invisible” to signature-based detection. Some malware authors pack and encrypt their creations in multiple layers, creating a “matryoshka doll” effect that’s difficult to analyze.

Sometimes, cybercriminals use another method: a server hosting the malware runs a script that, when triggered (e.g., by a user clicking a link), fetches the binary, repacks it on the fly, and only then delivers it. This way, each victim receives a unique, undetectable copy.

Obfuscation

Obfuscation (from “obfuscate”—to confuse or make unclear) is the deliberate complication of malware code while preserving its functionality, making analysis more difficult. Malware authors may add junk code, unnecessary instructions, multiple jumps, and repeated function calls. There are special tools called obfuscators for this purpose.

Obfuscation hinders reverse engineering—decompiling and studying the malware’s code—but also makes debugging harder for the malware author and can increase file size and reduce performance.

Anti-Debugging

Most modern malware includes strong anti-debugging mechanisms to prevent analysis. Some viruses and Trojans check at startup whether they’re running in a sandbox, under a debugger, or in a virtual machine. This can be done by checking running process names or searching for specific strings in window headers. If the malware detects a virtual environment or debugger, it shuts down.

Similarly, many malware samples look for installed or running antivirus programs and try to terminate them; if unsuccessful, they may exit themselves. Some go further: for example, the Trojan.VkBase.73 changed Windows boot parameters, installed a service that deleted antivirus software in Safe Mode, and then placed a fake antivirus icon in the system tray. The user wouldn’t realize their antivirus was gone. After successful removal, a message would appear (in Russian or English, depending on the system) stating: “Attention! [Antivirus name] is running in enhanced protection mode. This is a temporary measure required for immediate response to virus threats. No action is required from you.” This message was meant to prevent users from worrying when the antivirus icon became unresponsive.

To bypass sandboxes, some malware includes “delaying mechanisms” that slow down or “sleep” their malicious functions, activating them only after a certain time. This can fool security software into thinking the program is safe. For example, a modern Trojan might create a file in a temp folder, write one byte to it a million times, then read from it a million times. These harmless, lengthy cycles can cause behavioral analysis to finish before the Trojan starts its real work.

Conclusion

Malware detection algorithms, as well as methods for bypassing antivirus protection, are constantly evolving. However, truly unique evasion techniques are rare—malware authors usually rely on standard, time-tested methods. Security software developers, well aware of these tricks, often stay a step ahead. That’s why antivirus protection remains an effective way to combat malicious and potentially dangerous programs—especially for inexperienced users.