Encrypt Smartly! Why Messengers Can’t Fully Protect the Privacy of Your Chats

End-to-end encryption (E2EE) is often seen as a cure-all against hackers and law enforcement trying to access your online conversations. The idea is that encryption keys are stored only on users’ devices and never reach the server—but that’s not entirely true. Let’s take a closer look at how E2EE really works in popular messengers.

Encryption in Messengers

This article was inspired by the study “Obstacles to the Adoption of Secure Communication Tools.” The authors found that “the vast majority of survey participants do not understand the basic concept of end-to-end encryption.” In other words, people usually pick a messenger with their heart, not their head.

E2EE works differently in each messenger. In Signal, it’s almost exemplary. WhatsApp uses a similar system, except for one crucial point: if a WhatsApp user changes their main key, it doesn’t block messages from being sent to them. At best, you can enable a mostly useless notification (which is off by default). In Viber, E2EE isn’t enabled by default and only appeared in version six. In Telegram, E2EE is only used in secret chats, and even then, it’s implemented in a rather odd way.

The conflict between Russia’s Roskomnadzor and Telegram gave the latter a lot of publicity. Many users now see Telegram as a real thorn in the side of law enforcement, who supposedly can’t do anything about this “bulletproof” innovative service. Telegram fans compare it to Signal and claim it’s superior.

But in applied cryptography, there are no miracles. Many mathematically elegant ideas are ruined by their implementation, where convenience and control are prioritized over security and privacy (which is almost always the case).

From OTR to Double Ratchet

Originally, messengers used the OTR (Off-the-Record) protocol. It uses symmetric AES encryption in CTR mode, the DH key exchange protocol, and the SHA-1 hash function. AES-CTR provides “deniable” encryption, meaning you can plausibly deny authorship of a message if it’s intercepted. For example, “buy bread” could be decrypted as “poison the queen”—this is by design.

OTR authenticates users and encrypts their chats. It’s secure as long as users regularly verify each other’s public key fingerprints and guard against other attack vectors (including social engineering).

The main drawback of OTR is that after sending a new key, you have to wait for the other person to confirm it. If they’re offline, communication is temporarily impossible. The Double Ratchet (DR) algorithm, developed by Trevor Perrin and Moxie Marlinspike at Open Whisper Systems, solved this. Today, DR is used in Signal, WhatsApp, Viber, and many other messengers that support E2EE by default or as an option (like secret chats).

How End-to-End Encryption Works

E2EE uses a combination of public and private key cryptography. The general idea is simple, but the details are complex. Many interrelated keys are used, some of which inevitably end up on the server—and are uploaded before the conversation even starts, so you can begin chatting at any time.

It starts with each user generating a key pair. Public keys are exchanged openly and are used to encrypt messages and verify signatures. Private keys are kept secret and used for decryption and signing.

“Message” here means any data: text, media, or metadata exchanged with the server. Some of this data includes timestamps, client app state, and new keys.

This system works for email, where users don’t have to be online at the same time. All messages are stored on the server and downloaded after authentication. Decryption happens locally with private keys that never leave the device. Email with PGP is popular but far from perfect.

However, pure asymmetric encryption isn’t suitable for messengers, which are designed for rapid, ongoing exchanges of short messages. Messages must appear in order, and users can go offline at any time, disrupting the flow.

Encrypting many short messages with a single key is a bad idea. Hundreds or thousands of messages are created daily, many of which are predictable (like emojis or stickers) and have standard headers, making cryptanalysis easier.

Because of typical metadata, an attacker can quickly intercept a large amount of predictable ciphertext. If all of it is encrypted with one key, a successful attack could compromise all past and future messages.

To prevent this, messengers use properties like forward and backward secrecy, meaning you can’t read past or future messages with just the current key. This is achieved with layered encryption, moving from asymmetric to symmetric cryptography, and using keys with different lifespans.

Diffie-Hellman and X3DH

According to public documentation, Telegram uses the classic Diffie-Hellman (DH) protocol for authenticated key distribution. DH bridges asymmetric (RSA) and symmetric (AES) encryption, allowing multiple users to chat securely by exchanging only public keys. Session keys are generated as shared secrets based on one user’s private key and another’s public key. Ephemeral keys are authenticated with long-term public keys.

The DH channel doesn’t have to be protected from eavesdropping but must be protected from man-in-the-middle (MITM) attacks. If an attacker can alter traffic, the whole scheme falls apart.

Signal uses the X3DH (Extended Triple Diffie-Hellman) method with Curve25519 (a fast elliptic curve) or X448. X3DH also uses HMAC-SHA-256 and AES-256.

X3DH establishes a shared secret between two parties who mutually authenticate each other’s public keys. Keys are checked right after the session is established and before messages are sent, minimizing MITM risk.

X3DH uses four types of keys, three of which change regularly:

• IK (identity keys): Secret keys created once and used as the basis for all others.
• EK (ephemeral key): Used to verify the other party’s identity without revealing the real one.
• SPk (signed public key): An ephemeral key signed by the secret key, usually changed every day or week, or after a set number of messages.
• OPK (one-time public key): A one-time ephemeral key created before a session and deleted after a successful handshake.

The Legacy of Enigma

Key rotation in X3DH is handled by the Double Ratchet algorithm, which introduced the concept of a key chain or pool. If a user is offline, a pool of OPKs is created in advance and uploaded to the server. This allows the server to accept encrypted messages and authenticate the sender with a new key pair, even if the recipient is offline. If the OPK pool runs out, the server uses a backup EK.

The name “Double Ratchet” refers to the Enigma cipher machine’s gears, which prevented backward movement and reuse of old values. Digitally, DR generates new ephemeral keys for each message (or small batch), ensuring they’re always different and unpredictable.

The X3DH-based TextSecure protocol was later renamed Signal. In pure or slightly modified form, the Signal protocol is used in Signal, WhatsApp, Viber, and others. Developers may give protocols different names, but they’re essentially all X3DH with varying hash functions, random number generators, and cryptographic primitives.

The Problem with Group Chats

Group chat organization in messengers is detailed in the article “More is Less: On the End-to-End Security of Group Chats in Signal, WhatsApp, and Threema.” Here are the main findings:

Our systematic analysis shows that communication integrity (the integrity of all messages) and group membership (the ability of group members to manage them) are not end-to-end protected. We also showed that backward secrecy (a key security property) is not preserved when using the Signal protocol for group chats.

In short, authenticated key distribution via DH only works for two users forming a shared secret. In group chats, DH isn’t used, and the message exchange structure lacks key cryptographic properties.

Encryption in group chats. A — sender, B — recipient, G — group of users.

The authors show how a server controlled by an attacker can manipulate group chats due to the lack of E2EE. Their research focused on Signal and WhatsApp, but it’s unlikely other messengers have solved this elegantly.

The Oddities of Telegram

Telegram is shrouded in secrecy. There’s only partial information about the MTProto 2.0 protocol. It hasn’t been externally audited, and the open-source model is used in a heavily modified form, mainly for marketing. Here’s why:

According to official descriptions, all undelivered messages are temporarily (hopefully) stored on Telegram’s servers, which are scattered worldwide and synced in a virtual cloud. They synchronize to deliver messages in order as soon as recipients come online. Thus, encryption happens in two stages: client-server and server-server. What’s odd is that direct client-to-client connections are never used.

Even secret chats in Telegram go through servers, though a P2P connection would make more sense. It seems Telegram simply doesn’t work without constant server involvement. Other messengers use servers only at the start—to match IP addresses and set up direct connections. Telegram can’t do this, which looks a lot like MITM by design.

Discussions about MTProto 2.0’s security focus on DH’s resistance to interception. But DH is actually vulnerable to MITM attacks. In Telegram’s case, it may be further weakened at the random number generator level.

The Telegram client relies on a vague assessment of entropy. Instead of generating random numbers locally, it requests them from the server. There’s no way to know what RNG is used, how good the primes are, or whether the server selectively sends certain primes to certain clients. The client only does a simplified check, since a thorough test would be too slow for a smartphone.

Another common argument for Telegram’s security is its open source. But the server code isn’t public, and the client code is often outdated. Telegram’s repositories are updated with long delays (except maybe the web version), and only old versions are available. There’s no way to verify that the distributed binaries match the source code.

So, talking about a real audit is pointless. By the time experts finish reviewing old code, a dozen new Telegram versions will have been released. Making the whole encryption system vulnerable could be as simple as changing one byte.

Telegram’s hackathons don’t replace audits, since they prove nothing. The challenges create artificial situations where the attacker only has one encrypted message. In real life, there are many more attack vectors.

A Thousand and One Vulnerabilities

Signal is one of the few messengers whose protocol has undergone external audit. The report is extensive, but here are the main conclusions:

Our analysis shows that [the] Signal [protocol] meets standard cryptographic assumptions and security properties. We found no major flaws in its design, which is very encouraging. In real-world use, uncertainties remain. Therefore, it’s impossible to say whether [the app] Signal always achieves its stated goals.

Protocol analysis is important, but it’s only one part of security. Any messenger operates in a vulnerable environment. Usually, it runs on an outdated Android version, alongside dozens of shady apps (some of which abuse permissions or even contain trojans), and the account is tied to a phone number.

A huge vulnerability is that confirmation codes are sent via SMS. These can be intercepted using known flaws in the SS7 cellular protocol. An attacker can access all your chats without knowing encryption keys or hacking Signal/Proteus/MTProto/other secure protocols. The messenger server will change the key and decrypt your recent chats (at least undelivered messages). Even your sticker packs will be restored. But hey, it’s convenient, right?

Another gaping hole is push notifications. Without them, you won’t know you have a message unless you open the app. With them, the push notification server becomes a legalized “man in the middle.” For example, iMessage sends encryption keys to Apple’s servers for notifications. Apple then authenticates users and (at least) decrypts message headers. Restore your Apple account on another device, and you get everything back—including chats and Wi-Fi passwords. Google and Microsoft servers work similarly. Do you still believe in E2EE tied to your phone number and main smartphone account?

Poor key management and a large attack surface affect all messengers. WhatsApp, Viber, and many others allow chat backups (including cloud backups) and don’t encrypt metadata (sometimes the fact of a conversation is more important than its content). Signal is a bit better, but it’s not perfect for several reasons:

1. Signal also uses Google’s push notification service. On phones without Google services (like Chinese models for the domestic market), it simply doesn’t work.
2. For voice calls, Signal uses the closed-source RedPhone server.
3. Signal (like many others) allows you to open a parallel session on another device just by scanning a QR code.
4. At HITBSecConf2017, researchers described several conceptual problems with Signal and demonstrated a successful attack.

XMPP: An Open Alternative

As you can see, it’s hard to trust third-party or proprietary messengers, even if they’re recommended by Snowden, Assange, or the EFF. Some people set up their own open-source messenger with plugins. For simple chats, the OTR plugin works, but it doesn’t support group chats. Related protocols like mpOTR and GOTR add this feature.

For group communication, the open XMPP (Extensible Messaging and Presence Protocol, formerly Jabber) is more convenient. XMPP is fully open source. You can run your own XMPP server, be independent, and pay nothing. There are plenty of ready-made servers and clients—like Pidgin for desktop and Xabber for Android.

XMPP’s extensibility means you can send not just text, but also other data types, and add features and encryption schemes. For example, you can send voice messages, video, and files over XMPP, encrypting them with TLS or PGP. Recently, the OMEMO extension protocol was created for XMPP, using the same Double Ratchet as Signal and WhatsApp, but without their other drawbacks.

Conclusions

Modern messengers claim to support end-to-end encryption, but it’s often implemented with oddities. Their code is full of other holes, left either accidentally or intentionally—the latter is more likely, considering the money and expertise invested. I try to balance comfort and healthy paranoia. I use different messengers (whichever my contacts prefer), but never discuss truly private matters through them. There are plenty of open-source alternatives. Besides Xabber, Android users should check out Conversations—a free XMPP client supporting OTR, OMEMO, openPGP, and SOCKS5.