Alternative VPN Protocols: Exploring the Options

When it comes to VPN protocols, the ideal choice should be secure, functional, and fast. However, there’s another important factor: popularity. Less popular protocols are harder to implement and maintain, as their software needs to be installed and configured, and both users and administrators require training. Sometimes, protocols become popular despite technical shortcomings, simply due to aggressive promotion by large companies. Other times, an independent developer’s protocol solves a pressing problem for a specific group of users and quickly gains traction, as happened with OpenVPN and WireGuard.

Some protocols lose popularity over time, while others never become widely known—sometimes deservedly, sometimes not. In this article, we’ll discuss several such protocols.

PPTP

The PPTP (Point to Point Tunneling Protocol) has rightfully faded into obscurity. Hopefully, younger readers have never encountered it, but about a decade ago, it was a textbook example of an undeservedly popular protocol.

Its popularity was driven by Microsoft’s monopoly. From the mid-90s to the late 2000s, the vast majority of client devices were Windows computers. The presence of a built-in client in Windows automatically made PPTP at least widespread.

Microsoft, true to form, used this to maintain and strengthen its monopoly. PPTP used standard PPP and GRE for data transmission, but for authentication and encryption, it relied on proprietary protocols: MPPE (Microsoft Point-to-Point Encryption) and MS-CHAP.

Because of this, open-source implementations of both PPTP clients and servers were as problematic as GIF and MP3 once were. Eventually, the patents expired, and poptop for Linux and MPD for FreeBSD became popular alternatives to proprietary products.

However, warnings about the security issues of homegrown cryptography were not unfounded. The strength of MPPE and MS-CHAP was repeatedly downgraded, and in 2012, researchers proved that MS-CHAP-v2 was no stronger than DES. After that, PPTP could no longer be considered secure and quickly lost its remaining popularity.

Should you use PPTP?
Definitely not recommended.

SSTP

SSTP (Secure Socket Tunneling Protocol) was Microsoft’s second attempt at creating its own VPN protocol. This time, they didn’t invent their own cryptographic algorithms but used standard SSL/TLS. They also no longer prevent the creation of open-source implementations.

SSTP is essentially PPP over HTTPS. Its obvious advantage is that it works well through NAT and, theoretically, even through proxies. However, this isn’t unique—OpenVPN has supported running over TCP/443 for a long time.

OpenVPN, by default, uses UDP instead of TCP for a reason. Tunnels over TCP can have serious performance issues—on the same hardware, they can be many times slower.

Windows has a built-in SSTP client starting with Windows Vista. For Linux, there are client implementations and NetworkManager plugins. There are also third-party clients for macOS, such as EasySSTP. For mobile devices, you’ll need to find and install third-party apps.

If you need to deploy an SSTP server, open-source projects like ACCEL-PPP and SoftEther support it.

Should you use SSTP?
Only if required by corporate policy.

SoftEther

SoftEther is a multi-protocol VPN server, similar to MPD or ACCEL-PPP. It supports L2TP/IPsec, PPTP, SSTP, OpenVPN, and its own proprietary SoftEther protocol. It’s a relatively young project, with its first version released in 2014.

The SoftEther protocol is essentially Ethernet over HTTPS. Since standard SSL handles encryption and authentication, security is not a major concern.

The developers claim performance ten times higher than OpenVPN. That’s hard to believe, but I haven’t had the opportunity to verify their claims. The client is available only for Linux and Windows, so on other platforms, you’ll need to use different protocols.

Should you use SoftEther?
If the developers’ performance claims are true, it might be worth considering.

OpenConnect

The term “SSL VPN” without context is common but meaningless. “Supports SSL VPN” could mean SSTP, OpenVPN, or a variety of incompatible proprietary protocols.

Almost every vendor has its own protocol, such as Cisco AnyConnect, Juniper Pulse Connect, or Palo Alto GlobalProtect. If an organization widely uses a client for such a protocol, switching VPN concentrator hardware can be very difficult—which is exactly what vendors want.

The open-source project OpenConnect provides server and client implementations for Cisco, Juniper, and Palo Alto protocols. The OpenConnect client works on Windows and many UNIX-like systems: not just Linux and macOS, but also BSD family systems and even Solaris.

The OCServ server can save organizations significant money, as proprietary implementations often license these protocols per user.

Should you use OpenConnect?
If your organization is stuck with one of these protocols and isn’t happy about it—absolutely. Since none of these protocols are patent-protected (and there’s not much to patent anyway), the only real risk to the project is trademark lawsuits. The project name doesn’t use any registered trademarks, so the risk is minimal. Moreover, the project has existed since 2009, and no vendor has sued the developers yet.

IPsec Variations

IPsec seems like the most standardized protocol of all, supported by all network equipment vendors. But a standardized protocol doesn’t lock users into a vendor, so proprietary IPsec variations are regularly invented.

Sometimes, they solve real problems that are hard to address with pure IPsec. For example, Cisco GETVPN (Group Encrypted Transport) simplifies deploying a secure network for MPLS users, since MPLS itself offers no traffic interception protection.

In other cases, like EZVPN, vendors try to attract users with easier setup compared to “normal” IPsec.

Should you use proprietary IPsec variations?
Only if you’re not worried about being locked into a single vendor. With EZVPN, for example, some devices support only the server, others only the client, so your options may be further limited by specific models.

Client IPsec

Speaking of IPsec, it’s usually used for fixed site-to-site tunnels or as a secure transport for another protocol like L2TP. The old IKEv1 protocol was indeed poorly suited for client connections. However, modern IKEv2 handles this much better. In fact, built-in support for this type of tunnel is present in all systems, including Windows, macOS, and mobile devices.

There are also no issues with open-source server implementations—StrongSWAN, for example, officially supports client connections.

Should you use client IPsec?
If you’re setting up a server from scratch and want built-in client support in all major operating systems, you should at least consider this option alongside L2TP/IPsec.

Tinc

Most VPN protocols are designed for point-to-point or star topologies. Mesh networks are still quite an exotic scenario. Nevertheless, protocols for these purposes exist and are evolving. The Tinc project has been in development since 1998, making it older than OpenVPN, which released its first version in 2001. Tinc supports Windows and all UNIX-like OSes, but there are no mobile versions.

The main feature is automatic mesh network creation. Even if the network has many nodes, traffic between them is transmitted directly, not through a central server. This could make Tinc a working alternative to Dynamic Multi-Point VPN and the aforementioned GETVPN for corporate networks—if only network equipment vendors and popular open-source network OSes supported it.

Should you use Tinc?
At the very least, it’s worth experimenting with—it’s definitely interesting.

Conclusion

There are countless VPN protocols in the world. Even if you prefer to use only the most popular ones, knowing about the alternatives is useful—it will help you make a more informed choice.