What is IPSEC? - Internet Protocol Security Explained

By Fortra's Digital Defense

What is IPSEC?

In the world of VPNs, there are typically two types that an organization can choose from, IPSEC or OpenSSL.  While many people have migrated to OpenSSL mode because of its new relative ease of deployment, there are still companies that deploy IPSEC-based VPNs because of the additional layers of security they provide that are not available in OpenSSL-based VPNs.

Why Choose an IPSEC VPN over an OpenSSL-Based One?

While SSL-based VPNs do have their own set of security mode features, IPSEC VPNs take it to the next level and provide robust means of ensuring the security of the data being transmitted that are not available in SSL-based VPNs.

What Does IPSEC Stand for and What Does It Do?

IPSEC stands for IP Security.  It is an Internet Engineering Task Force (IETF) standard suite of protocols between 2 communication points across the IP network that provide data authentication, integrity, and confidentiality. It also defines the encrypted, decrypted and authenticated packets. The IPSEC protocols needed for secure key exchange and key management are defined in it.1

What Ports Does IPSEC Operate On?

UDP port 500 should be opened as should IP protocols 50 and 51.  UDP port 500 should be opened to allow for ISAKMP to be forwarded through the firewall while protocols 50 and 51 allow ESP and AH traffic to be forwarded respectively.2

What is ISAKMP?

ISAKMP stands for Internet Security Association and Key Management Protocol.  These are two key components of an IPSEC VPN that must be in place in order for it to function normally and protect the public traffic that is being forwarded between the client and VPN server or VPN server to VPN server.

To help test online security for weaknesses, Frontline VM™ is a vulnerability management solution that uses proprietary scanning technology to perform comprehensive security assessments

What are ESP and AH?

No, ESP is not Extra-Sensory Perception!  ESP stands for Encapsulating Security Protocol and AH stands for Authentication Header. 

Encapsulating Security Protocol 

ESP gives protection to upper layer new  protocols, with a Signed area indicating where a protected data packet has been signed for integrity, and an Encrypted area which indicates the information that’s protected with confidentiality. Unless a data packet is being tunneled, ESP protects only the IP data payload (hence the name), and not the IP header.

ESP may be used to ensure confidentiality, the authentication of data origins, connectionless integrity, some degree of traffic-level confidentiality, and an anti-replay service (a form of partial sequence integrity which guards against the use of commands or credentials which have been captured through password sniffing or similar attacks).

Authentication Header

Authentication Header (AH) is a new protocol and part of the Internet Protocol Security (IPsec) protocol suite, which authenticates the origin of IP packets (datagrams) and guarantees the integrity of the data. The AH confirms the originating source of a packet and ensures that its contents (both the header and payload) have not been changed since transmission.

If security associations have been established, AH can be optionally configured to defend against replay attacks using the sliding window technique.3

How Do They All Work Together?

When properly configured, an IPSEC VPN provides multiple layers of security that ensure the security mode and integrity of the data that is being transmitted through the encrypted tunnel.  This way an organization can feel confident that the data has not been intercepted and altered in transit and that they can rely on what they are seeing.

Is an IPSEC VPN Right for Me?

That really depends upon what you are trying to accomplish as well as what the security and privacy controls are that you have within your organization.  While OpenSSL VPNs do provide a great deal of security, there are aspects that that an IPSEC VPN that an OpenSSL VPN simply can’t provide.

About the Author

Our Vulnerability Research Team consists of credentialed (Security+, Network+, CISSP) cybersecurity experts with decades of combined experience in research, analysis, and the discovery of unknown vulnerabilities. 

Create Layers of Cybersecurity

Keep your organization proactively secure by adding offensive security layers including vulnerability management.

Get A Demo

Share This