Definition and the basic idea

An IPSec VPN (Internet Protocol Security Virtual Private Network) is a way to protect IP traffic between endpoints by applying cryptographic protection to packets. In practice, it uses two main capabilities: (1) encryption to keep packet contents confidential and (2) authentication/integrity checks to detect tampering. A secure channel is usually set up by negotiating cryptographic parameters and exchanging keys, after which protected traffic can flow.

IPSec is a framework: it defines mechanisms that are combined and configured together. That is why “how it works” can involve multiple moving parts—most importantly key management and the way packets are wrapped or protected.

A simple model: negotiate, establish keys, protect packets

A useful mental model is a three-step flow.

  1. Negotiate security settings. Endpoints agree on which algorithms to use for confidentiality and integrity, and they also decide how the traffic should be protected.

  2. Exchange keys safely. Before regular traffic is protected, the endpoints run a key-management procedure so both sides derive the same cryptographic keys. Many deployments use IKE (Internet Key Exchange) for this negotiation and key exchange.

  3. Encrypt and authenticate IP packets. Once keys and protections are in place, IPSec applies protection to packet data. The exact behavior depends on the chosen mode (for example, how much of the original IP packet is encapsulated or covered by the protection).

This explains the core “VPN” behavior: it isn’t just about encrypting; it’s about negotiating and using keys so both sides can reliably verify and decrypt what they receive.

The building blocks: SAs and common encapsulation/protection modes

IPSec protection is organized around the idea of Security Associations (SAs). An SA is essentially a security agreement that identifies the cryptographic parameters and keys used for protecting traffic in one direction (many setups create SAs for inbound and outbound separately).

Two commonly discussed protection modes are:

  • Tunnel mode: IPSec typically wraps more of the original packet, creating a “tunnel” between endpoints. This is often used when you want to connect networks (for example, one site to another) because the outer packaging can route traffic through intermediate networks while preserving protection between the VPN endpoints.

  • Transport mode: IPSec generally protects the payload (and relevant header parts depending on the protocol) rather than encapsulating the entire original packet in a new outer header. This can be used for host-to-host scenarios.

A protocol choice also matters. IPSec commonly uses two related mechanisms for different parts of protection work:

  • AH (Authentication Header) focuses on authentication/integrity.
  • ESP (Encapsulating Security Payload) provides encryption and can also provide integrity.

If you’re trying to understand “how it works” at a troubleshooting level, thinking in terms of “which SAs exist,” “which mode is active,” and “is encryption expected to be present” is often more actionable than focusing only on the word “IPSec.”

Differences and limits: what changes the behavior and what can break it

Several factors can change how IPSec behaves.

  • Which key-management method is used. Key exchange is what enables the endpoints to derive matching keys. If key negotiation fails, traffic protection can’t be applied correctly. The exact steps can vary by deployment and configuration.

  • Algorithm and key-strength choices. The security of IPSec depends on cryptographic strength and correct parameter selection (for example, choosing strong encryption and integrity algorithms, and using appropriate lifetimes/renewal behavior). Weak choices can reduce practical security.

  • Configuration consistency. Both sides must agree on the relevant parameters (mode, SAs, identifiers, and expected protections). A mismatch can prevent successful establishment or decryption.

  • Traffic scope expectations. IPSec protects IP traffic according to the configured selectors. If you expect certain applications to be protected but selectors don’t match, some traffic may not be covered.

It’s also important to set boundaries on what IPSec does not automatically guarantee. IPSec can secure the traffic it protects, but it doesn’t eliminate all operational risks: endpoints must still be configured securely, and the overall network design (routing, firewalls, and key handling) affects the real-world outcome.

Practical checks: how to verify you understand what’s happening

You can validate your understanding without relying on vendor-specific details by checking these concepts:

  • Is key exchange succeeding? If SAs aren’t established, encrypted traffic won’t start.