IPsec in plain terms

IPsec (Internet Protocol Security) is a set of standards that secures traffic at the IP layer. Instead of securing only a single application, IPsec can protect IP packets themselves. In practice, IPsec provides two main cryptographic functions:

  • Authentication/Integrity: ensures packets haven’t been modified.
  • Confidentiality: encrypts packet contents so eavesdroppers can’t read them.

IPsec is commonly used in two broad scenarios: site-to-site VPNs (protecting traffic between networks) and remote access (protecting traffic from a client to a gateway). Even within those scenarios, the exact design choices—like which encryption methods are used—determine what “secure” means operationally.

How IPsec works (high level)

IPsec is typically described in terms of components and message flow rather than a single monolithic “protocol.” The most important building blocks are:

  1. Security Associations (SAs) An SA defines how to protect a stream of traffic: which algorithm(s) to use, key material, and related parameters (such as lifetimes). An SA is not “global”; it’s tied to the peer and the traffic selector rules that decide which packets should use it.

  2. AH and ESP Two widely referenced IPsec payloads provide different protection profiles:

  • AH (Authentication Header) focuses on integrity and authentication.
  • ESP (Encapsulating Security Payload) supports encryption, integrity, and authentication depending on configuration.

Because AH and ESP handle protection differently, “IPsec is enabled” is not enough—you need to know whether AH or ESP is used, and which algorithms and integrity modes are configured.

  1. Keying and negotiation (often via IKE) In real deployments, endpoints must agree on SA parameters and keys. Many systems use an ISAKMP/IKE-style negotiation process to establish matching SAs. If negotiation succeeds but the data path fails (or vice versa), the root cause is usually in mismatch or policy/routing/firewall issues.

  2. Transport vs. Tunnel mode IPsec can be applied directly to the payload portion of IP traffic (transport mode) or by encapsulating the protected packet in a new IP header (tunnel mode).

  • Transport mode typically protects end-to-end host traffic.
  • Tunnel mode is often used for VPN-like scenarios between gateways, because it can preserve routing structure for protected network-to-network traffic.

Differences and limitations that change outcomes

IPsec’s security properties depend heavily on configuration and operational details. Key limitations and common “gotchas” include:

Cryptography and interoperability

Even if both sides “use IPsec,” they may not agree on:

  • encryption algorithm
  • integrity/authentication algorithm
  • key sizes
  • SA lifetimes

When these don’t match, negotiation may fail or the established SAs won’t actually be usable for the desired traffic. This is one of the most common causes of broken tunnels.

What IPsec protects—and what it doesn’t

IPsec protects IP packets according to SAs and selectors. If a packet doesn’t match the selectors (or the expected routing path), it may travel without IPsec protection. Conversely, traffic you didn’t intend to protect might match selectors and get routed through the tunnel.

Also note that IPsec doesn’t magically fix issues like incorrect routing tables, NAT interactions (which can be managed but require care), or application behavior that assumes plain transport characteristics.

Network path and firewall requirements

IPsec typically requires specific network flows to be allowed. If firewalls block negotiation (key establishment) or data traffic (protected packets), you’ll see symptoms like:

  • SA negotiation not completing
  • the tunnel “coming up” but passing little or no traffic
  • sporadic packet loss

The exact ports/protocols and firewall rules depend on the implementation and which IPsec components are used, so treat firewall checks as part of the validation process.

Operational fragility around SAs

Because SAs are time-bounded and parameter-specific, expired lifetimes, key rollover, or rekey timing can cause intermittent failures. Two sides can also “agree” on something during negotiation while still disagreeing on packet classification due to traffic selector or routing differences.

Practical checks: how to validate an IPsec setup

If you’re troubleshooting or confirming that IPsec is functioning as intended, use checks that map to the system’s guarantees: negotiation correctness, SA match, and packet flow.

1) Confirm negotiation and SA establishment

Check whether:

  • the security association negotiation completes
  • the established SAs are in an active/usable state
  • both peers reference matching parameters

If SAs are not established, focus on configuration mismatch and connectivity for the negotiation phase.

2) Verify what is protected (selectors and mode)

Ensure the traffic selectors and routing map to the packets you expect to protect.

  • Is the deployment using transport or tunnel mode?
  • Do the protected subnets/hosts align on both ends?

A tunnel can be “up” while your specific traffic is not protected because it doesn’t match the intended policy.

3) Check cryptographic and integrity algorithm agreement

Confirm that both ends use compatible algorithms and keying material conventions. Even small differences—like integrity method selection—can prevent successful data exchange.

4) Validate packet flow and firewall rules

Look for evidence that protected traffic actually traverses the intended path:

  • firewall rules allow the needed traffic for both negotiation and data
  • there is no asymmetric routing that sends packets out one way and replies out another
  • MTU/fragmentation issues are not breaking packets (a common cause of “tunnel works but some traffic breaks”)

5) Re-check after rekey events

If the tunnel fails periodically, rekeying/SA expiry might be involved. Validate that both endpoints reestablish SAs smoothly and that there is no persistent parameter drift.

A few terms show up around IPsec, and separating them helps avoid misunderstandings:

  • VPN (generic term): “VPN” describes a goal (secure connectivity). IPsec is one standardized approach to achieve that goal.
  • Transport security vs. IP-layer security: application-layer security (like TLS) secures application sessions; IPsec secures IP packets.
  • Authentication/integrity vs. confidentiality: some IPsec configurations prioritize integrity (or both). So you should confirm which protections are actually active.

What to do when you’re unsure

Because IPsec implementations vary and configuration details matter, treat verification as a sequence: negotiation → SA parameters → traffic classification → packet path. If any step fails, the most likely cause is a mismatch (algorithms, selectors, mode), or a networking issue (firewall/routing/MTU). When documentation or vendor behavior is unclear, the most dependable approach is to validate the observed states (negotiated SAs and packet counters/logs) against your intended policy.