What a VPN is
A virtual private network creates logical connectivity across an underlying network that you do not fully control. VPNs can connect sites, remote users, cloud environments, or isolated virtual networks. The term describes the connectivity and security objective rather than one single protocol.
IPsec is a standards-based family that provides security services at the IP layer and is widely used for site-to-site and remote-access designs.
IPsec architecture
IPsec separates data protection from key management. Security Associations define parameters used to protect traffic in one direction, while policy determines which traffic should be protected. Implementations maintain state for negotiated algorithms, keys, lifetimes, sequence numbers, and selectors.
A working tunnel therefore depends on both negotiation state and a correct forwarding/policy path for the protected traffic.
ESP and AH
Encapsulating Security Payload (ESP) provides confidentiality when encryption is used and can also provide integrity and authentication. Authentication Header (AH) provides integrity and authentication but does not encrypt payload and is much less common in modern VPN deployments.
NAT compatibility and common implementation practice make ESP the protocol most engineers encounter in production IPsec.
Transport mode and tunnel mode
Transport mode protects the payload of the original IP packet while retaining the original IP header as the outer header. Tunnel mode encapsulates the original packet inside a new IP packet and is common for gateway-to-gateway VPNs.
The mode changes overhead, addressing visibility, and how traffic is routed through the protected path.
IKEv2 and negotiation
IKEv2 authenticates peers, negotiates cryptographic parameters, establishes an IKE Security Association, and then creates Child SAs used to protect traffic. Peers must agree on proposals, identities, authentication, and traffic selectors.
Troubleshooting negotiation requires knowing where it failed: reachability, proposal mismatch, authentication, certificate validation, selector mismatch, or Child SA installation.
Traffic selectors and policy
Traffic selectors describe which source and destination ranges and protocols belong to an IPsec Child SA. If peers propose incompatible selectors, negotiation may fail or only part of the expected traffic may enter the tunnel.
Document protected subnets explicitly and keep them aligned with routing and firewall policy. Broad selectors can create unintended reachability, while narrow selectors can leave required traffic unprotected.
NAT Traversal
NAT modifies IP headers and can interfere with protocols that protect or depend on those fields. NAT Traversal encapsulates IPsec ESP in UDP, commonly using port 4500, so the flow can cross NAT devices more reliably.
If peers negotiate successfully but data does not pass through a translated path, confirm NAT-T state, intermediate firewall policy, and whether the expected UDP encapsulation is present.
MTU, overhead, and fragmentation
IPsec adds headers and sometimes UDP encapsulation, reducing the payload that fits within the same physical MTU. Large packets may require fragmentation or Path MTU Discovery. If ICMP messages needed for PMTUD are blocked, users can see classic “small packets work, large transfers fail” symptoms.
Account for tunnel overhead and test realistic application packet sizes.
Routing and VPNs
A tunnel does not replace routing. The device must still know which traffic should enter the tunnel and how to reach the remote peer over the underlay. Route-based VPNs expose tunnel interfaces; policy-based designs may tie protection more directly to selectors.
Check both routes: one for the encrypted peer reachability and another for the protected destination.
ACLs, firewalls, and NAT around the tunnel
VPN traffic often crosses ACL, firewall, and NAT policy before or after encryption. Order of operations varies by platform, so verify whether addresses are matched before or after translation and which interface or zone sees the encrypted versus clear-text traffic.
A common failure is translating traffic that should be exempt from NAT or filtering IKE/ESP/NAT-T before negotiation can complete.
Phase-by-phase troubleshooting
First prove peer IP reachability. Then validate IKEv2 negotiation and authentication, followed by Child SA creation and selectors. Next inspect routing, policy, counters, encryption/decryption statistics, and return traffic. Finally test MTU and application-specific behavior.
This sequence separates control-plane negotiation from data-plane forwarding and prevents repeated reconfiguration of working phases.
Operational security
Use current cryptographic suites supported by both peers, protect private keys and pre-shared secrets, rotate credentials, restrict management access, and monitor failed negotiations. Avoid obsolete algorithms simply for backward compatibility unless a documented transitional requirement exists.
Keep a clear inventory of peers, protected networks, certificates, and expiration dates.
Site-to-site flow example
A host in Site A sends traffic to a subnet in Site B. The gateway at Site A routes the packet toward the VPN policy, matches the selector, encrypts it, and sends the protected packet to the public address of Site B. The remote gateway decrypts it and forwards the original packet toward the destination.
The reply must match compatible routing and selectors in the opposite direction. Asymmetry can break an otherwise healthy tunnel.
Rekey, lifetimes, and peer liveness
Security Associations have lifetimes and need to be replaced before expiry. Rekey events should be normal and ideally transparent. Dead Peer Detection or similar liveness mechanisms help identify peers that are no longer reachable.
Repeated rekey failures can cause periodic outages, so logs and SA lifetime counters are useful when an incident follows a regular interval.
Interoperability between vendors
Standards improve interoperability, but vendors expose different names, defaults, proposal order, route integration, and policy models. Compare the negotiated result rather than assuming identical configuration syntax means identical behavior.
When troubleshooting cross-vendor tunnels, document algorithms, DH groups, authentication, identities, selectors, lifetimes, NAT-T, and routing on both sides.
Monitor tunnel health
Monitor IKE and Child SA state, bytes encrypted/decrypted, packet drops, replay errors, negotiation failures, peer availability, and certificate validity where applicable. Correlate tunnel events with routing and interface health.
A tunnel reported as “up” may still carry no useful traffic if selectors, routes, or policy are wrong.
References
IPsec architecture, ESP, and IKEv2 are defined through IETF standards including RFC 4301, RFC 4303, and RFC 7296 plus later updates. NAT Traversal and algorithm guidance have their own standards and updates.
Use implementation documentation for platform-specific order of operations and supported cryptography.
Next step
Continue with Layer 2 security to protect the local access network before traffic ever reaches a routed firewall or VPN boundary.