We haven't said anything about how can Security Associations be managed, agreed upon. Naturally, system administrator can do so manually, but automated approach is offered by IKE (Internet Key Exchange) suit of algorithms defined under RFC 2409, and, precisely, a framework defined in RFC 2408- ISAKMP (Internet Security Association and Key Management Protocol).
Basic concepts of IKE (otherwise quite complex and hard to follow):
Remember, that IKE runs prior to IPSec (in fact, it facilitates the use of IPSec). It uses non-stateful communication in Layer 3 (i.e. UDP protocol) on port 500.
There are two phases of IKE:
establishing a secure channel. Authenticates both machines using identities (names, public or pre-shared keys). Establish material for DH in Phase 2. Can be run in two modes: main - providing integrity of messages, giving more flexibility in the use of cryptographic algorithms, and aggressive - executed faster (3 instead of 6 message exchanges), but with limited optional settings.
establish IPSec SA's, using the channel from phase 1 for protection. Use keys derived in Phase 1 to secure integrity and confidentiality of messages.
As a result, the integrity and encryption secret keys are established between the two communicating parties that can be furter used for symmetric encryption (such as in IPSec).