Service discovery is one of the big challenges for applications with a microservices architecture and for other distributed or service‑oriented systems. The network locations of service instances can change frequently due to autoscaling, failures, or upgrades, so they are usually dynamically assigned. That means applications need a sophisticated service discovery mechanism for tracking the existence and current location of a service instance.
This blog post describes how to dynamically add or remove servers in an NGINX Plus upstream group by registering them with etcd, a popular open source tool for service discovery. The solution uses NGINX Plus’ on‑the‑fly reconfiguration API, which eliminates the need to reload the NGINX Plus configuration when changing the set of load‑balanced servers. With NGINX Plus Release 8 (R8) and later, changes you make with the API can be configured to persist across restarts and configuration reloads.
CoreOS has developed etcd as a distributed, consistent key‑value store for shared configuration and service discovery. Its service discovery protocol helps any new etcd member to discover all other members during the cluster bootstrap phase, using a shared discovery URL.
To make it easier to combine the NGINX Plus on‑the‑fly reconfiguration API with etcd, we’ve created a GitHub project called etcd-demo, which includes step‑by‑step instructions for setting up the demo deployment described here. Using tools like Docker, Docker Compose, Homebrew, and jq, you can spin up a Docker–based environment in which NGINX Plus load balances HTTP traffic to a couple of hello‑world applications, with all components running in separate Docker containers.
Editor – Demos are also available for other service discovery methods:
- Service Discovery for NGINX Plus Using Consul APIs
- Service Discovery for NGINX Plus Using DNS SRV Records from Consul
- Service Discovery for NGINX Plus with ZooKeeper
How the Demo Works
First we spin up a separate Docker container for each of the following apps:
- etcd – Performs service discovery.
- Registrator – Registers services with etcd. Registrator monitors for containers being started and stopped and updates etcd when a container changes state.
- hello – Simulates a backend server. This is another project from NGINX, Inc., an NGINX web server that serves an HTML page containing the web server’s hostname, IP address, and port.
- A second instance of the hello app – Simulates another backend server.
- NGINX Plus – Load balances the other listed services.
The NGINX Plus container listens on the public port 80, and the built‑in NGINX Plus dashboard on port 8080. The etcd container listens on ports 2379, 2380, and 4001, and advertises the IP address of the Docker host (HOST_IP) as the address for other containers to use when communicating with it.
Registrator monitors Docker for new containers that are launched with exposed ports, and registers the associated service with etcd. By setting environment variables within the containers, we can be more explicit about how to register the services with etcd. For each hello (web server) container, we set the SERVICE_TAGS environment variable to production to identify the container as an upstream server for load balancing by NGINX Plus. When a container quits or is removed, Registrator removes its corresponding key from etcd automatically.
Finally, we use the script included in the demo, etcd_exec_watch.sh, to invoke an external handler, script.sh, every time there is an update to the list of registered services (that is, a change to etcd keys). The external handler uses environment variables set by etcd to determine which servers to add and remove from the NGINX Plus upstream group. If ETCD_WATCH_ACTION=set, the handler adds servers named by ETCD_WATCH_VALUE, and if ETCD_WATCH_ACTION=delete it removes servers that are not also present in etcd.
Summary
Using a script and handler like the ones in etcd-demo to dynamically reconfigure your upstream groups based on the services registered with etcd automates the process of upstream reconfiguration in NGINX Plus. This automation also frees you from having to figure out how to issue the API calls correctly and reduces the amount of time between the state change of a service in etcd and its addition or removal in the NGINX Plus upstream group.
Check out our other blogs about service discovery for NGINX Plus:
- Service Discovery for NGINX Plus Using Consul APIs
- Service Discovery for NGINX Plus Using DNS SRV Records from Consul
- Service Discovery for NGINX Plus with ZooKeeper
And try out automated reconfiguration of NGINX Plus upstream groups using etcd for yourself:
- Download etcd-demo from our GitHub repository
- Start a free 30‑day trial of NGINX Plus today or contact us for a live demo