The way companies deploy new features in their applications has changed dramatically over the past few years. Whereas before companies would wait and deliver a large set of new features all at once, the focus now is on delivering features to customers as soon as the code is ready. Using automation and a strong tool set, some companies are able to do this over 50 times per day.
Deploying changes this frequently, however, puts additional requirements on your application delivery infrastructure, because each deployment can require a change in configuration. For example, if you delete and replace application servers during a deployment rather than updating them, you must also update the load balancer configuration. Or, if the newly deployed functionality requires access to additional resources, such as a different database, then you also need to update the security and access control configuration. On the negative side, if you discover a bug in the new functionality, you might need to roll back the configuration.
In a traditional development environment where an IT team controls deployment, making changes to the application delivery infrastructure typically requires filing a ticket and waiting for a change window, which can sometimes take weeks. If you’re deploying 50 times per day, that model clearly doesn’t work. Making configuration changes should be flexible, automatable, and easily reversible. In addition, the infrastructure must be able to stay online and process traffic while undergoing constant configuration changes.
NGINX and NGINX Plus are application delivery platforms designed to work in the most demanding environments. NGINX can be easily reconfigured without any downtime or loss of traffic, and NGINX Plus adds additional features to help further automate and ease the process of modifying configuration. This blog post covers techniques for dynamically modifying and automating configuration for both NGINX and NGINX Plus.
Modifying NGINX Configuration
NGINX configuration is captured in configuration files. When you change a configuration file and restart NGINX to pick up the new configuration, it implements a “graceful restart”. Both the old and new copies of NGINX run side by side for a short period of time. The old processes don’t accept any new connections and terminate once all their existing connections terminate.
One of the most common reasons to change NGINX configuration is to add and remove backend servers. As discussed earlier, if servers are deleted and replaced rather than updated, this type of change will happen often. Another use case is scaling: adding servers to handle a traffic spike and then removing them when the traffic has died down.
It may not always be convenient to modify configuration files and restart NGINX or NGINX Plus. For example, if you are experiencing large amounts of traffic and high load, restarting NGINX and reloading the configuration at that point further increases load on the system and can temporarily degrade performance.
For the common case of adding and removing backend servers, we have added an HTTP‑based API as well as DNS‑based reconfiguration in NGINX Plus to perform these operations without restarting NGINX Plus.
DNS-Based Reconfiguration with NGINX Plus
With NGINX Plus, your DNS server can keep the definitive list of the servers that belong to an upstream group. Update the list of available servers in your DNS and NGINX Plus automatically picks up the changes.
To use DNS‑based reconfiguration, create multiple A records with the same hostname but different IP addresses, as in this example from Digital Ocean. The result is similar to what happens with Round Robin DNS, except that in this case NGINX Plus doesn’t just use the first address in the list from DNS as browser clients tend to do. Instead it applies the configured load‑balancing algorithm to all the servers in the list, which can be Least Connections, Round Robin, or any of its other load balancing algorithms. NGINX Plus is effectively using the DNS server as a database for upstream servers.
The NGINX Plus configuration looks like this:
http {
upstream backend {
zone backend 64k;
server backend.example.com resolve;
}
resolver 192.168.1.50;
}Putting resolve after the hostname of the server instructs NGINX to refresh the list of IP addresses for that hostname periodically. (NGINX Plus’ default behavior is to fetch DNS information only once, as it initializes.)
The resolver directive specifies the DNS server to use. By default, NNGINX Plus refreshes a DNS entry according to the time‑to‑live (TTL) that DNS sets for it. To override the default, include the valid parameter to the resolver directive:
resolver 192.168.1.50 valid=30s;In this example, NGINX Plus caches DNS entries for 30 seconds before refreshing them.
On-The-Fly Configuration Using an HTTP‑Based API
NGINX Plus also provides an HTTP interface for adding, removing, and modifying backend servers dynamically and without having to reload the configuration. This is useful for autoscaling and when you want to take a server down temporarily for maintenance. It’s especially great for transient changes, as changes made using this interface are not preserved when you restart NGINX Plus or reload its configuration.
Configuring the HTTP-Based API
To use the HTTP interface, create a separate location block in the configuration block for a virtual server and include the upstream_conf directive. We’re using the conventional name for this location, /upstream_conf. We want to restrict use of the interface to administrators on the local LAN, so we also include allow and deny directives:
server {
listen 8080; # Listen on a local port
allow 10.0.0.0/8; # Allow access only from LAN
deny all; # Deny everyone else
location /upstream_conf {
upstream_conf;
}
}A shared memory zone is required to use the HTTP‑based API, so we configure an upstream group of servers called backend like this:
upstream backend {
zone backend 64k;
server 10.2.2.90:8000;
server 10.2.2.91:8000;
server 10.2.2.92:8000;
}Using the HTTP-Based API
Let’s say we want to remove a faulty server from the backend upstream group. First we can send a list request with any HTTP tool to see the IDs assigned to the servers. Here we use curl:
# curl 'http://localhost:8080/upstream_conf?upstream=backend'
server 10.2.2.90:8000; # id=0
server 10.2.2.91:8000; # id=1
server 10.2.2.92:8000; # id=2The list of servers returned here follows the same syntax as the NGINX configuration file. It can be easily included in the main NGINX configuration file simply by saving it to a file and using the include directive.
Once we see the list of servers, we can specify the one we want to remove by its ID:
# curl 'http://localhost:8080/upstream_conf?remove=&upstream=backend&id=2'
server 10.2.2.90:8000; # id=0
server 10.2.2.91:8000; # id=1Let’s now add more servers to the upstream group. We can do this easily with an add request:
# curl 'http://localhost:8080/upstream_conf?add=&upstream=backend&server=10.2.2.93:8000'
server 10.2.2.93:8000; # id=3This is just a sample of what you can do with the API. You can also put servers into draining mode, alter their weights, and reconfigure them in a variety of other ways.
Conclusion
With NGINX, you can make configuration changes with no downtime and no interruption to traffic processing. NGINX Plus adds additional methods for modifying configuration dynamically. By using DNS as the definitive repository for upstream servers, you can make changes without touching NGINX Plus configuration files. An HTTP‑based API provides a familiar interface to make transient changes to the NGINX configuration, again without having to touch any configuration files.
To try NGINX Plus, start your free 30‑day trial today or contact us for a demo.