DNS

Aliases

Each messaging service (foo) has a matching Load Balanced DNS alias: foo-mb.cern.ch. This alias contains all the currently available brokers. Note that the order of the addresses returned this way is not meaningful.

Most messaging services also have another Load Balanced DNS alias: foo-lb-mb.cern.ch. It contains only the least loaded broker. This is updated every few minutes.

Clients must always use the DNS alias and never the real names of the machines currently providing the service since these might change at any time.

Producers

For producers, the motto is "send to any" so they should resolve the DNS alias and pick any address behind the alias, randomly.

Short lived producers can also simply use the foo-lb-mb.cern.ch alias instead of choosing a random broker.

Long running producers should reconnect from time to time in order to better take advantage of all the available brokers. A typical use case is when a new broker is added: long running producers that do not reconnect will never use the new broker.

Consumers

For consumers, the motto is "receive from all" so they should resolve the DNS alias and establish connections to all the brokers.

In addition, consumers should have a sticky behaviour and stay connected to a broker that gets removed from DNS. This ensures that as many messages as possible are consumed before a broker undergoes a maintenance operation.

For Puppet managed consumers, PuppetDB can return the list of brokers.