Load Balancer

Ocelot can load balance across available downstream services for each route. This means you can scale your downstream services, and Ocelot can use them effectively.

LoadBalancerOptions Schema

Class: FileLoadBalancerOptions

Here is the complete load balancer configuration (the schema) of top-level properties. You do not need to set all of these options, but the Type option is required.

"LoadBalancerOptions": {
  "Expiry": 2147483647,
  "Key": "",
  "Type": ""
}

The actual LoadBalancerOptions schema with all the properties can be found in the C# FileLoadBalancerOptions class.

Configuration

The types of load balancer available are:

Type	Description
CookieStickySessions	This uses a cookie to stick all requests to a specific server. More information can be found in the CookieStickySessions Type 2 section.
LeastConnection	This tracks which services are dealing with requests and sends new requests to the service with the fewest (“least”) existing requests. The algorithm state is not distributed across a cluster of Ocelots.
NoLoadBalancer	This takes the first available service from configuration or service discovery.
RoundRobin	This loops through available services and sends requests. The algorithm state is not distributed across a cluster of Ocelots.

You must choose which load balancer to use in your configuration.

The following shows how to set up multiple downstream services for a route using ocelot.json and then select the LeastConnection load balancer. This is the simplest way to set up load balancing.

{
  "UpstreamPathTemplate": "/posts/{postId}",
  "UpstreamHttpMethod": [ "Put", "Delete" ],
  "DownstreamPathTemplate": "/api/posts/{postId}",
  "DownstreamScheme": "https",
  "DownstreamHostAndPorts": [
    { "Host": "10.0.1.10", "Port": 5000 },
    { "Host": "10.0.1.11", "Port": 5000 }
  ],
  "LoadBalancerOptions": {
    "Type": "LeastConnection"
  }
}

Service Discovery [1]

The following shows how to set up a route using Service Discovery and then select the LeastConnection load balancer.

{
  // ...
  "ServiceName": "product",
  "LoadBalancerOptions": {
    "Type": "LeastConnection"
  }
}

When this is set up, Ocelot will look up the downstream host and port from the Service Discovery provider and load balance requests across any available services. If you add and remove services from the Service Discovery provider [1], Ocelot should respect this and stop calling services that have been removed and start calling services that have been added.

CookieStickySessions Type [2]

We have implemented a basic sticky session type of load balancer. The scenario it is meant to support involves having a number of downstream servers that do not share session state. If you receive more than one request for one of these servers, it should go to the same server each time; otherwise, the session state might be incorrect for the given user.

In order to set up the CookieStickySessions load balancer, you need to do something like the following:

{
  "UpstreamPathTemplate": "/posts/{postId}",
  "UpstreamHttpMethod": [ "Put", "Delete" ],
  "DownstreamPathTemplate": "/api/posts/{postId}",
  "DownstreamScheme": "https",
  "DownstreamHostAndPorts": [
    { "Host": "10.0.1.10", "Port": 5000 },
    { "Host": "10.0.1.11", "Port": 5000 }
  ],
  "LoadBalancerOptions": {
    "Type": "CookieStickySessions",
    "Key": "ASP.NET_SessionId",
    "Expiry": 1800000 // milliseconds
  }
}

The LoadBalancerOptions are:

Option	Description
Type	This needs to be CookieStickySessions.
Key	This is the key of the cookie you wish to use for the sticky sessions.
Expiry	This is how long, in milliseconds, you want the session to be stuck for. Remember, this refreshes on every request, which is meant to mimic how sessions usually work.

Note 1: If you have multiple routes with the same LoadBalancerOptions, then all of those routes will use the same load balancer for their subsequent requests. This means the sessions will be stuck across routes.

Note 2: If you define more than one DownstreamHostAndPort, or if you are using a Service Discovery provider such as Consul and it returns more than one service, then CookieStickySessions uses RoundRobin to select the next server. This is hard-coded at the moment but could be changed.

Custom Balancers [3]

In order to create and use a custom load balancer, you can do the following. Below, we set up a basic load balancing configuration, and note that the Type is MyLoadBalancer, which is the name of a class we will set up to perform load balancing.

{
  // ...
  "DownstreamHostAndPorts": [
    { "Host": "10.0.1.10", "Port": 5000 },
    { "Host": "10.0.1.11", "Port": 5000 }
  ],
  "LoadBalancerOptions": {
    "Type": "MyLoadBalancer"
  }
}

Then, you need to create a class that implements the ILoadBalancer interface. Below is a simple round-robin example:

using Ocelot.LoadBalancer.LoadBalancers;
using Ocelot.Responses;
using Ocelot.Values;

public class MyLoadBalancer : ILoadBalancer
{
    private readonly Func<Task<List<Service>>> _services;
    private static object Locker = new();
    private int _last;

    public MyLoadBalancer() { }
    public MyLoadBalancer(Func<Task<List<Service>>> services)
        => _services = services;

    public string Type => nameof(MyLoadBalancer);
    public void Release(ServiceHostAndPort hostAndPort) { }

    public async Task<Response<ServiceHostAndPort>> LeaseAsync(HttpContext context)
    {
        var services = await _services.Invoke();
        lock (Locker)
        {
            _last = (_last >= services.Count) ? 0 : _last;
            var next = services[_last++];
            return new OkResponse<ServiceHostAndPort>(next.HostAndPort);
        }
    }
}

Finally, you need to register this class with Ocelot.

1. We have used the most complex example below to show all of the data and types that can be passed into the factory that creates load balancers.

using Ocelot.Configuration;
using Ocelot.DependencyInjection;
using Ocelot.ServiceDiscovery.Providers;

Func<IServiceProvider, DownstreamRoute, IServiceDiscoveryProvider, MyLoadBalancer> lbFactory
    = (serviceProvider, Route, discoveryProvider) => new MyLoadBalancer(discoveryProvider.GetAsync);
builder.Services
    .AddOcelot(builder.Configuration)
    .AddCustomLoadBalancer(lbFactory);

2. However, there is a much simpler example that will work the same way:

using Ocelot.DependencyInjection;

builder.Services
    .AddOcelot(builder.Configuration)
    .AddCustomLoadBalancer<MyLoadBalancer>();

Notes

1. There are numerous IOcelotBuilder methods to add a custom load balancer. The interface is as follows:

   IOcelotBuilder AddCustomLoadBalancer<T>()
       where T : ILoadBalancer, new();
   IOcelotBuilder AddCustomLoadBalancer<T>(Func<T> loadBalancerFactoryFunc)
       where T : ILoadBalancer;
   IOcelotBuilder AddCustomLoadBalancer<T>(Func<IServiceProvider, T> loadBalancerFactoryFunc)
       where T : ILoadBalancer;
   IOcelotBuilder AddCustomLoadBalancer<T>(Func<DownstreamRoute, IServiceDiscoveryProvider, T> loadBalancerFactoryFunc)
       where T : ILoadBalancer;
   IOcelotBuilder AddCustomLoadBalancer<T>(Func<IServiceProvider, DownstreamRoute, IServiceDiscoveryProvider, T> loadBalancerFactoryFunc)
       where T : ILoadBalancer;
   

2. When you enable custom load balancers, Ocelot looks up your load balancer by its class name when it decides whether to perform load balancing.

   • If it finds a match, it will use your load balancer to load balance.

   • If Ocelot cannot match the load balancer type in your configuration with the name of the registered load balancer class, then you will receive an HTTP 500 Internal Server Error.

   • If your load balancer factory throws an exception when Ocelot calls it, you will receive an HTTP 500 Internal Server Error.

3. Remember, if you specify no load balancer in your Configuration, Ocelot will not attempt to load balance.

---

[1] (1,2)

Currently supported Service Discovery providers are Consul, Kubernetes (K8s) , Eureka 4, Service Fabric, and manually developed Custom Providers.

[2]

The “CookieStickySessions Type 2” feature was requested in issue 322, though what the user wants is more complicated than just sticky sessions. Anyway, we thought this would be a nice feature to have! Initially, the feature was released in version 6.0.0.

[3]

The “Custom Balancers 3” feature by David Lievrouw implemented a way to provide Ocelot with a custom load balancer in PR 1155 (his issue 961, released in version 15.0.3).