As we wrote at VK Cloud SDN

With the advent of broadband access and high speeds of mobile Internet, network load has become one of the key bottlenecks for system performance. Network providers are faced with the need to continually increase and optimize network throughput and capacity while maintaining profits. An attempt to solve this problem was the Software Defined Network - the concept of moving network functions from specialized hardware to the software level and further dividing responsibilities into different layers.

Since the existence of SDN, many reasons have emerged to use this approach:

• cost reduction at all levels - it is easier to manage the network state and infrastructure, clearer scaling and fault tolerance, clear architecture with a central controller;
  
• the ability to automatically configure and adjust the network;
  
• availability of conditionally infinite flexibility in setting up traffic processing logic;
  
• simplifying the inventory of network elements - for example, automating the removal of objects upon failure or disposal and rebalancing the network state;
  
• moving towards vendor-neutral solutions (which is especially important).
  

Although the latest developments and attention are relatively recent, the principles and foundation for the Software Defined Network have been laid over the past decades. We are in VK Cloud We are also making our own SDN called Sprut. Several years ago, developers began migrating to it from the previous one, Neutron. We asked them what's going on with the development of the new SDN.

Below the cut is the history of SDN for those who are not immersed in the context. And if you already know, then skip and read on.

SDN in the cloud

SDN has become a foundational service for cloud platforms. One of the main characteristics of the cloud is elasticity. This, as well as the speed and other characteristics of deployments, imply that underlying services should be as automated as possible, tracking the life cycle of the objects on which the deployed service operates. For example, for virtual machines this could be storage and network settings. When you set up a virtual machine in any cloud and go to the networks tab to configure which subnet the virtual machine will be on, you are involved in generating a request for SDN.

SDN is an important part of network setup automation. Therefore, SDN is an integral part of any cloud product. 

History of SDN development in VK Cloud 

VK Cloud started with OpenStack. SDN was taken from the very beginning as part of OpenStack - Neutron. The main reason was that it was necessary to start quickly and with limited human resources. At the beginning of any project, personnel is everything - when the cloud is formed, this is always a key issue. When time is short, it is very difficult to write something serious from scratch.

Technical Note

Neutron is integrated into OpenStack and has:

• many different networking options: Vlan, Overlay, Flat;
  
• a wide range of network functions: routers, load balancers, VPN, DNS, Firewall, BGP signaling;
  
• detailed and comprehensive documentation.
  

OpenStack Neutron has a simple architecture of:

• Neutron API, which accepts incoming user requests;
  
• ML2 plugin, which implements all the basic logic of Neutron;
  
• databases;
  
• RabbitMQ on the transport layer;
  
• agents involved in setting up the Data plane;
  
• Data plane, which can be OVS, NetNS, Daemons and others. 
  

OpenStack Neutron works like this:

1. To create a virtual network, the user makes a request to the Neutron API.
   
2. The Neutron API passes the request to the ML2 plugin.
   
3. The ML2 plugin sends network creation notifications to all agents via RabbitMQ.
   
4. After receiving notifications, agents determine on which computing nodes the network is created. If there are nodes, they begin setting up the Data plane; if not, they ignore them.
   

The beginning of problems with operation at large volumes

A couple of years later, the team grew, the customer base increased, and along with this, some difficulties arose with Neutron - mostly operational, since there were no problems adding features. OpenStack Neutron itself is quite simple in terms of performance, but in practice, due to architectural features and the lack of initial adaptation to large volumes, failures often occur. Several of them are listed below, and it is worth clarifying that they become especially inconvenient when operating on cloud volumes.

Agents do not store state (stateless). They don’t remember the Data plane settings, there is no feedback on how exactly it was configured or whether it was configured at all. This creates difficulties. For example, an agent may miss some events due to a temporary shutdown, overload or workload of the node on which it is located. This is critical in the cloud, where the state of resources is constantly changing.

This problem is partially mitigated by the Full Sync mechanism. It allows the agent to request all information about its state from the server in case of any errors. But with hundreds of entities in a node, Full Sync can take several hours. 

Overcomplicated Data plane. Typically, in a Neutron overlay configuration, the center of the entire Data plane is OVS. It is responsible for connecting virtual machines and connecting network functions: DHCP, DNS, Metadata Proxy and others. The OVS Agent is responsible for configuring OVS. Essentially, the logic of the entire network resides within OVS and the OVS Agent.

This complication makes it almost impossible to keep track of events. If several virtual machines are running on a node, OVS will have tens of thousands of rules that are difficult to understand. Even OVS tracing and monitoring does not always help with this.

Event-based communication via RabbitMQ. Agents communicate with the server using events. But there are errors in the code of all services - Neutron, agents, RabbitMQ. Because of this, agents can partially ignore events, and RabbitMQ can lose them during delivery due to different Race conditions. If it fails, the entire SDN will be unavailable and unmanageable. In addition, the scalability of RabbitMQ is limited: as the flow of events increases, some events may be lost. This leads to numerous problems. Gaps occur in the Data plane configuration and errors occur during traffic transmission. Monitoring these events is difficult.

More functionality - more events. OpenStack Neutron is configurable and allows you to enable many different functions. But this leads to an avalanche-like increase in the number of events from each of them. For example, Neutron with a simple Plane or VLAN configuration and basic services such as DHCP and Metadata Proxy can run reliably on 1000 hypervisors. But when overlay networks with a distributed routing mechanism, BGP signaling and other functions are enabled, problems appear already with 100 hypervisors.

Preliminary load testing helps prevent them. This must be done before enabling any additional functionality.

Architecture Limits. Mirantis itself says that 500 nodes is the limit and Neutron is not positioned as an infinitely scalable product that will withstand cloud scale. 

Choosing between your own development and a ready-made solution

The reliability and stability of the cloud is directly related to the choice of solution. For some time, Neutron met the needs of VK Cloud, but then it stopped. An objective market analysis showed that adding features to some SDNs is easy, to others it’s difficult, and to others it’s very difficult.  

Due to the lack of a worthy alternative, developing your own SDN has become a necessity.
When choosing an alternative SDN, only two options were considered:

1. Move to another SDN: Tungsten Fabric/OpenContrail, OpenDayLight, OVN. 
   
2. Rework OpenStack Neutron.
   

The required SDN had several requirements:

1. Support for the current set of networking features. Many functions that customers need have already been enabled. They cannot be disabled: the new solution must have the same functionality.
   
2. DC network type “L3 to rack (ToR)”. In data centers, networks are built on the “L3 per rack” principle. Because of this, some fault tolerance protocols, such as VRRP, which rely on a single L2 broadcast domain between all servers, are not supported. In addition, external traffic needs to be signaled towards ToR over BGP.
   
3. Scaling. The ability to scale or shard is needed to ensure network resiliency when bottlenecks are detected and make it easier to eliminate them.
   
4. Seamless migration from current SDN. Moving to a new SDN should not affect client experience or functionality.
   
5. Opportunity and potential for improvement. There was no ready-made solution “out of the box” for the stated requirements, so we only need Open-Source solutions that can be modified independently.
   

Based on the requirements, the choice was reduced to two options: Tungsten Fabric and the same OpenStack Neutron. But:

1. U Tungsten Fabric there is no necessary functionality, it needs to be completed immediately. Refinement is complicated by a huge code base, which is difficult to understand due to the lack of publicly available documentation. An additional challenge is the large array of technologies. This complicates implementation, operation and support. As a result, the disadvantages of the solution outweigh the advantages of Tungsten Fabric - wide functionality and the ability to integrate into “L3 on a rack”.
   
2. In processing OpenStack Neutron there are many advantages: the necessary functions are already there, migration is not needed, improvement can be carried out step by step, and the team has sufficient expertise. But the shortcomings of Neutron, which were listed above, do not go away - because of them, we were looking for an alternative SDN. In addition, reworking would require changing too much code and working with an inappropriate architecture. 
   

Since both Tungsten Fabric and OpenStack Neutron only met part of the requirements, we decided to develop our own SDN. 

How to write your own SDN and migrate the platform to it when everything works on the old one

There are two key disadvantages to writing your own solution:

1. Writing SDN is difficult. We need a large team with expertise in the Control plane and Data plane and knowledge of networks.
   
2. Great Time-to-market. It takes a lot of time to get a ready-made solution and put it into operation.
   

But there are more advantages, and they are more significant:

1. You can make any architecture without additional unnecessary elements. 
   
2. Any migration can be envisaged. 
   
3. You can solve a problem that is interesting for developers.
   

After weighing the pros and cons, the VK Cloud team began developing their SDN. They called her Sprut. It took less than a year to develop. Of course, now Sprut does not yet have all the functionality, but it is already in production together with Neutron.

What's important is that Sprut doesn't have the same problems as Neutron. For this purpose, its architecture was completely redesigned:

1. Agents have the ability to continuously collect information about Data plane settings.
   
2. There is no longer an event-based model of communication between components. Agents now always receive the target state they should be in from the server and continually re-query it. The result is an analogue of permanent Full Sync, in which agents compare the current state of the Data plane with the target state from the server, apply the necessary diff to the Data plane and bring it to the current state. In automatic control theory, this approach is called a closed control loop. 
   
3. RabbitMQ has been replaced by the regular HTTP REST API. It copes better with large amounts of data on the target state of agents, and is easier to develop and monitor. 
   

Comparison of OpenStack Neutron and SPRUT architectures

But these alterations did not solve the main problem - the Data plane remained complex. To understand how to simplify it, I had to delve into the theory and analyze the basic virtual primitives of Neutron: port, network, subnet, router.

It is easy to understand the principle of operation of a cloud network if you project the components of a virtual network onto a familiar physical one.

The virtual network turns into a Switch and a Metadata Proxy. The virtual subnet turns into a DHCP server. The virtual router becomes physical. Ports become regular or virtual machines

In a physical infrastructure format, all these components are distributed across a huge cluster of physical nodes that are interconnected. To do this, the components are connected by a set of switches.



Segmentation among these switches occurs using VLAN technology or the more modern MPLS. In this case, all segments represent different virtual networks. 

The division that is used in standard networks fully meets our operational needs, so when designing SPRUT, we similarly divided the areas of responsibility into three levels: NFV, SDN and APP.


NFV (Network Function Virtualization) is responsible for providing network entities, SDN is responsible for their communication with each other. APP ensures API level compatibility

Separating responsibilities helped us simplify the Data plane in the original SPRUT architecture.



After making changes, two independent layers appeared in the architecture, each with its own agents, controllers, databases and APIs. Levels work with their own primitives and are independent of each other. The connection between the levels is provided by Application (APP).

This modernization simplified the development of the Control plane and helped to natively get rid of the problems characteristic of Neutron. In SPRUT:

• agents receive information about Data plane settings;
  
• events are not lost due to abandoning the event model;
  
• Data plane is easier and more flexible to configure.
  

Technical information

Arrangement of SPRUT architectural layers

In the SPRUT architecture, each layer has its own characteristics.

SDN. The layer consists of SDN agents running on computing nodes, an SDN controller and an SDN API. The interaction occurs as follows:

1. After launch, the agents create virtual switches on their hosts and connect them to each other with tunnels
   
   Tunnels can have any configuration: full connectivity, partial, separated regions or another.
2. After creating tunnels, agents report this to the controller. At the same time, agents monitor the tunnels, sending data packets through them. Information about the state of the tunnels in real time is transmitted to the controller, which builds a network graph.
   
   The vertices of the graph are switches, the edges are constructed tunnels. The graph is constantly updated taking into account current information
3. After this, SDN is ready to create primitives. It works with two: Link and Endpoint. Link is an entity connecting two Endpoints. Endpoint - connection point to a system of switches. Only one Link connects to one Endpoint, and a Link connects no more than two Endpoints. 
   
4. When creating a connection between nodes, two Endpoints are launched on them via the API. The addition of events to the API and database begins. Then a Link is built between the Endpoint, after which the controller determines the shortest path for transmitting traffic from one node to the second. To do this, it uses a network graph and Dijkstra's algorithm.
   

All switches on the selected route receive traffic rules indicating the sequence of ports. Subsequently, the rules are loaded into agents, which build a route on real switches. In part, this scheme resembles the operation of the OSPF routing protocol. 

This interaction format ensures speed and eliminates errors during traffic transmission. But SDN is not the only layer in SPRUT, because using Link and Endpoint you can only connect two virtual machines. This is not enough to create a full-fledged virtual network - you need network entities for which the NFV layer is responsible. 

NFV layer device. The design of the NFV layer is somewhat similar to SDN:



NFV-level agents may not be located on all nodes, but only on those selected by the operations team: this helps save network resources of virtual machines. The NFV layer operates DHCP, Switch, Router, Metadata Proxy and other primitives, the number of functions is constantly increasing. The level components interact as follows:

1. When launched, NFV agents create the number of entities determined by the operation command. 
   
   Entities are created empty and not used by anyone
2. Agents transmit information to the controller. After this, the layer is ready to serve user requests.
   

The principle of joint operation of the SDN and NFV layers

In practice, creating a virtual network using SPRUT occurs as follows:

1. A user who wants to create a virtual network makes a request to the Application layer. 
   
2. The Application layer redirects received requests and transforms them into a set of entities of the SDN and NFV layers.
   
3. Unused entities are taken from the NFV layer and connected to each other using Link and Endpoint. This network is still missing only virtual machines.
   
   
4. Next, the user accesses the OpenStack service, which is responsible for virtual machines. The user's request reaches the Nova-compute agent, which starts the QEMU process.
   
   
5. Simultaneously with the launch of QEMU, the Nova-compute agent contacts the Application layer to connect the virtual machine to the network. In response to this, the Application layer creates a set of Links and Endpoints to connect the virtual machine to the switch. This completes the creation of a virtual network. 

In its full version, the SPRUT circuit is more extensive and includes more components:

General diagram of SPRUT components. We do not list all of them, so that the article does not become completely astronomical in size; we touch only on the most important

The Application layer in SPRUT is a custom plugin for Neutron that replaces the ML2 plugin. It is responsible for converting Neutron entities into the required components. Other SDNs use a similar approach. In addition, for stable work with Nova, Sprut provides a Nova Driver.

History of SDN development in VK Cloud: today and plans for tomorrow

August 22, 2023 marks the 10th anniversary of one of the most successful SDNs in history - OpenDaylight. Over the years, developers and architects have improved scalability, thrown out a lot of Legacy code, and then plan to deal with compatibility: a lot has been done, and now we need to make sure that they will not interfere with each other. 

It also became obvious that the transition to a modular microservice architecture is a step that needs to be taken for the development of the product. Microservice architecture allows you to be more flexible with hosting tools, such as containerization. The result is scalability, manageability and the ability to reduce the connectivity of components.

Our Sprut is younger. We are very pleased that we were able not only to take advantage of the work of our comrades from the industry, but also to improve the product. As a result, Sprut received a number of important features, including:

• microservice architecture,
  
• OpenVSwitch as Data plane,
  
• integration with Nova and Neutron,
  
• implementation and use of a closed control loop,
  
• small code base - about 15,000 lines.
  

What’s important is that development to production took less than a year. In the future, we plan to add to the product and will definitely tell you what we got.

The Highload++ conference will be held in Moscow on November 27–28. SDN Sprut developers will speak at it report, in which they will talk about the implementation of EVPN in VK Cloud and the problems of vendor solutions.