Creating a file using ovs-vsctl

Okay as you saw in last post, we have come up with a set of approaches to solve our problem statement. In this blog we will tell you the modifications we did in OpenVswitch for supporting multiple controllers.
 
     We modified the 'ovs-vsctl' file in order to give a new command-line argument support. The new argument was "net-split" on specifying which a function called as "cmd_split_network()" will be called. The command is called from Mininet like this:

self.cmd( 'ovs-vsctl net-split' , obj )

Here ovs-vsctl is being called with an argument net-split and passing an object which contains the details of  how the network should be split. In OpenVswitch they have used callback functions to be called whenever a command-line is being specified. So we had to register the function "cmd_split_network()' we wrote with the 'net-split' command.

 {"net-split", 1, 1, NULL, cmd_split_network, NULL, "", RW}

So when Mininet calls ovs-vsctl with net-split argument, the function 'cmd_split_network()' is called. This function creates a file in which we will store the details of network split i.e., a table-like data will be stored which the switch can refer to when a new packet comes in and then find out which controller it has to talk to depending on the source address of the incoming packet.


Wow..if only life was that simple!! We have modified ovs-vsctl to create a file which will store the data sent by Mininet. But the problem here is that when Mininet creates processes for switches and hosts, it gives each process a separate code-space but they all share they same file system. So when a file for forwarding is created by a switch, all switches share the same file. We can solve this by appending each switches forwarding instructions in each row but then there is another problem. There is clarity between  the commonality of how both Mininet and OpenVswitch refer or distinguishes each switch so that a switch will look at only its row in the file and know which controller to talk to.
   As we now, we are trying to solve the above problem. You will see how much we succeed in our next post.

Moving ahead with OpenVswitch

Hi guys..sorry for a giving a long gap after the last post, we had our final exams so we couldn't do much.

   Okay so where are we? In the previous blog we told you the we are shifting to OpenVswitch part in order to support multiple controllers. As you all know, Mininet makes calls to the underlying OpenVswitch by passing the commands to the 'cmd()' method.

Example:  self.cmd( 'ovs-vsctl add-br', self ).  Here 'ovs-vsctl add-br' is a command and self is the parameter you are passing to the command.

    Let me explain what is our goal and what is the approach we have decided to take-

Our goal: We want each switch to talk to a particular controller depending on the source address of the incoming packet. An example will make this much clear- Assume there are two networks represented by network numbers, n1 and n2. Both these networks are connected to switch, s1, which is connected to, say, two controllers, c1 and c2. Now what we want to achieve is that whenever a new packet comes from, say, network n1, we want the switch to contact controller c1 in order to obtain the flow; similarly when new packet comes from network n2, the switch should contact controller c2. This was we are trying to split the network and making the switch contact different controller for different packets.

Our approach: After some brain storming sessions, we have decided on a way to specify the switch about the way to split the network. As I told before, the commands to switch are sent via 'cmd()' in Mininet. Yes, we added a new command line argument to specify the network splitting in Mininet as explained in previous post, but what about specifying the OpenVswitch? How will the switch know which controller does is have to talk to? Moreover, when should the switch make the decision of sending a packet to a controller?

    Here are the solutions we have decided to provide for all the above questions:

Firstly, we have to give a new command line argument for one of the commands on OpenVswitch. Mininet will call the command along with the command line argument, notifying OpenVswitch that the user wants to split the network.

Secondly, we decided to provide some form of table which can be referred by the switch whenever a new packet comes in and the table would contain information like n1:c1 specifying which controller to talk to for a given network. The progress about this will be explained in the next post.

    Thirdly, we have to look-up this table when a packet-in action occurs and there is no flow entry for that packet, as in it is a new packet.

But if you have tried to see how source code of OpenVswitch looks, it is all written in C!! Till now we looked at source code of Mininet written in Python, now we have to shift to C, which is going to cause quite some trouble. And also OpenVswitch is itself an open-source project which is really very huge! You can clone the source code by the directions given here. Next blog, we will tell you what are the modifications done in OpenVswitch.

 

Execution and Surprises

We decided to filter the packets based on the network number of the source packet as a first step. To do that we need to enhance the source of openvswitch and also Mininet. Why Mininet? It's because Mininet is like an interface between the users and the switch. We are going with the following command in the Mininet for specifying about the network split.

sudo python mn --controller=remote,192.168.1.10,192.168.1.11 --multictrl=s1:c0@c1,s2:c2 --netsplit=c0:172.16.10.0/24@172.16.11.0/24,c1:172.16.12.0/24

Why '@' for specifying multiple network numbers? Well, it's just for a start. We will make it to ':' soon as even we find it difficult to remember the input sequence. :P

Doing what we did for multictrl previously
Coming through, the following line is added in the file bin/mn class MininetRunner's parseArgs method.

    opts.add_option( '--netsplit', '-s', type='string', default=False,
        help='set the network splitting for multiple controllers')

Similary, the following line in the begin method of MininetRunner class.

    networkSplit = self.options.netsplit

and another parameter in the Net instance creation

    networkSplit=networkSplit

Going into the file mininet/net.py we added another parameter in the init method of the Mininet class, ie, networkSplit=False.

Knock knock, who is it? Redundancy!
Now, our objective is to prepare an associative array of controller to list of network numbers. But wait! Aren't we familiar with this operation? Yes we are! We did the same thing for switchControlPair method in the following post. So, as we learned about the importance of redundancy while going through the source, we were delighted to experience it as we wouldn't have to rethink or rewrite the code (except for renaming the method from switchControlPair to pairGenerate for being generic).

As of now this is where we have stopped in Mininet. We are looking into openvswitch to see how source IP could be put into use for deciding the controller it should forward to. And also, we will use ovs-vsctl command for creating files containing the network splitting information via OpenVSwitch. We will post about the changes made in the OpenVSwitch in the coming post for the file creation and then come back to Mininet as it will use the same commands that we wrote for OpenVSwitch.

So till then, adios Mininet! See ya later.

Controller or OpenVswitch?

As you guys saw in the last blog, our modified Mininet can have switches connected to multiple controllers based on the input the user specifies. But there is a problem here!

Look at this output:

[...]

h1 ping h2

PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.

64 bytes from 10.0.0.2: icmp_req=1 ttl=64 time=104 ms

64 bytes from 10.0.0.2: icmp_req=1 ttl=64 time=105 ms (DUP!)

64 bytes from 10.0.0.2: icmp_req=1 ttl=64 time=143 ms (DUP!)

64 bytes from 10.0.0.2: icmp_req=1 ttl=64 time=143 ms (DUP!)

64 bytes from 10.0.0.2: icmp_req=2 ttl=64 time=0.432 ms

64 bytes from 10.0.0.2: icmp_req=3 ttl=64 time=0.058 ms

[...]

What is this (DUP!) packets?
 It means the reply packet is a duplicate. Each ICMP packet sent out to ping a host has a sequence number that is returned with the response.
   The reason why we are getting duplicate packets is because when host,h1, pings host, h2, the ping packets is sent to switch S1. Switch, by default forwards this new packet to all the connected controllers as it does not have a flow entry for this new packet. The controllers are configured in a way that both of them give the same flow. So the host is getting multiple packets consisting of same flow thus generating a DUP packet.


Well, this is not a good sign of progress :( For one thing we are getting duplicate packets, which is generally considered as a phenomenon of faulty networks, and also when you think about it, there is a lot of bandwidth wasted. In the example, switch was connected only to 2 controllers but imagine if it was connected to 100 controllers and for every new packet, it is broadcasting to all those controllers!! That will be one heck of a nightmare.

    In order to solve this problem, we have to either go to the controller and build a mechanism where in the actions specify that switch should talk to specific controller for specific packets. Or we have to modify OpenVswitch such that, given a packet, it will decide which controller to forward to based on the specifications we configure it with. So we are going the latter approach i.e., we are diving into OpenVswitch and modifying it such that whenever a packet in action takes place, the switch will know which controller to send it; thus avoiding the broadcasting.

In the next blog, you will see how we are going to divide the network and configure OpenVswitch such that it will forward each packet to  single controller.

Modifying the Source Code

Previously we handled the error that we faced in the Mininet constructor. Following are the 2 information that is needed by Mininet software in order to establish a connection to multiple controllers:

• Controllers to which Mininet should connect
• Which switches are connected to which controllers

We will describe point by point and see what we did to achieve it.

1. Specifying multiple remote controllers for Mininet

You mention about the remote controller using --controller=remote,<ip_addr_of_controller>. We thought of extending the same method for specifying multiple IP addresses. Hence, the format we have specified is:
--controller=remote,<ipaddr1>,<ipaddr2>,<ipaddr3>" and so on.
Now, since there is a variation in the input style, we also have to manage the underlying code.

After the command-line parsing in parseArgs() method, the string passed in the RHS of = in the command-line is made available via self.options.controller. We modified the controller creation code with the following:

So,
controller = customConstructor( CONTROLLERS, self.options.controller )

was transformed to:

[...]
if self.options.controller.split(',')[0] == 'remote':
    controller = list()
    length = len(self.options.controller.split(',')) - 1
    addresses = self.options.controller.split(',')[1:]
    for i in range (0, length):
        controller.append(customConstructor( CONTROLLERS, 'remote,%s' % addresses[i] ) )
else:
    controller = customConstructor( CONTROLLERS, self.options.controller )

[...]

With no intention to disturb the underlying method of customConstructor, we decided to create the controllers one at a time in a loop. Therefore, we first split the left most argument to check if it is remote. If it isn't, it would go back as how normally the controllers are created using customConstructor. If it was remote controller, we further segragate the remaining ip addresses passed by
addresses = self.options.controller.split(',')[1:]
and then append each address into customConstructor in the loop using
controller.append(customConstructor( CONTROLLERS, 'remote,%s' % addresses[i] ) )
Well, this way it's easy to test too. If the user passed a single address, then controller is just an instance of type customConstructor. If he passed multiple addresses, then controller is a list.

2. Switch to Controller association

For this association, we added multictrl as a command-line input. So a typical multictrl command would appear as:

--multictrl=s1:c0@c1,s2:c2 and so on.

Rules:

• Use a switch name first followed by a colon, :, and then the controller name.
• To mention 2 or more controllers then use '@' in between, like, s1:c1@c2@c3
• c0, c1 are actually controllers in which the remote controllers are written.
  For eg: --controller=remote,192.168.1.2,192.168.1.3, c0 will take the value as 192.168.1.2 and c1 will take the value as 192.168.1.3

Everything in the RHS of = in multictrl command-line is passed into the __init__ of the Mininet class. Now we have to parse the input and create a dictionary as described in the previous post. So, we added the following line into the __init__ method:
self.multiCtrl = switchControlPair( multiCtrl ) if multiCtrl else False

and defined switchControlPair in util.py, where all utility methods are written. The switchControlPair will convert the string s1:c0@c1,s2:c1 into a dictionary as {'s1':['c0', 'c1'], 's2':['c1']} and return this dictionary. This returned dictionary is of not much use to us, we have to create an association between the switch and the controller object, not string. So we will setup another associative array where we will have the controller associated to switch name. So, in the addController method of the same class Mininet, we added the following lines:

[...]
if self.multiCtrl is not False:
    for switchName, controlList in self.multiCtrl.items():
        switchKeys = self.switchToCtrl.keys()
        if name in controlList:
            if switchName not in switchKeys:
                self.switchToCtrl[ switchName ] = list()
            self.switchToCtrl[switchName].append(controller_new)
[...]

In the above lines, we collect switchName will be the key and controlList will be the list of controller names. We have another empty associative array called self.switchToCtrl defined in the __init__. We then check the existence of the controller object's "name" created in this method with those in that controlList. If they exist then we check if switchName exists in the keys of switchToCtrl dictionary, switchKeys. If they don't, we create a list() against the switch name, and start appending.

In buildFromTopo() method where addController is called, there is a comparison that was already present in the source to check if the controllers taken from user is a list or not, if not then make it to a list. So since we passed the controller as a list in case of multiple remote controllers, we don't have to bother much.

[...]
   #the snippet converting the controller variable to a list
   classes = self.controller
   if type( classes ) is not list:
       classes = [ classes ]
   for i, cls in enumerate( classes ):
       self.addController( 'c%d' % i, cls )

[...]

The for loop then iterates to add those many controllers in the list.
We know that the changes made is consuming more memory, as in, we are associating a copy of the controller object into switchToCtrl. Also we know that we are doing double work of first creating an associative array of strings to strings and using the same for strings to controller objects. Better memory usage and implementation will be managed in further iterations when we confirm that our logic is meeting the results.

The next post will address the execution of the changes made in the code and their results.

and we attack!

Hi guys..so now that you know what our project is about and thus, this blog is about, I think it’s time we start making progress towards our aim of having multiple controllers. You would have gotten some idea as to how to control flows in Mininet from the previous post. Well then, we don't have to worry for writing code to enable multiple-controller in OpenVSwitch's source since we know that it supports a switch talking to multiple controllers. We hopefully won't be leaving Mininet at all during the development.

We intend to create a dictionary of lists in order to store the Switch:Controller associations. The keys of the dictionary will be the switch names as in s1, s2, etc. Each key will map to a list of remote controller ip-addresses and probably their port numbers too for establishing a connection between the switches and their assigned controllers.

So, in python our variable would look like:

mulitpleController = {
                       's1':['127.0.0.1','128.0.0.1','129.0.0.1'],
                       's2':['127.0.0.1','130.0.0.1']
                       's3':['130.0.0.1','129.0.0.1']
                     }

This dictionary will be passed to the function where the controller-switch connection is established.

 So, we thought of how to attack the problem, and we needed to add an option to allow the users to specify the connectivity of switch to controllers via command-line or through a config file. Command-line seemed to be a good way to start as it is convenient to specify. For this, we have added a new command-line argument, --multictrl, which stands for multiple-controllers. Now if someone wants to specify which controller should talk to which switch, the command will look something like this: 

sudo mn --mutictrl=s1:c1,s2:c2,s3:c3

Now lets get into more interesting topic of all the trouble we faced in order to make this happen and how we resolved it :P

• Getting the input through command line and storing it in a variable called ‘multictrl’ in the file mn was not all that difficult. We added an option in parseArgs()
  [...]
      opts.add_option( '--multictrl', '-m', type='string', default=False,

                                              help='use switches with multiple controllers' )

[...]

          This will enable parsing with multictrl command line argument during execution. Later we can access the string using self.options.multictrl.

• Once we got that, we had to pass it to the constructor of the class to which ‘Net’ object belonged to (i.e., either Mininet or MininetWithControlNet). We passed it like this in mn file:

[...]

         mn = Net(

                 topo=topo,

                 switch=switch, host=host, controller=controller,

                 link=link,

                 ipBase=ipBase,

                 inNamespace=inNamespace,

                 xterms=xterms, autoSetMacs=mac,

                 autoStaticArp=arp, autoPinCpus=pin,

                 listenPort=listenPort,

                 multictrl=multictrl                 

                  )

        [...]

  

In the net.py file, there is a contructor __init__() inside Mininet class which accepts the passed keyword arguments. So we added another keyword parameter multictrl and gave it a default value of None. And we wrote a method which would parse the given input, make and return a dictionary object which would hold the associations of Controllers and Switches. The returned dictionary would be stored in a data member called self.multiCtrl.

Now when we ran the program, the program caught an exception :

TypeError: __init__() got an unexpected keyword argument 'multictrl'

So the __init__() method would not accept the newly added parameter! In order to solve this, quite obviously, the first thing we did was we Google-d the error and shockingly we did not get any clear result which would tell us what exactly this error was or how to resolve it. So we read quite a bit about keyword arguments, what is it, how is it done, what are the different methods we can pass the keyword arguments in and things like that. Later we tried to pass our ‘multictrl’ from ‘begin()’ method to ‘__init__()’ method by using *args and **kwargs  method i.e.,

[...]

def __init__( self,topo=None, switch=OVSKernelSwitch, host=Host,

                 controller=Controller, link=Link, intf=Intf,

                 build=True, xterms=False, cleanup=False, ipBase='10.0.0.0/8',

                 inNamespace=False,

                 autoSetMacs=False, autoStaticArp=False, autoPinCpus=False,

                 listenPort=None, *args, **kwargs):

[...]

In this, the *kwargs argument hold all the unmatched key:value pairs of keyword arguments. So we were hoping that our input would be stored in this at least. But once we ran, we ended up getting the same error!!

Actually, the main cause of all the problems which we were facing was the self.multiCtrl data member. We noticed how other keyword arguments are getting stored while multictrl is failing. It came to our notice that the other keyword arguments and their respective data members to which they were assigned were just the same. Each character’s case matched exactly with that of the data member.

[...]

def __init__( self, topo=None, switch=OVSKernelSwitch, host=Host,

              controller=Controller, link=Link, intf=Intf,

              build=True, xterms=False, cleanup=False, ipBase='10.0.0.0/8',

              multictrl=False, inNamespace=False,

              autoSetMacs=False, autoStaticArp=False, autoPinCpus=False,

              listenPort=None):

self.topo = topo

    self.switoch = switch

    self.host = host

    self.controller = controller

    self.link = link

    self.intf = intf

    self.ipBase = ipBase

    self.ipBaseNum, self.prefixLen = netParse( self.ipBase )

    self.nextIP = 1  # start for address allocation

    self.inNamespace = inNamespace

    self.xterms = xterms

    self.cleanup = cleanup

    self.autoSetMacs = autoSetMacs

    self.autoStaticArp = autoStaticArp

    self.autoPinCpus = autoPinCpus

    self.numCores = numCores()

    self.nextCore = 0  # next core for pinning hosts to CPUs

    self.listenPort = listenPort

    self.multiCtrl = switchControlPair(multictrl) if multictrl else False
           #^the line causing error

[...]

On changing the keyword argument such that it exactly matched with the data member, the program did not encounter any errors and we could neatly run with whatever changes we had made in the code.

The CRUX of our Project

After a brief introduction to SDN and followed by mininet, lets dive into the main objective of our project. In this particular post we would introduce you to the problem statement and the solution we are trying to provide in order to extend the capabilities of a switch in Mininet.

In SDN we came across the concept of controllers and switches, their interaction with each other through the openFlow protocol. In the first network topology we have seen switches connected to each other resulting in flooding and broadcast storm which was solved by a single controller connected to multiple switches as show in our previous post related to Flooding and Broadcast storm.

/*EDITED*/Now if the controller fails, it will go into emergency mode providing emergency flows.So how can we change or modify the network in order to make our openVswitches more intelligent? That is the CRUX of our project.

1. First approach would be introduction of more controllers such that each controller is connected to multiple switches.
2. Second would be to modify the switch such that multiple controllers can be connected to a single switch.

Approach 1

As the first approach is already implemented and tested in Mininet 2.1, we would be going with the second approach.

Now at this stage, you will be having a lot of questions about - How will the topology look like? What are the advantages? And how are we implementing this? Are there any drawbacks or extensions?

Lets answer them, since Mininet uses OpenVSwitch, it is necessary to first check if the service of multiple controller is provided by the OpenVSwitch. As per the following link it is clear that OpenVSwitch does have a support for multiple controllers.

Hence we can GO ahead with our second approach.

With respect to the benefits, there are two major advantages compared to the switches talking to a single controller:

1. Decentralization: As we have multiple controllers talking to a switch, there are multiple flow entries provided by different controllers to the switch. If one of the controller fails we can still have the network working which is not possible in the older case. This solves the problem of single point failure.
2. Load Balancing: As the packet is sent to multiple controllers, we can expect a fast response from at least one of the controllers whereas in the other case a single controller is bombarded with packets from different switches, hence there is a high chance of delay in the response.

The next section will cover the implementation aspect related to mininet and the process of how we are gonna ATTACK !!