ovs-test: A new tool that allows to diagnose connectivity and performance issues

This tool will be a replacement for the current ovs-vlan-test
utility. Besides from connectivity issues it will also be able
to detect performance related issues in Open vSwitch setups.
Currently it uses UDP and TCP protocols for stressing.

Issue #6976

4 files changed, 310 insertions, 0 deletions

diff --git a/utilities/automake.mk b/utilities/automake.mk
index 420d5fc3..df94dd1e 100644
--- a/utilities/automake.mk
+++ b/utilities/automake.mk
@@ -9,6 +9,7 @@ if HAVE_PYTHON
 bin_SCRIPTS += \
 	utilities/ovs-pcap \
 	utilities/ovs-tcpundump \
+	utilities/ovs-test \
 	utilities/ovs-vlan-test
 endif
 noinst_SCRIPTS += utilities/ovs-pki-cgi
@@ -23,6 +24,7 @@ EXTRA_DIST += \
 	utilities/ovs-pki.in \
 	utilities/ovs-save \
 	utilities/ovs-tcpundump.in \
+	utilities/ovs-test.in \
 	utilities/ovs-vlan-test.in
 MAN_ROOTS += \
 	utilities/ovs-appctl.8.in \
@@ -36,6 +38,7 @@ MAN_ROOTS += \
 	utilities/ovs-pki.8.in \
 	utilities/ovs-tcpundump.1.in \
 	utilities/ovs-vlan-bug-workaround.8.in \
+	utilities/ovs-test.8.in \
 	utilities/ovs-vlan-test.8.in \
 	utilities/ovs-vsctl.8.in
 MAN_FRAGMENTS += utilities/ovs-vlan-bugs.man
@@ -55,6 +58,8 @@ DISTCLEANFILES += \
 	utilities/ovs-pki.8 \
 	utilities/ovs-tcpundump \
 	utilities/ovs-tcpundump.1 \
+	utilities/ovs-test \
+	utilities/ovs-test.8 \
 	utilities/ovs-vlan-test \
 	utilities/ovs-vlan-test.8 \
 	utilities/ovs-vlan-bug-workaround.8 \
@@ -71,6 +76,7 @@ man_MANS += \
 	utilities/ovs-pki.8 \
 	utilities/ovs-tcpundump.1 \
 	utilities/ovs-vlan-bug-workaround.8 \
+	utilities/ovs-test.8 \
 	utilities/ovs-vlan-test.8 \
 	utilities/ovs-vsctl.8
 dist_man_MANS += utilities/ovs-ctl.8
diff --git a/utilities/ovs-test.8.in b/utilities/ovs-test.8.in
new file mode 100644
index 00000000..afc8221e
--- /dev/null
+++ b/utilities/ovs-test.8.in
@@ -0,0 +1,117 @@
+.TH ovs\-test 1 "October 2011" "Open vSwitch" "Open vSwitch Manual"
+.
+.SH NAME
+\fBovs\-test\fR \- check Linux drivers for performance and vlan problems
+.
+.SH SYNOPSIS
+\fBovs\-test\fR \fB\-s\fR \fIport\fR
+.PP
+\fBovs\-test\fR \fB\-c\fR \fIserver1\fR
+\fIserver2\fR [\fB\-b\fR \fIbandwidth\fR]
+.so lib/common-syn.man
+.
+.SH DESCRIPTION
+The \fBovs\-test\fR program may be used to check for problems sending
+802.1Q traffic that Open vSwitch may uncover. These problems can
+occur when Open vSwitch is used to send 802.1Q traffic through physical
+interfaces running certain drivers of certain Linux kernel versions. To run a
+test, configure Open vSwitch to tag traffic originating from \fIserver1\fR and
+forward it to the \fIserver2\fR. On both servers run \fBovs\-test\fR
+in server mode. Then, on any other host, run the \fBovs\-test\fR in client
+mode. The client will connect to both \fBovs\-test\fR servers and schedule
+tests between them. \fBovs\-test\fR will perform UDP and TCP tests.
+.PP
+UDP tests can report packet loss and achieved bandwidth, because UDP flow
+control is done inside \fBovs\-test\fR. It is also possible to specify target
+bandwidth for UDP. By default it is 1Mbit/s.
+.PP
+TCP tests report only achieved bandwidth, because kernel TCP stack
+takes care of flow control and packet loss. TCP tests are essential to detect
+potential TSO related VLAN issues.
+.PP
+To determine whether Open vSwitch is encountering any 802.1Q related problems,
+the user must compare packet loss and achieved bandwidth in a setup where
+traffic is being tagged against one where it is not. If in the tagged setup
+both servers are unable to communicate or the achieved bandwidth is lower,
+then, most likely, Open vSwitch has encountered a pre-existing kernel or
+driver bug.
+.PP
+Some examples of the types of problems that may be encountered are:
+.so utilities/ovs-vlan-bugs.man
+.
+.SS "Client Mode"
+An \fBovs\-test\fR client will connect to two \fBovs\-test\fR servers and
+will ask them to exchange traffic.
+.
+.SS "Server Mode"
+To conduct tests, two \fBovs\-test\fR servers must be running on two different
+hosts where client can connect. The actual test traffic is exchanged only
+between both \fBovs\-test\fR server test IP addresses. It is recommended that
+both servers have their test IP addresses in the same subnet, otherwise one
+will need to change routing so that the test traffic actually goes through the
+interface that he originally intended to test.
+.
+.SH OPTIONS
+.
+.TP
+\fB\-s\fR, \fB\-\-server\fR \fIport\fR
+Run in server mode and wait for a client to establish XML RPC Control
+Connection on TCP \fIport\fR. It is recommended to have ethtool installed on
+the server so that it could retrieve information about NIC driver.
+.TP
+\fB\-c\fR, \fB\-\-client\fR \fIserver1\fR \fIserver2\fR
+Run in client mode and schedule tests between \fIserver1\fR and \fIserver2\fR,
+where each \fIserver\fR must be given in following format -
+ControlIP[:ControlPort][,TestIP[:TestPort]]. If TestIP is omitted then
+ovs-test server will use the ControlIP for testing purposes. ControlPort is
+TCP port where server will listen for incoming XML/RPC control
+connections to schedule tests (by default it is 15531). TestPort
+is port which will be used by server to listen for test traffic
+(by default it is 15532).
+.TP
+\fB\-b\fR, \fB\-\-bandwidth\fR \fIbandwidth\fR
+Target bandwidth for UDP tests. The \fIbandwidth\fR must be given in bits per
+second. It is possible to use postfix M or K to alter the target bandwidth
+magnitude.
+.
+.so lib/common.man
+.SH EXAMPLES
+.PP
+Set up a bridge which forwards traffic originating from \fB1.2.3.4\fR out
+\fBeth1\fR with VLAN tag 10.
+.IP
+.B ovs\-vsctl \-\- add\-br vlan\-br \(rs
+.IP
+.B \-\- add\-port vlan\-br eth1 \(rs
+.IP
+.B \-\- add\-port vlan\-br vlan\-br\-tag tag=10 \(rs
+.IP
+.B \-\- set Interface vlan\-br\-tag type=internal
+.IP
+.B ifconfig vlan\-br\-tag up 1.2.3.4
+.
+.PP
+On two different hosts start \fBovs\-test\fR in server mode and tell them to
+listen on port 15531 for incoming client control connections:
+.IP
+.B 1.2.3.4: ovs\-test \-s 15531
+.IP
+.B 1.2.3.5: ovs\-test \-s 15531
+.
+.PP
+On any other host start \fBovs\-test\fR in client mode and ask it to connect
+to those two servers - one at 1.2.3.4 and another at 1.2.3.5 (by default
+client will use TCP port 15531 to establish control channel).
+.IP
+.B ovs\-test -c 1.2.3.4 1.2.3.5
+.
+.TP
+
+.SH SEE ALSO
+.
+.BR ovs\-vswitchd (8),
+.BR ovs\-ofctl (8),
+.BR ovs\-vsctl (8),
+.BR ovs\-vlan\-test (8),
+.BR ethtool (8),
+.BR uname (1)
diff --git a/utilities/ovs-test.in b/utilities/ovs-test.in
new file mode 100644
index 00000000..6518dbcb
--- /dev/null
+++ b/utilities/ovs-test.in
@@ -0,0 +1,180 @@
+#! @PYTHON@
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at:
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+ovs test utility that allows to do tests between remote hosts
+"""
+
+import twisted
+import xmlrpclib
+import time
+import socket
+import math
+from ovstest import args, rpcserver
+
+
+def bandwidth_to_string(bwidth):
+    """Convert bandwidth from long to string and add units"""
+    bwidth = bwidth * 8 #  Convert back to bits/second
+    if bwidth >= 10000000:
+        return str(int(bwidth / 1000000)) + "Mbps"
+    elif bwidth > 10000:
+        return str(int(bwidth / 1000)) + "Kbps"
+    else:
+        return str(int(bwidth)) + "bps"
+
+
+def collect_information(node):
+    """Print information about hosts that will do testing"""
+    print "Node %s:%u " % (node[0], node[1])
+    server1 = xmlrpclib.Server("http://%s:%u/" % (node[0], node[1]))
+    interface_name = server1.get_interface(node[2])
+    uname = server1.uname()
+    mtu = 1500
+
+    if interface_name == "":
+        print ("Could not find interface that has %s IP address."
+               "Make sure that you specified correct Test IP." % (node[2]))
+    else:
+        mtu = server1.get_interface_mtu(interface_name)
+        driver = server1.get_driver(interface_name)
+        print "Will be using %s(%s) with MTU %u" % (interface_name, node[2],
+                                                    mtu)
+        if driver == "":
+            print "Install ethtool on this host to get NIC driver information"
+        else:
+            print "On this host %s has %s." % (interface_name, driver)
+
+    if uname == "":
+        print "Unable to retrieve kernel information. Is this Linux?"
+    else:
+        print "Running kernel %s." % uname
+    print "\n"
+    return mtu
+
+
+def do_udp_tests(receiver, sender, tbwidth, duration, sender_mtu):
+    """Schedule UDP tests between receiver and sender"""
+    server1 = xmlrpclib.Server("http://%s:%u/" % (receiver[0], receiver[1]))
+    server2 = xmlrpclib.Server("http://%s:%u/" % (sender[0], sender[1]))
+
+    udpformat = '{0:>15} {1:>15} {2:>15} {3:>15} {4:>15}'
+
+    print ("UDP test from %s:%u to %s:%u with target bandwidth %s" %
+                            (sender[0], sender[1], receiver[0], receiver[1],
+                             bandwidth_to_string(tbwidth)))
+    print udpformat.format("Datagram Size", "Snt Datagrams", "Rcv Datagrams",
+                            "Datagram Loss", "Bandwidth")
+
+    for size in [8, sender_mtu - 100, sender_mtu - 28, sender_mtu]:
+        listen_handle = -1
+        send_handle = -1
+        try:
+            packetcnt = (tbwidth * duration) / size
+
+            listen_handle = server1.create_udp_listener(receiver[3])
+            if listen_handle == -1:
+                print ("Server could not open UDP listening socket on port"
+                        " %u. Try to restart the server.\n" % receiver[3])
+                return
+            send_handle = server2.create_udp_sender(
+                                            (receiver[2], receiver[3]),
+                                            packetcnt, size, duration)
+
+            #Using sleep here because there is no other synchronization source
+            #that would notify us when all sent packets were received
+            time.sleep(duration + 1)
+
+            rcv_packets = server1.get_udp_listener_results(listen_handle)
+            snt_packets = server2.get_udp_sender_results(send_handle)
+
+            loss = math.ceil(((snt_packets - rcv_packets) * 10000.0) /
+                                                        snt_packets) / 100
+            bwidth = (rcv_packets * size) / duration
+
+            print udpformat.format(size, snt_packets, rcv_packets,
+                                '%.2f%%' % loss, bandwidth_to_string(bwidth))
+        finally:
+            if listen_handle != -1:
+                server1.close_udp_listener(listen_handle)
+            if send_handle != -1:
+                server2.close_udp_sender(send_handle)
+    print "\n"
+
+
+def do_tcp_tests(receiver, sender, duration):
+    """Schedule TCP tests between receiver and sender"""
+    server1 = xmlrpclib.Server("http://%s:%u/" % (receiver[0], receiver[1]))
+    server2 = xmlrpclib.Server("http://%s:%u/" % (sender[0], sender[1]))
+
+    tcpformat = '{0:>15} {1:>15} {2:>15}'
+    print "TCP test from %s:%u to %s:%u (full speed)" % (sender[0], sender[1],
+                                                    receiver[0], receiver[1])
+    print tcpformat.format("Snt Bytes", "Rcv Bytes", "Bandwidth")
+
+    listen_handle = -1
+    send_handle = -1
+    try:
+        listen_handle = server1.create_tcp_listener(receiver[3])
+        if listen_handle == -1:
+            print ("Server was unable to open TCP listening socket on port"
+                    " %u. Try to restart the server.\n" % receiver[3])
+            return
+        send_handle = server2.create_tcp_sender(receiver[2], receiver[3],
+                                                    duration)
+
+        time.sleep(duration + 1)
+
+        rcv_bytes = long(server1.get_tcp_listener_results(listen_handle))
+        snt_bytes = long(server2.get_tcp_sender_results(send_handle))
+
+        bwidth = rcv_bytes / duration
+
+        print tcpformat.format(snt_bytes, rcv_bytes,
+                               bandwidth_to_string(bwidth))
+    finally:
+        if listen_handle != -1:
+            server1.close_tcp_listener(listen_handle)
+        if send_handle != -1:
+            server2.close_tcp_sender(send_handle)
+    print "\n"
+
+
+if __name__ == '__main__':
+    try:
+        ovs_args = args.ovs_initialize_args()
+
+        if ovs_args.port is not None: #  Start in server mode
+            print "Starting RPC server"
+            try:
+                rpcserver.start_rpc_server(ovs_args.port)
+            except twisted.internet.error.CannotListenError:
+                print "Couldn't start XMLRPC server on port %u" % ovs_args.port
+
+        elif ovs_args.servers is not None: #  Run in client mode
+            node1 = ovs_args.servers[0]
+            node2 = ovs_args.servers[1]
+            bandwidth = ovs_args.targetBandwidth
+
+            mtu_node1 = collect_information(node1)
+            mtu_node2 = collect_information(node2)
+
+            do_udp_tests(node1, node2, bandwidth, 5, mtu_node1)
+            do_udp_tests(node2, node1, bandwidth, 5, mtu_node2)
+            do_tcp_tests(node1, node2, 5)
+            do_tcp_tests(node2, node1, 5)
+    except KeyboardInterrupt:
+        pass
+    except socket.error:
+        print "Couldn't establish XMLRPC control channel"
diff --git a/utilities/ovs-vlan-test.8.in b/utilities/ovs-vlan-test.8.in
index 602d785d..549dcad7 100644
--- a/utilities/ovs-vlan-test.8.in
+++ b/utilities/ovs-vlan-test.8.in
@@ -8,6 +8,12 @@
 .so lib/common-syn.man
 .
 .SH DESCRIPTION
+The \fBovs\-vlan\-test\fR utility has some limitations, for example, it does
+not use TCP in its tests. Also it does not take into account MTU to detect
+potential edge cases. To overcome those limitations a new tool was
+developed \- \fBovs\-test\fR. \fBovs\-test\fR is currently supported only
+on Debian so, if possible try to use that on instead of \fBovs\-vlan\-test\fR.
+.PP
 The \fBovs\-vlan\-test\fR program may be used to check for problems sending
 802.1Q traffic which may occur when running Open vSwitch. These problems can
 occur when Open vSwitch is used to send 802.1Q traffic through physical
@@ -82,5 +88,6 @@ Run an \fBovs\-vlan\-test\fR client with a control server located at
 .BR ovs\-vswitchd (8),
 .BR ovs\-ofctl (8),
 .BR ovs\-vsctl (8),
+.BR ovs\-test (8),
 .BR ethtool (8),
 .BR uname (1)