SDN Quick and Dirty First Steps

So I just attended Cisco Live in San Diego, and for the second year in a row, the buzz is all about SDN (Software Defined Networking).  This can mean a lot of things, kind of like “cloud computing”;  what is that really?   Well this year there was more practical knowledge and a few practical examples.  Through the week I managed to stitch together knowledge from several sessions to accomplish something near a “hello world” in the SDN realm.

What’s the point?  So what I am about to go through will demonstrate pulling a “show ver” off of a nexus (7k/5k/1k).  You say there are lots of ways to do that, and you would be right but this way is much easier than past methods and is very scalable to performing much larger jobs.

First the environment on the client machine.  You need Python,  I have a mac so here are the steps for installing Python on the Mac:

This article says it best:   https://vandadnp.wordpress.com/2014/10/20/building-and-running-python-scripts-with-xcode-6-1/

But here are the broad strokes:

sudo easy_install cmd2

sudo easy_install pip

sudo pip install lxml

sudo pip install pexpect 

sudo pip install requests

Next we need to setup the Nexus switch to accept commands via the NxAPI which is a REST/JSON interface to the CLI.  They promise a NETCONF/YANG model is in the works but not available yet.

On the Nexus

nx-osv-1# conf t

Enter configuration commands, one per line.  End with CNTL/Z.

nx-osv-1(config)# feature nxapi

That’s all there, hey that was easy.

Now the code:

 I got this from

https://github.com/datacenter/nxos

Python Script

#Print Chassis info, Hostname and software version of a given switch.

import json

import requests

print “enter ip address”

ip=raw_input()

my_headers = {‘content-type’: ‘application/json-rpc’}

url = “http://”+ip+”/ins”

username = “cisco”

password = “cisco”

payload=[{“jsonrpc”: “2.0”,

          “method”: “cli”,

          “params”: {“cmd”: “show version”,

                     “version”: 1},

          “id”: 1}

         ]

response = requests.post(url, data=json.dumps(payload), headers=my_headers, auth=(username, password)).json()

#Now Process the response

kick_start_image = response[‘result’][‘body’][‘kickstart_ver_str’]

chassis_id = response[‘result’][‘body’][‘chassis_id’]

hostname =  response[‘result’][‘body’][‘host_name’]

print “ip : {0} is a \”{1}\” with hostname: {2} running software version : {3}”.format(ip , chassis_id, hostname, kick_start_image)

OutPut

alexbaldwin@Alexs-MBP ~/Google Drive/gitrepository/nxos/nxapi/samples> python ./show_version.py

enter ip address

172.16.1.68

ip : 172.16.1.68 is a “NX-OSv Chassis” with hostname: nx-osv-1 running software version : 7.2(0)D1(1) [build 7.2(0)ZD(0.120)]

alexbaldwin@Alexs-MBP ~/Google Drive/gitrepository/nxos/nxapi/samples>

Posted in Uncategorized | Tagged , , , | Leave a comment

How to open / close your automatic gate opener with your smart phone

I dabble in all sorts of technical fields and in my capacity as an engineer for Gatekeeper Ltd, I developed the OSM-1 (think Open Says Me).   The OSM-1 device allows you to open or close your automatic gate opener, garage door opener, or really any device that is controlled by a normally open momentary switch.

The OSM-1 is built on a common single board computer the size of a credit card and provides wired and wireless Ethernet so you can connect it to your home network easily.  The OSM-1 builds an outbound TCP connection to the servers at Gatekeeper Ltd so it works behind virtually any firewall.  There is a simple setup process to complete and you must create a free account on gatekeeper’s website.

Once those steps are complete you attach the device to your automatic gate opener and download the smart phone app, available for iOS and Android.  After all of that setup is complete, you can easily open and close your gate from your smart phone  Enjoy.

To buy the device go here.    If you need help start here. support@gatekeeperltd.com  http://www.gatekeeperltd.com/support

P.S. Oh by the way if you don’t have wired Ethernet at your gate and your wireless network doesn’t reach, you can just install Ethernet over Power adapters like these.

Posted in Uncategorized | Tagged , , , , , , , | Leave a comment

the dark

My home network is not so afraid of the dark

20141026_6347acxyx500

nothing to be afraid of here . . . right ?

I have no guarantee of absolute protection from zero-day threats, but no one does.
We can only do our best to keep up with current anti-virus and firewall rules, but protection of critical infrastructure is more than managing software updates. Some very common problems are often self-inflicted growing pains. Maintaining sufficient bandwidth and growing room is one of those things that still benefits from human assessment and guidance. A distributed denial of service (DDoS) attack will certainly impact your users and customers, but we can hurt ourselves just as badly if we don’t recognize our own limitations and bottlenecks. If we are not actively polling and monitoring the data consumption of our various elements – we have no baseline of expected performance.
I like to know what nominal behavior looks like, even at home.

Most network elements and servers provide SNMP protocol support for monitoring statistics and performance detail, but establishing an agent to help monitor and track this information can be an expensive process if you want a turn-key solution. There are plenty of high-end programs and appliances to provide graphs, alarms, and notifications – but that expense often gets deferred. A medium to large operation may be able to devote a full time person to manage projects utilizing NetBrain, CA_Spectrum, or HP_OpenView, but many don’t. Even minimal monitoring is better than none.
It is not that hard to give yourself a few simple and inexpensive tools.

scratchpad-1x500

screen shots from BigBrother, Cactii, & Nagios

The open-source world offers several tools in this arena, but these usually require some local talent to integrate and provide a product that can satisfy your requirements. Again, many operations hesitate to fund such a monitoring environment, or don’t feel justified in retaining such talent in-house. There is not a huge return on that investment, until something breaks.
Hey, it is working fine now, right..?

SomeNet.NET was registered on the internet in 1996, establishing a base for my research and consulting activities. Mostly it has my ham radio activities and some photography, as I have not so had much time available for consulting until recently. This type of network management and monitoring, is what I do. I even understand it, and know the value. I intend to be contributing a lot more on this blog in the near future, specifically on open-source tools and solutions that you might find useful.
I use these tools, even at home – and I AM an expert…

Thanks  /;^)

Posted in consulting, internet, Linux, network | Tagged , , , , , , , , | Leave a comment

Even old servers deserve our help.

You too may have spent some time recently, responding to the Shellshock threat. Most (many ?) sites had a pretty good grip and were able to apply available patches easily and reduce their visibility quickly. I thought of one client that still has an old RHEL3 system with a public face. Not used for much, easily overlooked…

Shellshock_NSHx

.. manually update your old bash ..

Not finding ready to apply RPM packages for this RHEL3 bash – I turned to the search engines and found Steve Jenkins blog post on this very topic. With his excellent guidance I patched my exact version source and produced new binaries for this old system, much easier than I had expected. AND, there were no versionitus complications that often plague older system updates. Of course it helps that something as fundamental as GNUbash would have an extensive archive of old versions, and many, many patches.

This will not always be so easy to fix such a wide-spread problem, so be sure to check with your clients and look for old half-forgotten systems that may still be vulnerable. This is an easy way to become a hero.

Digging deeper into Steve’s blog, I learned some of his history and now more fully recognize his contributions over the years. Anyone else remember WinFiles.COM ?

I’ll probably subscribe to his blog, and learn some more along the way.
Looks like a lot of really good stuff in here.

Thanks Steve.. /;^)

Posted in consulting, internet, Linux | Tagged , , , , , | 1 Comment

I used to live here.

There was a time, when I lived in the hardware.

20131013_1183acx

Ever been to this City ?

Then it got to where they could train anybody to swap cards, so I had to move on.
But, there was a time I could troubleshoot the microcode that flowed and controlled the gates between the pads of these chips. Like a traffic cop, “So you want to add those two registers?; WELL Now, do they both have the same number of bits .?.” ; “And where did you think you would go after that ?” ; “Do you mind if I run a trace on your backplane connections ?” (We didn’t need no stinkin warrants back then either..)..
By Golly, you go droppin some bits around here, we gonna find you……

As much as you would like to hear the rest of that story..   Eh – maybe later…
I’ll be telling you some other stories soon enough.
As much as I have learned from the past, it has to relate to ‘right now’.
To quote Sir Paul McCartney “Do some good before you say GoodBye“.
Thanks  /;^)

Posted in anything, everything | Tagged , , , , | Leave a comment

How did this jumbo packet get here?

So recently I was asked to explain why we had jumbo packets on the network when we don’t have jumbo enabled on either our network devices or hosts.  Good question.

capture

 

It says clearly that it is 2974 bytes on wire.   Well long story short, it lies.

Turns out a feature that is on by default called “generic receive offload” glued the fragments back together before passing the frame up to pcap.  So bottom line, pcap only sees what your NIC gives it so a capture is not truly a raw capture in all cases.  Other links that explain it better.

http://blog.securityonion.net/2011/10/when-is-full-packet-capture-not-full.html

https://lwn.net/Articles/358910/

It can be disabled in Linux until the next reboot with

ethtool -K eth0 gro off
Posted in Uncategorized | Leave a comment

How to setup FCoE on Dell R710 Servers with X520 card using the Nexus 1000v

FCoE works well enough if you are deploying on a UCS or other blade server that is officially supported.  However it can get a bit interesting trying to get it to work on a general purpose server.  Here is my writeup on deploying FCoE using Dell R710 servers and the included X520-DA2 card.  Oh and the requirement to use the Nexus 1000V.

Dell Server

First on the Dell server you really don’t have to do anything in the BIOS, there is an FCoE configuration in the BIOS but that is just for boot from FCoE you don’t have to touch it at all.

VmWare

However you do have to build VmWare using the Dell specific build disc.

VMvisor-Installer-5.5.0-1331820.x86_64-Dell_Customized_A02.iso

http://www.dell.com/support/drivers/us/en/19/driverdetails?driverid=1P13P

Once VmWare is installed you will need to add the software FCoE adapters, here is the procedure (You only have to do the part on pages 14-17):  http://www.intel.com/content/www/us/en/network-adapters/10-gigabit-network-adapters/ethernet-x520-configuring-fcoe-vmware-esxi-5-guide.html

Nexus 5548

Getting the configuration on the 5k right is really the trick on this build, all sorts of weird problems will show up if you are missing anything and they are quite difficult to troubleshoot.  BTW I used version 7.0(2)N1(1) on the Nexus 5k.

1st the QoS configuration is essential, configure the 5k like this:

http://keepingitclassless.net/2012/11/qos-part-2-qos-and-jumbo-frames-on-nexus-ucs-and-vmware/

I used the above configuration with the only exception that I allowed jumbo frames in all traffic classes.

Next,

You need these features installed:

feature fcoe
feature npiv
feature lldp

The next thing to know is that if you are using a port channel to the 1000v the port channel cannot be the interface that binds to the vfc.  FCoE cannot ride on a VPC.  Instead you would want to create two VFC interfaces one on each 5k and have two fabrics A&B.

So the physical interface on one 5k would look like this:

interface Ethernet101/1/11
description esxi1-eth0
switchport mode trunk
spanning-tree port type edge trunk
channel-group 1011 mode active

And the virtual interface would look like this:

interface vfc1011
bind interface Ethernet101/1/11
no shutdown

The vsan must be created and the vfc must be in the vsan:

vsan database
vsan 100
vsan 100 interface vfc1011

The VSAN must be bound to a VLAN

vlan 600
fcoe vsan 100

Of course your server must be zoned in with your storage, I won’t go into that in this blog.

Before the 1000v is installed, you should be able to see your vfc interface (1011 in my case) when you perform a “show flogi database” command

Here are some useful troubleshooting commands

Debug lldp errors

Debug lldp warnings

Debug lldp dcbx_feat

Show sys internal dcbx info interface e101/1/11

Also login to the Esxi host and tail the /var/log/vmkernel.log file while flapping the interface is useful.

Nexus 1000v

Finally on the 1000v, currently the 1000v latest version 4.2.1.SV2.2a does not support these software FCoE adapters, however I can confirm that the latest beta of the dao release (5.2.1.SV3.1.0.276) does support them.  Follow the normal process for adding the host to Vcenter and then to the 1000v.

 

Posted in network, Uncategorized | Tagged , , , , , | Leave a comment

Nexus VRF route confusion

Here is something that confuses more than a couple people on the Nexus platform.

You might think that because the IOS/XE command to create a default route in your management VRF is:

ip route  vrf management 0.0.0.0 0.0.0.0 192.168.1.1

That the command:

ip route 0.0.0.0/0 mgmt0 192.168.1.1 vrf management

would do the same on the Nexus platform, but you would be wrong.
It does not create a default route in VRF management  it creates a route which has it’s next hop in VRF management but the route itself would go into the default VRF (or whichever VRF you are in when you enter the command).  This is useful for MPLS L3-VPN route leaking.

The correct command syntax on the Nexus is:

n7k-i2(config-if)# vrf context management
n7k-i2(config-vrf)# ip route 0.0.0.0/0 192.168.1.1
n7k-i2(config-vrf)# end

 

Posted in network, Uncategorized | Tagged , , | Leave a comment

FCoE and FP ?

I had a little trouble tracking down the answer to what I thought was a simple question so I thought I would post it for anyone “googling”

The question was “Does FCoE work with fabric path?”  Specifically my question was on the Nexus 5548UP in a single hop mode not FCoE multi-hop.

Some very smart people at Cisco dug up the answer for me and here it is as of 8/14/2014.

A device attached to a N5K leaf switch would be supported today with FP enabled, however the FCoE VLAN/VSAN traffic would NOT traverse the FP fabric.

An FCoE device attached to a N7K spine and/or next hop switch (MDS/Nexus) would be supported today, however only with dedicated FCoE uplinks to a Storage VDC and/or downlinks to an MDS/Nexus switch…essentially a parallel VLAN/VSAN network. This once again would NOT traverse the FP fabric.
An FCoE device in the future, with Dynamic FCoE, would allow FCoE VLAN/VSAN traffic

So translating slightly you can enable fabric path and run it on the Nexus, but your FCoE VLAN must remain classical Ethernet not fabric path.  Apparently forwarding of FCoE frames over FP links is disallowed in the code.

Posted in network, Uncategorized | Leave a comment

Multicast Traffic over an Encrypted OTV Unicast Transport on the CSR

I set out trying to create a way to funnel multicast traffic from various back end networks to a remote facility without creating a bunch of very special new interfaces at the remote facility.  OTV seems like the perfect technology for this because of they have addressed alot of the problems that normally come with layer 2 overlays.  My remote facility might not have multicast available so that meant I needed the OTV unicast transport option.  This is also sensitive data so I needed to encrypt the traffic and I might not have jumbo frames available so that meant the only platform that could encrypt and fragment OTV left to me was the ASR or CSR routers.  I started with the CSR(virtual)  since all of my ASR (physical) were busy.

There is a lot of documentation about how to perform OTV on the Nexus 7000, but much less on the ASR/CSR line.

It was fairly easy to get unicast up and running, I started with this blog which is very well written but contains one mistake from what I can tell I’ll get to that later.

http://www.layerzero.nl/blog/2013/05/otv-and-lisp-on-the-csr-1000v/

otv_blog_1

 

So my test started with the above configuration which consisted of two eSXi 5.5 hosts connected by an Ethernet link crudely simulating the WAN.

 

HOME Router:

otv site bridge-domain 1

!

otv fragmentation join-interface GigabitEthernet1

otv site-identifier 0000.0000.0001

license boot level premium

interface Overlay1

no ip address

otv join-interface GigabitEthernet1

otv use-adjacency-server 10.10.10.78  unicast-only

service instance 1 ethernet

encapsulation dot1q 633

bridge-domain 633

interface GigabitEthernet1

ip address 10.10.10.79 255.255.255.128

negotiation auto

!

interface GigabitEthernet2

description inside

no ip address

service instance 1 ethernet

encapsulation untagged

rewrite ingress tag push dot1q 633 symmetric

bridge-domain 633

REMOTE Router:

otv site bridge-domain 1

!

otv fragmentation join-interface GigabitEthernet1

otv site-identifier 0000.0000.0002

license boot level premium

interface Overlay 1

 no ip address

otv join-interface GigabitEthernet1

 otv use-adjacency-server 10.10.10.78 unicast-only

 otv adjacency-server unicast-only

 service instance 1 ethernet

  encapsulation dot1q 633

  bridge-domain 633

interface GigabitEthernet1

ip address 10.10.10.78 255.255.255.128

negotiation auto

interface GigabitEthernet2

 description inside

 no ip address

 negotiation auto

 service instance 1 ethernet

  encapsulation untagged

  rewrite ingress tag push dot1q 633 symmetric

  bridge-domain 633

 service instance 2 ethernet

  encapsulation dot1q 2

  bridge-domain 1

 
 
REMOTE-1#  sh otv ro
Codes: BD – Bridge-Domain, AD – Admin-Distance,
       SI – Service Instance, * – Backup Route
OTV Unicast MAC Routing Table for Overlay1
 Inst VLAN BD     MAC Address    AD    Owner  Next Hops(s)
———————————————————-
 0    633  633    000c.293b.9dd6 40    BD Eng Gi2:SI1
 0    633  633    0050.56b1.cf06 50    ISIS   HOME-1
2 unicast routes displayed in Overlay1
 ———————————————————-
2 Total Unicast Routes Displayed
 

 

Once unicast worked, I proceeded to multicast but could not get multicast traffic to work.

I could see the multicast traffic locally at the sites, but the *,G from the remote site was never showing up as you can see below.

HOME#sh otv mro
 OTV Multicast Routing Table for Overlay1
 Bridge-Domain = 633, s = *, g = *
 Outgoing interface list:
 Default, NoRedist
 Incoming interface count = 0, Outgoing interface count = 1
 Bridge-Domain = 633, s = 10.10.100.2, g = 225.1.1.3
 Incoming interface list:
 Service Instance 1, GigabitEthernet2, 0050.56b1.cf06
 Incoming interface count = 1, Outgoing interface count = 0
 2 multicast routes displayed in Overlay1
 ---------------------------------------------------------
 2 Total Multicast Routes Displayed

 

Well I’ll skip to the good part, here is the magic command

ip igmp snooping querier

It must be on both routers.

I cannot understand why this command is so well hidden, in all of the documentation I could find it is nowhere to be found.  I opened a TAC case and the Cisco engineer could not find the answer even with his internal access.

I dont see it here?

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/wan_otv/configuration/xe-3s/wan-otv-xe-3s-book/wan-otv-adj-server.html#reference_EDD49D9567A440F5B191ACE3815CC8CB

I did find it sort of mentioned here, but not in context to the ASR/CSR and not exactly the same command so it wouldn’t help you if you didn’t already know the answer.

http://www.cisco.com/c/dam/en/us/td/docs/solutions/Enterprise/Data_Center/DCI/5-0/OTVmulticast.pdf

It probably should have been listed here:

http://www.cisco.com/c/en/us/support/docs/routers/asr-1000-series-aggregation-services-routers/117158-configure-otv-00.html

It definitely should have been included here:

http://www.cisco.com/c/en/us/td/docs/solutions/Hybrid_Cloud/DRaaS/CSR/CSR.pdf

Finally I found it here, http://d2zmdbbm9feqrf.cloudfront.net/2014/usa/pdf/BRKDCT-3103.pdf

Page 136 in the appendix !

Below you can see the output of the wireshark capture of the transferred packet prior to adding encryption:  IP in Eth in MPLS in GRE in IP in Eth

otv_blog_2

Well once that was working, getting it encrypted was a breezehttp://stayinginit.blogspot.in/2013/12/encrypting-overlay-transport.html

Adding IP Sec

On .78
 crypto isakmp policy 1
hash md5
authentication pre-share
group 5
 crypto isakmp key cisco123 address 10.10.10.79
 crypto ipsec transform-set ed1_ts esp-aes esp-sha-hmac
 ip access-list extended ed1_acl
permit gre host 10.10.10.78 host 10.10.10.79
 
crypto map ed1_map 1 ipsec-isakmp
set peer 10.10.10.79
set transform-set ed1_ts
match address ed1_acl
 interface GigabitEthernet1
crypto map ed1_map
 
 
On .79
 
crypto isakmp policy 1
 hash md5
authentication pre-share
group 5
 crypto isakmp key cisco123 address 10.10.10.79
 crypto ipsec transform-set ed1_ts esp-aes esp-sha-hmac
 ip access-list extended ed1_acl
permit gre host 10.10.10.79 host 10.10.10.78
 crypto map ed1_map 1 ipsec-isakmp
set peer 10.10.10.78
set transform-set ed1_ts
match address ed1_acl
 
interface GigabitEthernet1
crypto map ed1_map
 
 
Finally getting back to Tom Lijnse’ blog, he says:
 
 
Configuring OTV
The next thing I do is preparing the OTV join interface for multicast operation. I enable multicast routing, set the IGMP version to 3 and enable PIM in passive mode on the OTV join interface:
! ip multicast-routing distributed
!
interface GigabitEthernet1
ip pim passive
ip igmp version 3
!
Note: Unlike the Nexus 7000, the CSR requires multicast routing to be enabled in order to enable the IGMP functionality that is required for OTV. On the Nexus 7000 it is not necessary to enable multicast routing and PIM. Simply setting the IGMP version to 3 is sufficient on that platform

What I found was that none of that mattered, you could have left off all the statements listed there as they don’t appear to do anything in my lab, but what you must have is the

ip igmp snooping querier
 
 

Maybe some conditions of the test are at play here, or perhaps changes are being made in teh code from revision to revision I’m using 3.12   I’ll update if I find out more as this project progresses.

 

Posted in Uncategorized | Leave a comment