Multicast Traffic over an Encrypted OTV Unicast Transport on the CSR

I set out trying to create a way to funnel multicast traffic from various back end networks to a remote facility without creating a bunch of very special new interfaces at the remote facility.  OTV seems like the perfect technology for this because of they have addressed alot of the problems that normally come with layer 2 overlays.  My remote facility might not have multicast available so that meant I needed the OTV unicast transport option.  This is also sensitive data so I needed to encrypt the traffic and I might not have jumbo frames available so that meant the only platform that could encrypt and fragment OTV left to me was the ASR or CSR routers.  I started with the CSR(virtual)  since all of my ASR (physical) were busy.

There is a lot of documentation about how to perform OTV on the Nexus 7000, but much less on the ASR/CSR line.

It was fairly easy to get unicast up and running, I started with this blog which is very well written but contains one mistake from what I can tell I’ll get to that later.

http://www.layerzero.nl/blog/2013/05/otv-and-lisp-on-the-csr-1000v/

otv_blog_1

 

So my test started with the above configuration which consisted of two eSXi 5.5 hosts connected by an Ethernet link crudely simulating the WAN.

 

HOME Router:

otv site bridge-domain 1

!

otv fragmentation join-interface GigabitEthernet1

otv site-identifier 0000.0000.0001

license boot level premium

interface Overlay1

no ip address

otv join-interface GigabitEthernet1

otv use-adjacency-server 10.10.10.78  unicast-only

service instance 1 ethernet

encapsulation dot1q 633

bridge-domain 633

interface GigabitEthernet1

ip address 10.10.10.79 255.255.255.128

negotiation auto

!

interface GigabitEthernet2

description inside

no ip address

service instance 1 ethernet

encapsulation untagged

rewrite ingress tag push dot1q 633 symmetric

bridge-domain 633

REMOTE Router:

otv site bridge-domain 1

!

otv fragmentation join-interface GigabitEthernet1

otv site-identifier 0000.0000.0002

license boot level premium

interface Overlay 1

 no ip address

otv join-interface GigabitEthernet1

 otv use-adjacency-server 10.10.10.78 unicast-only

 otv adjacency-server unicast-only

 service instance 1 ethernet

  encapsulation dot1q 633

  bridge-domain 633

interface GigabitEthernet1

ip address 10.10.10.78 255.255.255.128

negotiation auto

interface GigabitEthernet2

 description inside

 no ip address

 negotiation auto

 service instance 1 ethernet

  encapsulation untagged

  rewrite ingress tag push dot1q 633 symmetric

  bridge-domain 633

 service instance 2 ethernet

  encapsulation dot1q 2

  bridge-domain 1

 
 
REMOTE-1#  sh otv ro
Codes: BD – Bridge-Domain, AD – Admin-Distance,
       SI – Service Instance, * – Backup Route
OTV Unicast MAC Routing Table for Overlay1
 Inst VLAN BD     MAC Address    AD    Owner  Next Hops(s)
———————————————————-
 0    633  633    000c.293b.9dd6 40    BD Eng Gi2:SI1
 0    633  633    0050.56b1.cf06 50    ISIS   HOME-1
2 unicast routes displayed in Overlay1
 ———————————————————-
2 Total Unicast Routes Displayed
 

 

Once unicast worked, I proceeded to multicast but could not get multicast traffic to work.

I could see the multicast traffic locally at the sites, but the *,G from the remote site was never showing up as you can see below.

HOME#sh otv mro
 OTV Multicast Routing Table for Overlay1
 Bridge-Domain = 633, s = *, g = *
 Outgoing interface list:
 Default, NoRedist
 Incoming interface count = 0, Outgoing interface count = 1
 Bridge-Domain = 633, s = 10.10.100.2, g = 225.1.1.3
 Incoming interface list:
 Service Instance 1, GigabitEthernet2, 0050.56b1.cf06
 Incoming interface count = 1, Outgoing interface count = 0
 2 multicast routes displayed in Overlay1
 ---------------------------------------------------------
 2 Total Multicast Routes Displayed

 

Well I’ll skip to the good part, here is the magic command

ip igmp snooping querier

It must be on both routers.

I cannot understand why this command is so well hidden, in all of the documentation I could find it is nowhere to be found.  I opened a TAC case and the Cisco engineer could not find the answer even with his internal access.

I dont see it here?

http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/wan_otv/configuration/xe-3s/wan-otv-xe-3s-book/wan-otv-adj-server.html#reference_EDD49D9567A440F5B191ACE3815CC8CB

I did find it sort of mentioned here, but not in context to the ASR/CSR and not exactly the same command so it wouldn’t help you if you didn’t already know the answer.

http://www.cisco.com/c/dam/en/us/td/docs/solutions/Enterprise/Data_Center/DCI/5-0/OTVmulticast.pdf

It probably should have been listed here:

http://www.cisco.com/c/en/us/support/docs/routers/asr-1000-series-aggregation-services-routers/117158-configure-otv-00.html

It definitely should have been included here:

http://www.cisco.com/c/en/us/td/docs/solutions/Hybrid_Cloud/DRaaS/CSR/CSR.pdf

Finally I found it here, http://d2zmdbbm9feqrf.cloudfront.net/2014/usa/pdf/BRKDCT-3103.pdf

Page 136 in the appendix !

Below you can see the output of the wireshark capture of the transferred packet prior to adding encryption:  IP in Eth in MPLS in GRE in IP in Eth

otv_blog_2

Well once that was working, getting it encrypted was a breezehttp://stayinginit.blogspot.in/2013/12/encrypting-overlay-transport.html

Adding IP Sec

On .78
 crypto isakmp policy 1
hash md5
authentication pre-share
group 5
 crypto isakmp key cisco123 address 10.10.10.79
 crypto ipsec transform-set ed1_ts esp-aes esp-sha-hmac
 ip access-list extended ed1_acl
permit gre host 10.10.10.78 host 10.10.10.79
 
crypto map ed1_map 1 ipsec-isakmp
set peer 10.10.10.79
set transform-set ed1_ts
match address ed1_acl
 interface GigabitEthernet1
crypto map ed1_map
 
 
On .79
 
crypto isakmp policy 1
 hash md5
authentication pre-share
group 5
 crypto isakmp key cisco123 address 10.10.10.79
 crypto ipsec transform-set ed1_ts esp-aes esp-sha-hmac
 ip access-list extended ed1_acl
permit gre host 10.10.10.79 host 10.10.10.78
 crypto map ed1_map 1 ipsec-isakmp
set peer 10.10.10.78
set transform-set ed1_ts
match address ed1_acl
 
interface GigabitEthernet1
crypto map ed1_map
 
 
Finally getting back to Tom Lijnse’ blog, he says:
 
 
Configuring OTV
The next thing I do is preparing the OTV join interface for multicast operation. I enable multicast routing, set the IGMP version to 3 and enable PIM in passive mode on the OTV join interface:
! ip multicast-routing distributed
!
interface GigabitEthernet1
ip pim passive
ip igmp version 3
!
Note: Unlike the Nexus 7000, the CSR requires multicast routing to be enabled in order to enable the IGMP functionality that is required for OTV. On the Nexus 7000 it is not necessary to enable multicast routing and PIM. Simply setting the IGMP version to 3 is sufficient on that platform

What I found was that none of that mattered, you could have left off all the statements listed there as they don’t appear to do anything in my lab, but what you must have is the

ip igmp snooping querier
 
 

Maybe some conditions of the test are at play here, or perhaps changes are being made in teh code from revision to revision I’m using 3.12   I’ll update if I find out more as this project progresses.

 

Advertisements
Posted in Uncategorized | Leave a comment

The joys of 3D printing

Normally I only post IT related stuff to this blog, but recently I was able to use my 3D printer to fix something that was very difficult to fix just a few years ago.  My Dodge Ram 1500 cup holder is made too weak in my opinion and was easily broken. From what I can tell you can’t buy a replacement anywhere so I  modeled it and printed a replacement. The part was a bit complex and has lots of curves, but this version is holding together so far in my truck.  It was great satisfaction to fix something in this way and to be able to share the part to help others.

http://www.thingiverse.com/thing:416627

http://www.ramforumz.com/showthread.php?t=52762&page=2

 

 

Posted in Uncategorized | Tagged , , , | Leave a comment

How to configure NAT for Xen virtual machines on OpenSuse 13

I was recently asked to help someone configure NAT for VMs running on Xen with OpenSuse 13.  I don’t have a ton of experience with Xen, but I must say this was much more difficult than I anticipated.  I believe the trouble stems from the recent change to the xl toolstack for Xen and the lack of support within OpenSuse for it although I’d be open to anyone with more experience correcting me on the subject.  I did eventually get it working and I’ve detailed the solution below.

First, verify that virtual bridge 0 or a similair bridge exists,  if not create it because we will need this bridge to perform the NAT.

brctl addbr virbr0
1

Don’t expect to see any interfaces under virbr0, just make sure it exists.

Give it an IP address on the private network, this address is being assigned to the host and will be used as the default gateway on the VMs

ifconfig virbr0 192.168.10.1

Determine if ip forwarding is enabled because it is required for NAT

cat /proc/sys/net/ipv4/ip_forward

If the command responds with a zero ( 0 ) then it is disabled, a 1 for enabled.

If not enable it:

echo 1 > /proc/sys/net/ipv4/ip_forward 

The above command only enables it until a reboot, to enable it permanently then edit the file /etc/sysctl.conf:

/etc/sysctl.conf:
net.ipv4.ip_forward = 1

 

NAT is accomplished by altering rules within the iptables Linux firewall.

This command adds a rule into the INPUT chain of the filter table that allows packets sourced by the network 192.168.10.0/24 (Our private network for virtual machines) to be accepted.  For more information on the iptables tables and chains see this write up: http://www.thegeekstuff.com/2011/01/iptables-fundamentals/

iptables --table filter --insert INPUT --source 192.168.10.0/255.255.255.0 --jump ACCEPT

This command adds a rule to the FORWARD chain of the filter table that allows our private network, so we accept the packet, then we forward the packet.

 iptables --table filter --insert FORWARD --source 192.168.10.0/255.255.255.0 --jump ACCEPT

This command adds a rule in the FORWARD chain of the filter table to allow return packets back to the private network if the TCP state is established, so this is basically stateful firewalling to our private network.

 iptables --table filter --insert FORWARD --destination 192.168.10.0/255.255.255.0 --match state --state ESTABLISHED,RELATED --jump  ACCEPT

Finally this command inserts a rule into the POSTROUTING chain of the nat table to actually translate the outgoing (and incoming packets) to our private network.

 iptables --table nat --insert POSTROUTING --source 192.168.10.0/255.255.255.0 --destination 192.168.10.0/255.255.255.0--jump MASQUERADE

You can see all of the iptables rules including the ones you just added using this command:

iptables --list

All of these commands can be added to a script, or the  iptables rules can be added to a file named /etc/sysconfig/scripts/SuSEfirewall2-custom then edit /etc/sysconfig/SuSEfirewall2

change

Code:
FW_CUSTOMRULES=""

to

Code:
FW_CUSTOMRULES="/etc/sysconfig/scripts/SuSEfirewall2-custom"

Now that networking is setup on the host, we need to define the VM under the xl toolset.  I have not been able to get this to work under virt-manager yet but if someone has, please comment to this post.

Edit a file we will call it vm-2.xl and add the following, I’m using inx 1.1 for this demo available here: http://inx.maincontent.net/

 name = “vm-2”
uuid = “db6f4bac-c17f-8856-3b1e-2b249206e28f”
maxmem = 1024
memory = 1024
vcpus = 1
builder = “hvm”
kernel = “/usr/lib/xen/boot/hvmloader”
boot = “d”
pae = 1
acpi = 1
apic = 1
hap = 0
viridian = 0
rtc_timeoffset = 0
localtime = 0
on_poweroff = “destroy”
on_reboot = “restart”
on_crash = “restart”
device_model = “/usr/lib64/xen/bin/qemu-dm”
sdl = 0
vnc = 1
vncunused = 1
keymap = “en-us”
disk = [ “file:/inx-1.1.iso,hdc:cdrom,r”]
vif = [ ‘bridge=virbr0,ip=10.0.0.2,”mac=01:0c:29:3f:00:d8’]
parallel = “none”
serial = “pty”
soundhw = “es1370”

 

You can export your virt-manager, virsh created VMs to xl using this command:

virsh -c xen:/// domxml-to-native xen-xm /etc/libvirt/libxl/vm-2.xml > vm-2.xl

Of course you will want to edit the resulting file so as to be like the listing above for my working VM.

This command instantiates the VM:

xl create vm-2.xl

You may get a couple of errors relating to the choices I made in the VM configuration file, but if it does not error out you should be able to check which interfaces are created with ifconfig.

ifconfig

3

You should have two interfaces vif2.0 and vif2.0-emu that are part of a bridge virbr0 or something similiar

brctl show

1

If you sniff on vif2.0-emu you should see traffic from the VM

tcpdump -i vif2.0-emu

Of course the VM needs to be configured with an IP address like 192.168.10.x/24 and a gateway of 192.168.10.1 and set the DNS in /etc/resolv.conf

Run the following command to get a console on your VM

xl vncviewer vm-2
sudo ifconfig eth0 192.168.10.10 netmask 255.255.255.0 
sudo route add -net default gw 192.168.10.1 

If all goes well, you should be able to ping google from the VM and get a response:

 

ping 8.8.8.8
64 bytes from 8.8.8.8: icmp_seq=1 ttl=47 time=46.063 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=47 time=46.178 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=47 time=49.135 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=47 time=46.251 ms
Posted in Linux, Uncategorized | Tagged , , , , , , , | Leave a comment

PIM BiDir Bidirectional Wireshark dissectors

Bidirectional Protocol Independent Multicast RFC 5015 (  http://www.ietf.org/rfc/rfc5015.txt ) is in my opinion the most bulletproof multicast routing protocol around, but it doesn’t have a huge install base.  Consequently, the PIM dissector in even the latest wireshark as of this writing 1.11 doesn’t support decode of the BiDir specific messages like Offer/Winner/Backoff/Pass which makes troubleshooting alot more difficult.  One of our developers Dave Zoller generously coded me up a dissector and is going to distribute it back to the wireshark project.  Until then the binary for windows is available here:
http://www.baldwinpines.com/Wireshark-win32-1.11.3-MSFC-PIM2.10.exe

Source code can be found here: http://www.baldwinpines.com/pim-dissector.zip

 

Posted in internet, network | 4 Comments

File Synchonization that works 2/2

What I’m going to show you how to do today is fix a couple of business problems with CloudStation.   We’ll be using two applications, Quickbooks and NeatWorks.

Businesses that use QuickBoooks have started using hosted QuickBooks because it allows them to outsource their accounting to anywhere on the globe, eliminates PC problems from taking down their business and provides better redundancy for the QB data file.  But the downside to that is that they no longer have their QB file on their local hard drive and so can’t load it locally if they needed to etc.  This problem can be solved by saving a backup each day of your QB data file on the cloudstation sync folder in your data center from your hosted QB account (www.cbonetworks.com suppports this option).  The file will automatically sync back to your local hard drive instantly.

The second problem Cloudstation can fix invovles NeatWorks.  Small businesses love Neatworks for cleaning up their desks and saving all of the documents in searchable format in a database.  Neatworks now offers a cloud service for this, but it costs money.  An alternative is to move the Neatworks database folder to a cloudstation folder which will sync up to the cloud and your other machines.  There is one trick to moving the Neatworks folder though.  First grab the folder in explorer and copy it to somewhere safe temporarily.  Then move it to the cloudstation folder.  Finally setup a symbolic link to point to cloudstation so that NeatWorks can find it.    To do that go to the command line in Windows and type this:

C:\Users\customer service\Documents>mklink /J “Neat Data” “c:\Users\Customer
Service\Documents\baldwinpines\Neat Data”

Where “customer service” is my user id in windows

baldwinpines is the name of the cloudstation folder

“Neat Data” is the neatworks directory

Now when you update neatworks on your PC it will sync the files up to the cloud and if your PC dies you can pull them back.  Maybe someday they will get Neatworks to work on a terminal server so you can view the documents in the cloud.

Good Luck

Posted in Uncategorized | Leave a comment

File Synchonization that works 1/2

If you haven’t heard about it Dropbox (www.dropbox.com) offers really awesome file synchronization services from your computer to your other computers.  Google was next to come along with their Google Drive and provide pretty much the same thing.  There are only two things wrong with these services, you lose control of your data, and they limit you to 2-5GB.  Well now we have CloudStation from Synology http://www.synology.com .  If you have a Synology NAS, which you should they are fantastic, you can use the cloudstation add on to sync your PCs and MACs with your NAS over the Internet.  You can even sync multiple folders like a shared folder and a home folder.  If you don’t own a Synology NAS or your Internet is not robust enough for this you could consider checking into this company www.cbonetworks.com which full disclosure myself and Bryan McJunkin founded in 1996 though I’ve moved on since then.  They offer hosted Cloudstation services among other cloud services.  Part 2 of this blog will show you some case studies in how to use cloudstation along with other cloud services to improve your business resiliancy.

Posted in Uncategorized | Leave a comment

Finally passed my IE

I’m CCIE # 39221 thanks everyone for the support!

Posted in Uncategorized | Leave a comment

I’ve been workin . . .

I scribbled this out some time back, in the mid-90s as I was pulling arrows out of my backside – learning how to manage an internet service. I only recently found where I had later saved it in a text file, dated 1999. Guess that means it is Copyrighted now.
Worth posting here I reckon.
(sung like ‘I’ve been workin on the railroad’)

I’ve been workin on the backbone,
All the live long day.
I’ve been workin on the backbone,
Just to clear this one alarm.
Don’t you feel the routes a’flappin,
aggregation filter’s bout to bust.
Don’t it make you want to reboot,
“Cisco, flush your cache.”

Cisco, won’t you flush,
Cisco, won’t you flush,
Cisco, won’t you flush your cache for me?
Cisco, won’t you flush,
Cisco, won’t you flush,
Cisco, won’t you flush your cache?

Someone dropped a packet in there sideways.
Someone dropped a packet, I know.
Someone dropped a packet in there sideways,
Pluggin up the ol backbone.

Fee, fie, fiddle-e-i-o.
Fee, fie, fiddle-e-i-o-o-o-o.
Fee, fie, fiddle-e-i-o.
Pluggin up the ol backbone.

/;^)

Posted in control, internet, network | Tagged , , , , , | Leave a comment

Not much will be going on here until after I pass the CCIE

I’m studying like crazy for the CCIE R&S so I wont post until thats over with.

Posted in Uncategorized | Leave a comment

Get rid of Clear line for good

It seems I have finally found a solution to the pesky problem of corrupt serial lines on Cisco 2500 routers.  I have read of quite a few people complaining of the same issue.  Tyson Scott of IPExpert gave me the following config:

line 1 16
exec-timeout 0 0
logout-warning 240
no exec
no history
transport input telnet
telnet speed 9600 9600
autohangup
stopbits 1

If you add “aaa new-model” to this, you get rid of having to type clear line all the time.  I hope this helps a few folks out there suffering with old 2500 routers.

Posted in Uncategorized | Leave a comment