Big changes

So, you wanna begin in automation? Begin with your changes and how you develop and test.

The biggest mistake network admins regularly make is to merely "think" about problems and then develop solutions by "thinking" and committee. The figuring is, it will all work if we just think enough about it. This is nonsense. You will make mistakes relying on eyes and perfection. Let machines do their thing.

Change analysis

I have a need to sling some routes to another part of the network. Will it break stuff?

By some work in determining the scope of the proposed changes, one can note the assumptions that take place in the change. Example, I will be doing redistribution from BGP into OSPF and I assume that all prefixes from BGP from BGP today originate on two routers. How can I verify?

Another (crappy) example, I believe RIP route distribution contains no filtering anywhere. How can this be checked?

Decomposing the change and verifying state

Consider a situation where we have an access switch hanging downstream from distribution and core switches and we want to assign a VLAN to a port. Fairly easy scenario!

1) We need to determine whether the VLAN exists
2) We need to determine where the VLAN exists
3) We need to interconnect the devices on the VLAN
4) Do not permit downstream devices to be on the VLAN prior to the upstream having been configured.
5) Apply all relevant policy for the spanning-tree instance (think like in PVST-ish setup)

In our situation, we must see two core switches, the ports to the distro switches, perhaps the VPC interconnects, and then the access switch.

How many different things do we need to check at each?

  • VLAN existence
  • Trunking state
  • Spanning-tree state
Why each one?
  • If the VLAN doesn't exist, we need to create it
  • If the trunks are not setup, we need to set these up
  • If spanning-tree does not show the correct root, this is a problem.
Similarly, to confirm that the work is done, we need to check all of the same parameters again to ensure that the configuration has been effective.

State to in-situ

You've designed some kind of redundancy for this VLAN, right? Confirming spanning-tree is a good step to ensure that redundancy is in place. Now, show me your redundancy. Right now. Can you do a basic spanning-tree test and show me that it will reconverge properly?

Maybe BGP is a bit easier to envison.. we can't simply "turn down" spanning-tree. We can, however, turn down the port upstream from our access switch. This would demonstrate the redundancy well.

These test cases should include all of the "as designed" behaviors known. This will also help you in unfucking your design. If the test case looks stupid, your design is likely stupid.


Comments

Post a Comment

Popular posts from this blog

Linux unnumbered interface source IP behavior

In certain cases, (looking at you, Fortinet), interfaces make more sense without IP addresses. But the kernel must determine which source IP to use for sockets that don't specify one. Linux maintains a scoping of IP addresses. Those exposed directly with iproute2 are host, link, and global. When you intend on taking an unnumbered path, one of two things may happen: 1) A route exists for the unnumbered path. This route is populated with a src address when fib_create_info() is called  (such as in routing table update) (occurs via  fib_info_update_nh_saddr) or; 2) No route exists, but an interface is specified for the connection and the source is derived in ip_route_output_key_hash_rcu(). In both cases, inet_select_addr() is called to determine the source IP, but with slightly different parameters. In the first case, the scope of the FIB table is taken into account. As an example, sources used for link-local routes will permit link-local or global scope. In the seco...
Read more

FortiManager TCL scripts

I had a request sent to me the other day to make a script push some config with the hostname of the node included. Pretty basic thing to do. I was further constrained to use the FortiManager script runner to make this work. As usual, CLI-like scripting sucks and the magic behind the scenes made things more difficult. FortiManager's Role Big HA pile of shit to sync configs with templates. Also has a script runner (used by the config pushes). The scripts are handled through a component called "deployment manager". Relevant debugs: diagnose debug application depmanager -1 -> Not so useful for TCL, good under-the-hood look diagnose debug dpm conf-trace enable -> Will show you SSH connection stuff. More useful IMO. FortiManager holds SSH credentials for the device. It isn't like, even though FortiManager is trusted to play with config, FortiManager can magically login via SSH. This seems like quite a stupid design, but that is the situation we'...
Read more

But Ansible does it all!!

Most people running networks have a decent fleet of Cisco IOS devices (or devices that feature "industry-standard" *cough* ripoff *cough* CLI). Often times, the discussion of Ansible as a framework comes up as the solution. What does Ansible do? Ansible is designed to express desired state, compare against existing state and then calculate whether a change is needed. A great example is that we'll generate a config file with a template, push this to a server, and then signal the relevant service (perhaps with a restart). Cool! Easy! But what if the config file is already the same as what we generated from template? No problem, we'll calculate that the files are the same and do nothing. No restart, no problem.. its already good. This technique is highly effective for things like webservers where the service supports a consistent and suitable approach (in context, of course) for loading the new config. kk.. cool but what is wrong with IOS? IOS-like CLIs ar...
Read more