Learning ACI - Part 6: Access Policies

So far in this series, we’ve covered some basic concepts in ACI, including fabric bringup, APIC familiarisation, application profiles / EPGs / contracts as well as some of the networking concepts in ACI. At some point though, you’ll want to actually start attaching hosts and other devices to the fabric - in order to do this, you’ll need to get familiar with the concept of access policies.

Access policies are used to control parameters relating to access into the fabric, such as which VLANs to use on a port, which leaf nodes and ports should be used for specific hosts or devices, as well as configuration of other access parameters such as LLDP, CDP or storm control.

In this post, I’m going to walk through the creation of an access policy specifically for the connection of a bare metal host to the fabric; I’ll cover VM connectivity in a future post. By the end of this exercise, I will have provisioned a VLAN pool on a particular port on a particular leaf node. One important point is that I am not yet enabling any VLAN on a port - that happens only when I create an EPG and link it to a port. I’ll also say here that there are easier ways of achieving the creation of access policies (using wizards, API, etc). The method I’m using here is somewhat long-winded, but is useful to show the relationship between policies.

So let’s start at the beginning: defining a VLAN pool.

1. Define a VLAN Pool

Although we talk about bridge domains inside the ACI fabric, we still need to use regular VLANs on the link between a leaf node port and the host or device that connects to it. The first thing we need to do as part of our access policy configuration is to define a range of VLANs that are available for this purpose on any given leaf node. From the access policies tab, we’ll expand pools and then VLANs and select Create VLAN Pool. In the example below, I’m creating a new pool with VLANs in the range 500 - 550.

Notice the ‘allocation mode’ option can be set to either ‘static’ or ‘dynamic’. Which one should you choose?

• Static VLAN allocation mode should be used if the VLAN pool in question will be used for bare metal hosts or other non-virtualised devices. Later, when we create EPGs, we will manually assign a VLAN from the static pool to the EPG and port.

• Dynamic allocation mode is used when connecting VMs into the fabric , specifically when using VMM integration with the hypervisor management system. In that case, a VLAN will be dynamically assigned to the port group that gets created on the Distributed Virtual Switch. I’ll cover this in more detail in a future post.

In my example, I want to connect a bare metal host, so I choose the static option.

2. Create a Domain

A domain defines the ‘scope’ of a VLAN pool, i.e. where that pool will be applied. A domain could be physical, virtual, or external (either bridged or routed) - in this example I am creating a physical domain as I want to use my VLAN pool for bare metal host connectivity. When you configure a domain, you reference the VLAN pool you created earlier:

Don’t worry about the Attachable Entity Profile in this step - I’ll cover that next.

3. Create an Attachable Access Entity Profile

Wait…a what?! The AAEP (as I will will refer to it from now on) is a way of grouping together multiple domains that may need to be associated with an interface. In my example, I will refer to my physical domain when creating my AAEP:

Note that I could attach my AAEP to an interface as part of this step - however, I’m not doing that here so I can explain how that works properly in later steps.

4. Define Interface Policies and an Interface Policy Group

An interface policy is simply a policy which specifies whether a particular feature will be enabled. Examples of features controlled by interface policies are LACP, CDP, LLDP and Storm Control. Interface policies, plus the AAEP we created in the last step are tied together using an interface policy group. You can think of an interface policy group as a port-profile in regular NX-OS environments. In my example, I want to use my AAEP and other policies on a regular access port (not a port-channel or vPC), so specifically I create an access port policy group:

In my example, I am referring to a link level policy that specifies 1Gbps port speed as well as a policy that enables LLDP.

5. Define an Interface Selector and Interface Profile

At this point, we are ready to start applying the policies we have created to one or more interfaces on one or more leaf nodes. The first thing we need to do is to create an interface profile. An interface profile is a policy which contains a port selector. The port selector is used to specify a particular port number - note that we are not yet specifying which leaf node we want to apply the policy to (that happens in the next step). In the example below, I am creating an interface profile containing a port selector referring to port 1/27. Note that the selector policy also refers to the interface policy group I created in the last step.

6. Define a Switch Profile

The final step in our access policy journey is to apply the policies we have created to one or more switching nodes (e.g. Nexus 9396 leaves). To do this, we need to define a switch profile. When I create a switch profile I need to refer to two things: the actual switching node I want to apply policy to and the interface profile I defined in the last step. In the example below, I have created a switch profile referring to leaf switch 101 and the ‘port-27’ interface profile I created earlier:

OK, so just to recap - what have we actually achieved by doing this? Fundamentally, we have ‘provisioned’ a range of VLANs on a specific leaf port. At this point, we have not actually enabled any VLANs on the port in question, so no traffic would flow yet (we will need to provision an EPG from the ‘tenants’ tab for that). But the access policy configuration shown above must be done first, before you can provision EPGs. The good news is that this configuration is not something that should be needed on a regular basis - you provision the VLANs and interface policies on the nodes and ports that you need in advance and then simply create EPGs when hosts need to be connected.

Thanks for reading - next time, I’ll look at a simple example of how to get two hosts communicating through the fabric.

Learning ACI - Part 5: Private Networks, Bridge Domains and Subnets

Welcome to part 5 of this blog series - so far I have covered the following topics:

• Part 1 contained a very brief overview of ACI and what the series would cover.

• In part 2, I talked through the fabric bring-up process.

• Part 3 was all about getting familiar with the APIC controller GUI.

• In the last blog, part 4, we took a look at some of the most important policy constructs within ACI - application profiles, EPGs, contracts and filters.

Next on the list, we’ll have a look at some networking concepts within ACI - namely private networks, bridge domains and subnets. Some of these are terms that you might not recognise, so what are they for?

Private Networks

In ACI, a private network - sometimes known as a context - is used to define a layer 3 forwarding domain within the fabric. A private network can support IP ranges which overlap with another private network on the fabric. If you’re thinking you’ve seen this somewhere before, you’re right - private networks look very similar to VRFs in the traditional networking world. In fact, a private network / context is a VRF - if you go to a fabric enabled Nexus 9K switch, you can see that each private network you define is instantiated as a VRF. Generally speaking, each tenant will have at least one private network defined.

A private network is defined under a tenant, under the ‘Networking’ side menu:

There are a number of policies you can associate with a private network, including OSPF and BGP timers, as well as how long end points should be retained. One interesting option within the private network configuration is the Enforced / Unenforced option. The default option here is ‘Enforced’ - this means that all security rules (contracts) associated with this private network will be enforced. It is however possible to disable this (Unenforced) which means that contracts will not be enforced within this private network. This is like having an ACL with ‘permit any’ at the end of it, which of course means that inter-EPG communication will be allowed without the need for contracts.

Bridge Domains

Private networks are fairly to easy to understand, so what about bridge domains? A bridge domain is simply a layer 2 forwarding construct within the fabric, used to define a flood domain. You’re probably now thinking “that’s just like a VLAN” - and you’d be right, except that bridge domains are not subject to many of the same limitations as VLANs, such as the 4096 segment limit. In addition, bridge domains have (by default) a number of enhancements such as optimised ARP forwarding and unknown unicast flooding.

When you define a BD (from the same ‘Networking’ side menu we used to define our Private Network), a number of forwarding options are available if you select the ‘custom’ option from the drop down menu:

Let’s look at these options in more detail:

• L2 Unknown Unicast: This option controls whether the ACI spine proxy database will be used for unknown unicast traffic (the default), or whether ‘traditional’ flooding of unknown unicast traffic should occur. You would typically use ‘flood’ mode if extending the BD to an external network (such as to a legacy environment).

• Unknown Multicast Flooding: This controls whether traffic should be flooded to all ports on a leaf node, or whether it should be constrained to only ports where say a multicast receiver resides.

• ARP Flooding: By default, ACI will convert ARP broadcast traffic into unicast traffic and send it to the correct leaf node. This option can be disabled if traditional ARP flooding is needed.

• Unicast Routing: Unicast routing is enabled by default and is required if the fabric is performing routing for a BD (e.g. if a subnet is defined). You might wish to disable this if the fabric is not performing routing for a BD (i.e. a gateway resides outside the fabric, such as on a firewall or external router).

A bridge domain is always associated with a private network - you’ll need to select one when you create the BD.

Subnets

When you create a bridge domain, you have the option of defining a subnet under the BD:

When you define a subnet under a BD, you are creating an anycast gateway - that is, a gateway address for a subnet that potentially exists on every leaf node (if required). In the traditional world, think of this as an SVI interface that can exist on more than one node with the same address.

You’ll also notice that we can control the scope of the subnet - private, public or shared. These are used as follows:

• Private: A private subnet is one that is not advertised to any external entity via a L3 outside and will be constrained to the fabric.

• Public: A public subnet is flagged to be advertised to external entities via an L3 outside (e.g. via OSPF or BGP).

• Shared: If a subnet is flagged as shared, it will be eligible for advertisement to other tenants or contexts within the fabric. This is analogous to VRF route leaking in traditional networks.

Hopefully this gives you a good overview of the main networking concepts within ACI - in part 6, I’m going to walk you through creating an access policy - this is a crucial part of attaching hosts and other devices to the fabric.

Learning ACI - Part 4: Application Profiles, EPGs, Contracts and Filters

In this post, we’ll take a closer look at some of the most important constructs within the ACI solution - application profiles, End Point Groups (EPGs), contracts and filters. Hopefully you’ve taken a look at the other parts in this series - in part 1, I gave a brief overview of ACI and what I would be covering in the series. Part 2 discussed the fabric bring-up process, with part 3 giving a short tour of the APIC.

End Point Groups

You’ll hopefully be aware by now that ACI introduces a number of new concepts that provide new ways of defining connectivity using a policy based approach. Of these, arguably the most important is the End Point Group (EPG). What is an EPG and how is it used within ACI? Well, the basic idea is that you define groups of servers, VMs, IP storage or other devices that have common policy requirements - once these devices are grouped together, it becomes much easier to define and apply policy to the group, rather than to individual end points.

Let’s imagine you have a standard three-tiered type of application (I’m aware that this is a rather overused example, but it serves its purpose in this case):

Within each of our application tiers, we have a number of ‘end points’ - we don’t really care too much what those are (bare metal, VMs, etc). The important thing is that these end points all require the same policy to be applied. If we can identify which group these end points need to be part of, we can create the corresponding EPGs within ACI:

How do we define which end points reside in which EPG? That can be done either statically or dynamically and depends somewhat on whether we are talking about bare metal hosts or virtual machines. If these are VMs, then ACI can integrate closely with the virtual machine manager (such as vCenter), with the process of attaching a VM to a network resulting in that VM becoming part of the desired EPG. Now that we have our EPGs defined and end points residing within them, what comes next? At this point, there are two important concepts to understand:

1) Within an EPG, communication is free-flowing - an end point can communicate freely with another end point within the same EPG, regardless of where those EPs sit physically or virtually.

2) Between EPGs, no communication is permitted by default - if you do nothing else at this point, an EP residing in the “Web” EPG will not be able to communicate with an EP in the “App” EPG.

Now that we know these rules, it follows that we need to put something extra in place if we want inter-EPG communication. This is where contracts and filters come in.

Contracts and Filters

What does a contract do? Essentially, it is a policy construct defining the potential communication between the EPGs on the system. A contract can be very restrictive (for example, allowing only one port), or it can be completely open (“permit any”) depending on your requirements.

A contract will refer to one or more filters. The filter is the construct used to actually define the specific protocols and ports required between EPGs. In the following example, I am creating a filter with a single entry - HTTP:

This filter can now be referenced when creating a contract - in this case, the filter is a ‘subject’ that gets defined under the contract:

Note that you can apply the contract to both directions if bi-directional communication is required between EPGs.

Application Profiles

OK, so we have some EPGs defined and the contracts / filters we need to communicate between them. We now need some way of tying all of this together - an Application Profile is simply a collection of EPGs and the policies needed to communicate between them:

During the process of creating an application profile and its associated EPGs, you will associate one or more contracts with the EPGs that you create. When a contract is associated with an EPG, that EPG can either provide or consume the contract. Let’s imagine our “Web” EPG is providing a service using HTTP (which seems perfectly reasonable), and that this service needs to be accessed from the “App” EPG. In that case, the “Web” EPG would provide a contract (and its associated filter) allowing HTTP. The “App” EPG would consume the very same contract. The APIC actually gives us a nice graphical representation of this relationship:

Contract Scope

When defining a contract, you can specify the scope of that contract - for example, Private Network, Application Profile, Tenant or Global. The scope of a contract defines the level of policy enforcement between the various entities within ACI. As an example, let’s say we create a single contract called “web-to-db”. Now let’s say we create two application profiles (app1 and app2), with the EPGs within those app profiles providing and consuming our single contract, as shown here:

If our contract scope is set to “application profile”, this means that communication is not permitted between application profiles (for example, no communication between ‘App1-Web’ and ‘App2-Web’). If we were to set the scope of our contract to ‘Private Network’, then communication would be possible between all four EPGs (assuming everything is part of one private network).

That’s it for this instalment - next up, we’ll look at some of the networking elements within ACI, such as bridge domains and private networks (contexts). Part 5 is here.

Learning ACI - Part 3: Getting Familiar with the APIC

Hello again! Hopefully you’re back after reading parts one and two of this series - in the first post, I covered a basic introduction to ACI and then in the last post, we looked at how the fabric initialisation and discovery process works. After the incredible excitement of building an ACI fabric, what do we do with it? Well, if you’ve never seen what an APIC looks like before, in this post we’ll have a look around the APIC and start to find our way around the GUI.

The Dashboard

After initialising your APICs, you can browse to any one of the APIC out-of-band IP addresses (there is no ‘cluster’ IP address with the APIC) and log on. The first thing you will see is the main ‘dashboard’ screen, which looks something like this:

At the top, we get an overall view of the system health - on my fabric, I have an overall health score of 91, so that tells me that there is something needing attention. Further down, we can see a ‘node’ view which shows the individual switching nodes in the fabric and their corresponding health scores. From here, I can see that one of my nodes has a health score of 87, so that is a good place to start in investigating my issue. Clicking on the health score in this section takes you directly to a more detailed view of the node in question, where you can investigate faults and other relevant information.

At the bottom of the screen we have the tenant health view, indicating where there may be issues with any particular tenant (for example, a problem with an application profile). Once again, clicking on the health score takes you directly to that tenant to allow you to investigate.

Moving to the right of the screen, you’ll see a couple of sections showing fault counts by domain and by type. You can clearly see from here how many faults of each category are present and once more, you are able to drill down further into the list of faults.

Finally, on the bottom right we have the APIC cluster health. In the example, I have only one APIC but you can see that it is in service and ‘fully fit’.

The ‘Tenants’ Tab

Moving along to the Tenants tab at the top of the page - after clicking here, you’ll see a screen similar to the following:

In the ‘sub-menu’ under the main Tenants tab, you’ll see a list of the most commonly used tenants. Under here, you will configure application profiles (EPGs, etc), networking (private networks, bridge domains, etc), security policies (contracts and filters) as well as a host of other items relating to troubleshooting and monitoring (SPAN and atomic counters for example). If you are using automated services insertion, that will also be configured under the tenant section.

The ‘Fabric’ Tab

The Fabric tab is used for a number of purposes: viewing fabric inventory information, fabric policy configuration (for example, altering the MTU used inside the fabric) and access policies (used to configure interfaces, VLAN pools, domains and so on - more of which in subsequent posts).

Clicking on the inventory sub-menu and then Topology, you’ll see a graphic representation of the entire fabric:

Right-clicking on any of the switches or APICs in the picture will allow you to perform some basic tasks such as decommissioning the node or configuring interfaces. Double clicking the node will take you to a more detailed view and allow you to perform further tasks.

If we now expand the ‘Pod 1’ menu item on the left and then expand one of the nodes, we can see a whole host of information about the device in question. For example, we can see information about the chassis, Supervisor, linecards, power supplies for the device, interface information, protocol information (BGP, IS-IS, OSPF and more) and processes running. In the following example, I’m looking at the IS-IS routes present on the fabric:

The ‘VM Networking’ Tab

VM Networking is where we configure ‘VMM domains’ - essentially, this is the integration with the virtualisation / hypervisor layer. I’ll be covering this in more detail in a later post, but assuming you have a VMM domain configured, from here you can view info such as ESXi hosts, virtual machines running on those hosts and the physical interfaces in use. You can also see info about the distributed virtual switch being used and any port groups that have been created.

The ‘L4-L7 Services’ Tab

This tab is mainly used to import device packages for a services device (such as a firewall or load balancer) into the APIC. Most of the actual L4-L7 services configuration (e.g. service graphs) are configured from the Tenants tab, so I won’t spend any time here.

The ‘Admin’ Tab

As you might guess, the Admin tab is the place where you will perform general administration tasks - for example, AAA config, firmware management and external monitoring destinations (Syslog, SNMP, etc) can all be managed from this tab.

What about the API?

So far we’ve only looked at the GUI, so those of you who are interested in using the APIC API might feel a bit left out. I’m hoping to cover some details about how to use the API in a later post, but for now there are a couple of handy tools that the APIC GUI contains to help you get familiar with how the API works.

The first of these tools is the API Inspector - you can get to this by clicking on the ‘Welcome’ button on the top right of the screen.

What this tool does is display the API calls made whenever you perform a task in the GUI. This is extremely useful because you can carry out a task in the GUI, capture it using the API Inspector and then use the JSON output to help you build your own scripts.

The second tool is the ability to ‘save’ the XML from almost any object in the GUI. Right clicking on an object (for example an application profile, EPG, Bridge Domain, etc) and selecting ‘Save As…’ will allow you to save the XML associated with that object to a local file. Again, you can use this XML to become familiar with the API, use in your own scripts, and so on.

That’s it for now - hopefully you’ve enjoyed this quick tour around the APIC. Now that you’re familiar with the basics, we can move on to the next steps - understanding application profiles, EPGs, contracts and filters. Check out part 4 for more on that.

Thanks for reading!

Learning ACI - Part 2: Bringing Up A Fabric

Welcome to the second part of my blog series on ACI. Before we start to delve into policies, contracts, filters and all the other goodness to be found in ACI, we need to actually provision a fabric and bring it online. As it turns out, this is a really easy process and one which should take only a short amount of time. Obviously before we begin, we need to make sure everything is cabled correctly (leaf nodes to spine nodes, APIC controllers to leaf nodes, APIC out of band connectivity, etc).

Before we start, there are a few parameters which are useful to think about in advance and have to hand during the setup process:

• The name you want to give your fabric.

• Controller Name - the names you want to give to your APIC controllers (normally there will be three of these).

• TEP IP address pool - this range will be used by the fabric to automatically allocate TEP (Tunnel End Point) addresses. Every node (leaf and spine) will be allocated at least one TEP address. It’s generally a good practice to dedicate a unique range for this.

• Multicast address pool - this pool of addresses will be used for multicast traffic through the fabric.

• VLAN ID for the infrastructure network - this is the VLAN ID used for internal communication in the fabric, but also may be extended outside the fabric for communication to AVS virtual switches. Therefore, it’s best to allocate an unused VLAN (be aware that this VLAN overlaps with the ‘reserved’ range on some other Nexus platforms).

• Management IP address and gateway - this will be the out-of-band address you will use to browse to the APIC itself.

• Admin password.

You will generally use the CIMC utility (this is essentially a UCS C series server) to access a KVM session so you can set up the APIC - once you have started the setup process, you’ll see a simple script which will ask you for the parameters above:

 Cluster configuration ...
  Enter the fabric name [ACI Fabric1 #1]: 
  Enter the number of controllers in the fabric (1-16) [3]: 
  Enter the controller ID (1-3) [2]: 
  Enter the controller name [apic2]: 
  Enter address pool for TEP addresses [10.0.0.0/16]: 
  Enter the VLAN ID for infra network (1-4094)[4093]:     
  Enter address pool for BD multicast addresses (GIPO) [225.0.0.0/15]: 

Out-of-band management configuration ...
  Enter the IP address for out-of-band management: 192.168.10.2/24
  Enter the IP address of the default gateway [None]: 192.168.10.254
  Enter the interface speed/duplex mode [auto]:

Administrator user configuration...
  Enable strong passwords? [Y]
  Enter the password for admin:

Once you have completed the script, the APIC initialises and after a short period of time, you should be able to browse to it using the out-of-band address you configured earlier. Now that you have a single APIC up and running, what’s next? We can now start to discover the physical switching nodes in our fabric.

In the APIC GUI, click on the Fabric tab and then the Inventory sub-menu. On the left hand side, click on Fabric Membership. If all is well, you should see a single leaf node listed in here - this is the physical switch that your APIC is connected to. Notice that there is no node ID or node name listed, and the IP address is set to 0.0.0.0. What this all means is that this switch is not yet registered - in order to register the node and add it to the network, you simply need to assign a node ID and node name by double clicking on those fields. Once done, you should - after a minute or so - see an IP address appear for that node. Notice that this address is taken from the TEP range you specified during the setup process. This means that the node is now ‘registered’.

OK, so we now have a single leaf node registered - how do we discover the rest of our network? Well, once you have registered the first node, you should magically start to see additional switches (spine nodes) pop up on the Fabric Membership page. Go ahead and register these in the same way, which in turn will lead to the remaining switches appearing and available for registration. What you should eventually end up with is something that looks like this:

Now at this point, we have our full switching topology discovered….but we still only have one APIC. As your other APICs are very likely to be connected to different leaf switches, we needed to wait until those switches were discovered before we could initialise the remaining APICs and form the cluster. So you can now run the setup script on the other APICs, using the same basic info as before, but making sure you use different controller IDs, management IP address, etc.

Now that your remaining APICs are up and running, your fabric is pretty much ready to go - if you go back to the Fabric Inventory **section in the APIC GUI and click on **Topology, you should see a graphical representation of your fabric:

Before you go any further, you may wish to configure your switching nodes with management IP addresses so you can SSH to them directly - do this by clicking on the Tenants tab and then the ‘Mgmt’ tenant. On the left hand side you’ll find Node Management Addresses, which will allow you to configure management IPs for every node in the fabric. Also under the Mgmt tenant, you’ll find Out-Of-Band Contracts listed under the Security Policies menu. You’ll need to configure at least one contract here to allow traffic into your OOB management interfaces. Finally, under Node Management EPGs, you can add the OOB contract to your OOB EPG.

What is really happening during discovery?

Now that we’ve looked at how to discover our fabric, it would be nice to understand exactly how this process is happening under the covers. It’s actually pretty simple - we use LLDP to determine what is connected to each port. The process is as follows:

1) When the first leaf initialises, it sends LLDP messages on its active ports to determine which port the APIC is connected to.

2) The leaf receives an LLDP message back from the APIC indicating the node type.

3) The leaf node sends a DHCP Discover message to the APIC - the APIC responds with an IP address from a locally configured pool (the pool you configured during the APIC setup script).

4) The leaf node uses this IP address as the Tunnel End Point (TEP) address.

5) Once the first leaf node has been discovered and registered by the APIC, a similar process is followed for the remaining devices in the fabric.

Thanks for reading - the next step is to have a look at the APIC and get familiar with it. Check out part 3 for this!