So you've introduced a Nexus stack in your datacenter instead of using those 3750x switches deployed because well..you want DC grade switches running in your DC.

One thing you may come across is particular tuning for different types of traffic passing through your new NX switches. Whether it be iSCSI, NFS, FCoE, or just normal web traffic with little payload. With different traffic, you may think about classifying this traffic with QoS and putting policies in place. Lets start simple by classifying your iSCSI traffic for jumbo frames. The first thing we need to do is to match the traffic with an ACL:

ip access-list jumbo-frames
  10 permit ip 10.0.20.0/24 any
  20 permit ip 10.0.45.0/24 any
  30 permit ip 172.19.25.0/24 any
ipv6 access-list jumbo-frames6
  10 permit ipv6 2001:db8:10::0/64 any

Now we can start by creating a class map for the QoS type of traffic using the above referenced ACL:

class-map type qos match-any jumbo-support
  match access-group name jumbo-frames
class-map type qos match-any jumbo-support6
  match access-group name jumbo-frames6

Now we have the class, where we can point out the networks we specified for jumbo frames. Now we can create a policy and mark the traffic accordingly. To keep it simple for this example, we can set a QoS group of 2, which is an internal label local to the Nexus. You can learn more about them in this article by Cisco. The next steps are to use these QoS markings to queue the traffic and set bandwidth, etc. First we will start by creating our first policy mapping, to do so:

policy-map type qos jumbo-support-qos-policy
  class jumbo-support
    set qos-group 2
  class jumbo-support6
    set qos-group 2

Now the traffic is officially marked. But if we were to apply it at this point, it still isn't doing anything. We configured the classes and applied them to the traffic. But we have no network or queuing policies to actually act on those class markings. So lets start off by configuring the queueing policies to allow at least 50% bandwidth for the jumbo frame traffic. By default, the default class is set to 100%, so the following policy will limit that also to 50:

class-map type queuing jumbo-support
  match qos-group 2
policy-map type queuing jumbo-support-queuing-policy
  class type queuing jumbo-support
    bandwidth percent 50
  class type queuing class-default
    bandwidth percent 50

And finally, to configure the network policies to actually allow for the jumbo frames, using a policy to set the MTU to a colossal size to accomodate:

class-map type network-qos jumbo-support
  match qos-group 2
policy-map type network-qos jumbo-support-network-qos-policy
  class type network-qos jumbo-support
    mtu 9216
  class type network-qos class-default

So now the class and policy mappings are complete. We are ready to apply them to your new nexus switches. To do this, we need to override the default policies. To override, you need to specify your three policies (qos, queueing, and network-qos) as the preferred system QoS policies. With the above policy examples, this looks like:

system qos
  service-policy type qos input jumbo-support-qos-policy
  service-policy type network-qos jumbo-support-network-qos-policy
  service-policy type queuing input jumbo-support-queuing-policy
  service-policy type queuing output jumbo-support-queuing-policy

So there you have it. Now your jumbo frames as specified in our ACL for IPv4 and IPv6 traffic, we now have set the proper markings to classify traffic to be used with our larger MTU size, with queuing and all. To verify that it is working, you can use the 'show queuing interface' command, like so:

prod-# show queuing int e 1/25
Ethernet1/25 queuing information:
  TX Queuing
    qos-group  sched-type  oper-bandwidth
        0       WRR             50
        2       WRR             50

  RX Queuing
    qos-group 0
    q-size: 100160, HW MTU: 1500 (1500 configured)
    drop-type: drop, xon: 0, xoff: 0
    Statistics:
        Pkts received over the port             : 76808
        Ucast pkts sent to the cross-bar        : 67186
        Mcast pkts sent to the cross-bar        : 9622
        Ucast pkts received from the cross-bar  : 104692
        Pkts sent to the port                   : 3995088
        Pkts discarded on ingress               : 0
        Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

    qos-group 2
    q-size: 100160, HW MTU: 9216 (9216 configured)
    drop-type: drop, xon: 0, xoff: 0
    Statistics:
        Pkts received over the port             : 4342
        Ucast pkts sent to the cross-bar        : 4342
        Mcast pkts sent to the cross-bar        : 0
        Ucast pkts received from the cross-bar  : 6337
        Pkts sent to the port                   : 8375
        Pkts discarded on ingress               : 0
        Per-priority-pause status               : Rx (Inactive), Tx (Inactive)

You want to make sure you don't have any discarded packets or other signs of issues, which could mean many things such as improper configuration of the classes, policies or issues on the ports connecting to your ESXi server for example. Like mentioned above, this is a basic set of policies to get the point across and could be expanded much more. You don't have to match based on an access list, you could also match based on protocol or other traits, leading to some more advanced markings based on that.

Let me know if you have any questions or improvements to this basic tutorial.

Cheers!