Tutorial: How to use Cisco MQC & NBAR to filter websites like Youtube

ardenpackeer.com

Helping You Become a Network Ninja

Search

// you’re reading...

Featured

Tutorial: How to use Cisco MQC & NBAR to filter websites like Youtube

By Arden Packeer, CCIE #20716⋅ December 14, 2007 ⋅

Email this post ⋅

Print this post ⋅ Post a comment

Site Search Tags: dynagen, dynamips, filtering, flash, http, mime, mqc, nbar, qos, web

Thanks for visiting! If you're new here, you may want to subscribe to my RSS feed. This blog posts regular tutorials, news, and study tips about networking, especially about Cisco CCIE related topics. Go ahead, subscribe to the rss feed! You can also receive updates from this blog via email. Thanks for visiting!

I was asked a great question by one of my clients regarding filtering of websites. He had filtered youtube and google video at his proxy server but with the number of different video sites popping up (metacafe, jibjab etc etc), his filters just couldn’t keep up…and neither could his bandwidth!

One solution to this problem is the use of Cisco’s Network Based Application Recognition (NBAR). NBAR is a deep packet inspection and classification engine. It was first introduced in experimental versions of IOS v12.1 and can be used with Cisco’s Modular Quality Of Service Command Line (MQC).

In this article we will look at using MQC to filter websites. I will demonstrate using the match protocol http command to match a URL, a host or MIME type. We will use the following topology for demonstration:

R3 will act as a webserver and R1 as a client. The filtering will be applied on R2. You can download the dynamips .net file the following topology here.
R1 Base Configuration:

hostname R1
!
int s1/0
 ip add 10.0.12.1 255.255.255.0
 no shut
!
router ospf 1
 network 10.0.12.1 0.0.0.0 area 0

R2 Base Configuration:

hostname R2
!
int s1/0
 ip add 10.0.12.2 255.255.255.0
 no shut
!
int s1/1
 ip add 10.0.23.2 255.255.255.0
 no shut
!
router ospf 1
 network 10.0.12.2 0.0.0.0 area 0
 network 10.0.23.2 0.0.0.0 area 0

R3 Base Configuration:

hostname R3
!
int s1/0
 ip add 10.0.23.3 255.255.255.0
 no shut
!
int f0/0
 ip add 192.168.1.100 255.255.255.0
 no shut
!
router ospf 1
 network 10.0.23.3 0.0.0.0 area 0
!
ip http server
ip http path flash:

We have set up R3 as a webserver. Details on how to setup R3 as a webserver using IOS can be found here.

R3#sh run | in ip http
ip http server
no ip http secure-server
ip http path flash:

R3#dir
Directory of flash:/

    1  -rw-          90                    <no>  picture.gif
    2  -rw-         329                    <no>  picture.jpg
    3  -rw-         174                    <no>  index.html

8388604 bytes total (8387812 bytes free)
</no></no></no>

Basic HTTP Filtering using NBAR

Lets set up basic http filtering with MQC on R2.

R2(config)#class-map match-all MATCH-HTTP
R2(config-cmap)#match protocol http
R2(config-cmap)#exit
R2(config)#policy-map HTTP-POLICY
R2(config-pmap)#class MATCH-HTTP
R2(config-pmap-c)#set dscp af13
R2(config-pmap-c)#exit
R2(config-pmap)#int s1/0
R2(config-if)#service-policy input HTTP-POLICY

In the code above we have a class map called MATCH-HTTP. The match protocol http command tells NBAR to match the http protocol. This will match all http traffic. The MATCH-HTTP class is then utilized in the HTTP-POLICY policy map. This policy map is used to set a DSCP marking on all traffic that matches the MATCH-HTTP class (ie all http traffic). The policy is then implemented on R2’s s1/0. Traffic is inspected and marked as it comes into that interface.

We can check how many packets have been marked using the show policy-map command.

R2#sh policy-map int s1/0
 Serial1/0 

  Service-policy input: HTTP-POLICY

    Class-map: MATCH-HTTP (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: protocol http
      QoS Set
        dscp af13
          Packets marked 0

    Class-map: class-default (match-any)
      2 packets, 168 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any
R2#

Lets generate some http traffic, and see if our policy marks some packets.

R1#copy http://10.0.23.3/index.html null:
Loading http://10.0.23.3/index.html
174 bytes copied in 0.544 secs (320 bytes/sec)

R2#sh policy-map int s1/0
 Serial1/0 

  Service-policy input: HTTP-POLICY

    Class-map: MATCH-HTTP (match-all)
      5 packets, 344 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: protocol http
      QoS Set
        dscp af13
          Packets marked 5

    Class-map: class-default (match-any)
      124 packets, 10340 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

We used the copy http://10.0.23.3/index.html null: command to generate some http traffic. We can see above that 5 packets were generated and were marked as af13. All other traffic will fall into the class-default class. With the packets marked, we could forward them or drop them.

Instead of matching all of the http protocol we can use NBAR to look further into the packet and classify or drop packets based on the host requested.

Match protocol HTTP host

The match protocol HTTP url command is used to match a url. It takes a regular expression as an argument. For example:

match protocol http host *youtube.com*
! This would match anything in youtube.com like http://www.youtube.com or http://video.youtube.com
!
match protocol http host *google*
! This would match anything with google in the host like http://mail.google.com or
http://www.google.com.au
!
match protocol http host google*
! This would match http://google.com but not http://video.google.com

Lets set up R2 to filter based on a host.

R2(config)#class-map MATCH-HTTP
R2(config-cmap)#no match protocol http
R2(config-cmap)#match protocol http host 10.0.23.3

R2#clear counters s1/0
Clear "show interface" counters on this interface [confirm]
*Mar  1 00:04:42.071: %CLEAR-5-COUNTERS: Clear counter on interface Serial1/0 by console
R2#
R2#sh policy-map int s1/0
 Serial1/0 

  Service-policy input: HTTP-POLICY

    Class-map: MATCH-HTTP (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: protocol http host "10.0.23.3"
      QoS Set
        dscp af13
          Packets marked 0

    Class-map: class-default (match-any)
      1 packets, 84 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

We’ve cleared the counters on R2, so lets generate some traffic on R1 again.

R1#copy http://10.0.23.3/index.html null:
Loading http://10.0.23.3/index.html
174 bytes copied in 0.596 secs (292 bytes/sec)

R2#sh policy-map int s1/0
 Serial1/0 

  Service-policy input: HTTP-POLICY

    Class-map: MATCH-HTTP (match-all)
      5 packets, 344 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: protocol http host "10.0.23.3"
      QoS Set
        dscp af13
          Packets marked 5

    Class-map: class-default (match-any)
      64 packets, 5300 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

We can see here it matched 5 packets based on the host. We can use this to match whole sites like youtube.com or video.google.com.

Match protocol HTTP url

We can match strings AFTER the host portion of a URL using the match protocol http url command. It also takes a regular expression as an argument. For example:

match protocol http url *video*
! This would match http://www.cisco.com/video/index.php or
http://www.google.com/stuff/video.html
!
match protocol http url video*
! This would match http://www.cisco.com/video but not http://www.cisco.com/stuff/video.html
! because stuff precedes the video portion of the url and in the expression above we have said
! it has to start with the string video
!
match protocol http url *.jpeg|*.jpg|*.gif
! This would match any .jpeg or .jpg or .gif extention in the url

Lets set up R2 to match based on a URL.

R2(config)#class-map MATCH-HTTP
R2(config-cmap)#no match protocol http host 10.0.23.3
R2(config-cmap)#match protocol http url *.jpg

As you can see above we have used the match protocol http url function of NBAR to match any url that ends in a .jpg. This effectively blocks jpeg images (unless they have a different extension).

Let test it, before we send some traffic we’ll reset the counters on the interface.

R2#clear counters s1/0
Clear "show interface" counters on this interface [confirm]
*Mar  1 00:43:39.135: %CLEAR-5-COUNTERS: Clear counter on interface Serial1/0 by console
R2#
R2#sh policy-map int s1/0
 Serial1/0 

  Service-policy input: HTTP-POLICY

    Class-map: MATCH-HTTP (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: protocol http url "*.jpg"
      QoS Set
        dscp af13
          Packets marked 0

    Class-map: class-default (match-any)
      1 packets, 84 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

If we request a gif file we shouldn’t match the class MATCH-HTTP. Lets test that first.

R1#copy http://10.0.23.3/picture.gif null:
Loading http://10.0.23.3/picture.gif
90 bytes copied in 0.644 secs (140 bytes/sec)

R2#sh policy-map int s1/0
 Serial1/0 

  Service-policy input: HTTP-POLICY

    Class-map: MATCH-HTTP (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: protocol http url "*.jpg"
      QoS Set
        dscp af13
          Packets marked 0

    Class-map: class-default (match-any)
      18 packets, 1209 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

Great Success! Looks pretty good. Now lets try a .jpg extension. We should match this.

R1#copy http://10.0.23.3/picture.jpg null:
Loading http://10.0.23.3/picture.jpg
329 bytes copied in 0.820 secs (401 bytes/sec)

R2#sh policy-map int s1/0
 Serial1/0 

  Service-policy input: HTTP-POLICY

    Class-map: MATCH-HTTP (match-all)
      7 packets, 433 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: protocol http url "*.jpg"
      QoS Set
        dscp af13
          Packets marked 7

    Class-map: class-default (match-any)
      22 packets, 1469 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

Awesome! You can see above we matched based on a URL.

match protocol http mime

We can also use the match protocol http mime to match internet mime types. The mime type has to be the same mime type that the web server responds with. For a list of valid mime types check out: http://www.sfsu.edu/training/mimetype.htm. Lets look at an example:

match protocol http mime image/jpeg
! This would match jpeg,jpg,jpe,jfif,pjpeg, and pjp types
!
match protocol http mime image/jpg
! This would not match anything as it is not a proper mime type. Get a list of the mime types
! here: http://www.sfsu.edu/training/mimetype.htm
!
match protocol http mime image*
! This would match all image mime types
!
match protocol http mime application/x-shockwave-flash
! This would not only match swf flash movies, but all of flash.

Lets set up R2 to filter the image/jpeg mime type:

R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
R2(config)#class-map MATCH-HTTP
R2(config-cmap)#no match protocol http url *.jpg
R2(config-cmap)#match protocol http mime ?
  WORD  Enter a string as the sub-protocol parameter

R2(config-cmap)#match protocol http mime image/jpeg
R2(config-cmap)#exit
R2(config)#exit

Once again, we’ll clear the counters so we can verify that this works correctly.

R2#clear counters s1/0
Clear "show interface" counters on this interface [confirm]
*Mar  1 01:12:10.759: %CLEAR-5-COUNTERS: Clear counter on interface Serial1/0 

R2#sh policy-map int s1/0
 Serial1/0 

  Service-policy input: HTTP-POLICY

    Class-map: MATCH-HTTP (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: protocol http mime "image/jpeg"
      QoS Set
        dscp af13
          Packets marked 0

    Class-map: class-default (match-any)
      1 packets, 84 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

On R1 lets generate some traffic. A gif file will be requested first. This should not match our policy.

R1#copy http://10.0.23.3/picture.gif null:
Loading http://10.0.23.3/picture.gif
90 bytes copied in 0.808 secs (111 bytes/sec)

R2#sh policy-map int s1/0
 Serial1/0 

  Service-policy input: HTTP-POLICY

    Class-map: MATCH-HTTP (match-all)
      0 packets, 0 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: protocol http mime "image/jpeg"
      QoS Set
        dscp af13
          Packets marked 0

    Class-map: class-default (match-any)
      10 packets, 689 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

All good! Ok lets do the final test and actually request a jpeg image and see if it matches our policy.

R1#copy http://10.0.23.3/picture.jpg null:
Loading http://10.0.23.3/picture.jpg
329 bytes copied in 1.216 secs (271 bytes/sec)

R2#sh policy-map int s1/0
 Serial1/0 

  Service-policy input: HTTP-POLICY

    Class-map: MATCH-HTTP (match-all)
      5 packets, 220 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: protocol http mime "image/jpeg"
      QoS Set
        dscp af13
          Packets marked 5

    Class-map: class-default (match-any)
      16 packets, 1162 bytes
      5 minute offered rate 0 bps, drop rate 0 bps
      Match: any

You can see above that the jpeg image was matched. It works!

Putting it all together

So lets put it all together. We can use all three match protocol http commands in a match-any class map. For example:

class-map match-any INTERNET-SCUM
 match protocol http host *youtube.com*|*video.google.com*
 match protocol http mime video/flv|video/x-flv|video/mp4|video/x-m4v|audio/mp4a-latm
 match protocol http mime video/3gpp|video/quicktime
 match protocol http url *.flv|*.mp4|*.m4v|*.m4a|*.3gp|*.mov
! uncomment below if you don't want ANY flash.
! match protocol http mime application/x-shockwave-flash
! match protocol http url *.swf
!
policy-map NOINTERNETVIDEO
 class INTERNET-SCUM
  drop
!
int s1/0
 service-policy input NOINTERNETVIDEO
!

This would match any traffic going to youtube or video.google.com, or any flash applications, or common video mime types, and any swf (flash or flash movie) files! Be aware that NBAR does make your router take a hit in CPU processor usage, I’d suggest evaluating your processor usage before using this in production.

HTH! Now back to labs!

Summary:

Use the match http protocol command to match the http protocol.
match protocol http host matches the host portion
match protocol http url matches the url after the hostname
match protocol http mime matches mime types

Resources
Webserver – Dynamips .net configuration file
QOS HTTP Filtering – R1 Final Configuration
QOS HTTP Filtering – R2 Final Configuration
QOS HTTP Filtering – R3 Final Configuration

Amazon.com Widgets

Discussion

16 comments for “Tutorial: How to use Cisco MQC & NBAR to filter websites like Youtube”

[...] instructor, his site has a wealth of well written training material from a QoS Mindmap to using NBAR and QoS to filter websites to setting up backup [...]

Posted by Returning Some Link Love « CCIE Pursuit | February 24, 2008, 11:04 am
Very useful in business environments, Here is another post explaining how to prevent non business related applications during work hours.

http://networkers-online.com/blog/?p=9

Posted by networkersOnline | October 8, 2008, 3:06 am
i thought for all these examples you will be appliying in the output direction since traffic from yuotube is out to the customer, corrent me

rgds

Posted by vinnie | October 20, 2008, 9:21 pm
@Vinnie: Great question! It needs to be applied to the incoming direction because thats where the packets are coming in as far as the router is concerned. Into the router.

We could apply it to the outbound direction, but then the router would waste resources processing a packet, choosing an outbound interface before its finally dropped.

Posted by Arden Packeer, CCIE #20716 | October 21, 2008, 7:28 am
Hey Arden

What layer is ” match protocol http mime video/flv|video/x-flv|video/mp4|video/x-m4v|audio/mp4a-latm
” actually filtering at?

Thanks bro

Posted by Dan | November 14, 2008, 10:23 am
@Dan: We are matching at the application layer.

Posted by Arden Packeer, CCIE #20716 | November 14, 2008, 10:59 am
Hi Arden,

matching on url needs to be applied in outbound direction isn’t it? I mean in this way you can prevent the client from accessing that page.
If you want to prevent him downloading the page you need to match based on mime types, isn’t it?

regards

Roger

Posted by Roger | February 10, 2009, 7:09 am
In the last example, the ‘match protocol http mime’ statements, if matched, won’t be actioned with a ‘drop’.

I tested this extensively in my lab — same basic topology as yours — and found that some (not all) actions associated with a class using ‘match protocol http mime’ only worked when the policy-map was applied in the *server -> client* direction.

Using your example topology, if the policy-map was applied as an input policy on R2-s1/0, it would successfully classify the traffic but the ‘drop’ action wouldn’t take effect.

If applied as an input policy on R2-s1/1, it would work as expected.

If applied as an output policy on R2-s1/0, it would work as expected.

If applied as an output policy on R2-s1/1, it would classify the traffic but the ‘drop’ action wouldn’t take effect.

My tests were conducted using 2811s and IOS 12.4(23).

I’m interested in your thoughts on this — did you actually verify that, for example, R2 drops the HTTP response from R2 when the content type is “application/x-shockwave-flash” ?

I recreated your exact topology, then added ‘match protocol http mime “image/jpeg”‘ to the ‘INTERNET-SCUM’ class-map. I was able to copy “picture.jpg” from R3 to R1 successfully.

Cheers!

Posted by Dale | February 24, 2009, 4:12 pm
this is from the best topics I have ever read, easy well prepared.

thanks Arden

keep it up

Salah

Posted by Salah | March 20, 2009, 7:15 am
Arden,

To use nbar for classification don’t you have to turn on nbar protocol-discovery??

Cheers,

Broeisi

Posted by Broeisi | May 17, 2009, 7:48 pm
@Broeisi: Nbar Protocol-Discovery is not required for classification

Posted by Arden Packeer, CCIE #20716 | May 21, 2009, 8:35 pm
Great one. very useful. I implemented this to cater one of my customer requirement.By the way do you have any idea of blocking P2P application & instant messaging. there are options to do this by using SDM. but i don’t like that as it highly rely on IOS capabilities.

Posted by Rajitha | July 2, 2009, 5:05 pm
great tutorial.This was so benefit for me.

thank you

Posted by elaheh | July 14, 2009, 6:50 pm
Great tutorial very useful, thank you.

Posted by Mesut Cap | July 23, 2009, 9:16 pm
[...] I have read this link from Arden, thanks Arden: http://ardenpackeer.com/qos-voip/tutorial-how-to-use-cisco-mqc-nbar-to-filter-websites-like-youtube/ [...]

Posted by class-map – match protocol http url command | August 15, 2009, 2:13 pm
nice read…is it possible to send me an example to block everything(streaming videos) but allowed youtube and some other video streaming website.

can this be done on nbar/qos.

ciao,
jr

Posted by Jack | January 21, 2010, 4:50 am

Post a comment

Twitter Feed...

When is a fabric not a fabric? http://forums.juniper.net/t5/Architecting-the-Network/When-is-a-Fabric-not-a-Fabric/ba-p/84022
Router candy! http://yfrog.com/h37krqyj
@matthewnorwood thanks man!
@matthewnorwood all I got out of them was "end of the year". The fabric bandwidth can be extended via fabric modules like the current n7k

Follow me...

Most Emailed

Tutorial: OSPF Network Types and Frame Relay Part 1 - 24 emails
Tutorial: How to use Cisco MQC & NBAR to filter websites like Youtube - 22 emails
Tutorial: What is the difference between bandwidth percent and bandwidth remaining percent? - 20 emails
Tutorial: How to set up a basic Dynamips Lab using Dynagen, Part 1 - 11 emails
Tutorial: BGP Route Aggregation Part 1 – Using an Advertise-Map to control aggregation - 10 emails

© 2007 Arden Packeer. All rights reserved. The opinions expressed herein are my own personal opinions and do not represent my employer's view in anyway. This material is not sponsored or endorsed by Cisco Systems, Inc. Cisco, Cisco Systems, CCIE and the CCIE Logo are trademarks of Cisco Systems, Inc. and its affiliates

Computer Blogs - BlogCatalog Blog Directory

Search

Featured

Tutorial: How to use Cisco MQC & NBAR to filter websites like Youtube

Discussion

16 comments for “Tutorial: How to use Cisco MQC & NBAR to filter websites like Youtube”

Post a comment

Categories

Twitter Feed...

Follow me...

Recent Comments

Most Emailed