SNMP – Simple Network Management Protocol is a popular protocol for network management, used for collecting information from and configuring network devices such as servers, printers, hubs, switches and routers on an Internet Protocol (IP) network.

With an SNMP management station, you can graph the performance of network devices. In addition, Cisco devices can send alerts (called traps) to the management station, which you can configure to send notifications or alerts.

How would SNMP beneficial to a Network Administrator?

There are so many things SNMP can help an Administrator, Here are some ways that helped me monitor our customers at Intelletrace

1. Graph Cisco router/switch to monitor bandwidth utilization over time, per interface, per direction, etc.
2. Graph Errors on Network devices
3. Monitor Connectivity and Uptime Status
4. Send Alerts when an Interface is Down or Up.

How to configure SNMP monitoring on Cisco Devices?

First, we need to define the community string to be used .

Router# configure terminal

Router(config)# snmp-server community XXXXXXXXX RO

where XXXXXXXXX is your own community string and RO means Read-Only, you may use RW if you need write access on your device.

Router(config)# snmp-server contact John Doe – Network Admin – 212 8644

Router(config)# snmp-server location Novato , California, USA

Router(config)# snmp-server chassis-id Cisco2610-Router

Traditional IP networks are connectionless: when a packet is received, the router determines the next hop using the destination IP address on the packet alongside information from its own forwarding table. The router’s forwarding tables contain information on the network topology. They use an IP routing protocol, such as OSPF, IS-IS, BGP, RIP or static configuration, to keep their information synchronized with changes in the network.

I am not able to diagnose the issue myself so i called in one of my mentors/guru/Boss Dean Zerbe to help me isolate the problem. If you want to know more of MPLS, check out his blog.

Dean gave me some advice on how to troubleshoot things…

Customer Has 4 routers running on an MPLS circuit, First on the checklist was to verify the Layer 1 and Layer 2 connectivity by looking at the interfaces and verifying if there are any errors on the circuit.

1. Show Interfaces – The purpose of the show interfaces command is rather self-explanatory, it displays the interfaces and their status. Just a priliminary on the troubleshooting checking for errors, current traffic and the status of the interface.
2. Show Process commands – displays the information about the active process in a cisco router.

router#show processes

CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0%
 PID Q  Ty       PC  Runtime(uS)    Invoked   uSecs    Stacks TTY Process
 1 C  sp 602F3AF0            0       1627       0 2600/3000   0 Load Meter
 2 L  we 60C5BE00            4        136      29 5572/6000   0 CEF Scanner
 3 L  st 602D90F8         1676        837    2002 5740/6000   0 Check heaps
 4 C  we 602D08F8            0          1       0 5568/6000   0 Chunk Manager
 5 C  we 602DF0E8            0          1       0 5592/6000   0 Pool Manager
 6 M  st 60251E38            0          2       0 5560/6000   0 Timers
 7 M  we 600D4940            0          2       0 5568/6000   0 Serial Backgroun
 8 M  we 6034B718            0          1       0 2584/3000   0 OIR Handler
 9 M  we 603FA3C8            0          1       0 5612/6000   0 IPC Zone Manager
 10 M  we 603FA1A0            0       8124       0 5488/6000   0 IPC Periodic Tim
 11 M  we 603FA220            0          9       0 4884/6000   0 IPC Seat Manager
 12 L  we 60406818          124       2003      61 5300/6000   0 ARP Input
 13 M  we 60581638            0          1       0 5760/6000   0 HC Counter Timer
 14 M  we 605E3D00            0          2       0 5564/6000   0 DDR Timers
 15 M  we 605FC6B8            0          2       011568/12000  0 Dialer event

• show process cpu -displays information about the active processes in the router and their corresponding CPU utilization statistics.

router#show processes cpu
 CPU utilization for five seconds: 8%/4%; one minute: 6%; five minutes: 5%
 PID Runtime(uS)   Invoked  uSecs    5Sec   1Min   5Min TTY Process
 1         384     32789     11   0.00%  0.00%  0.00%   0 Load Meter
 2        2752      1179   2334   0.73%  1.06%  0.29%   0 Exec
 3      318592      5273  60419   0.00%  0.15%  0.17%   0 Check heaps
 4           4         1   4000   0.00%  0.00%  0.00%   0 Pool Manager
 5        6472      6568    985   0.00%  0.00%  0.00%   0 ARP Input
 6       10892      9461   1151   0.00%  0.00%  0.00%   0 IP Input
 7       67388     53244   1265   0.16%  0.04%  0.02%   0 CDP Protocol
 8      145520    166455    874   0.40%  0.29%  0.29%   0 IP Background
 9        3356      1568   2140   0.08%  0.00%  0.00%   0 BOOTP Server
 10          32      5469      5   0.00%  0.00%  0.00%   0 Net Background
 11       42256    163623    258   0.16%  0.02%  0.00%   0 Per-Second Jobs
 12      189936    163623   1160   0.00%  0.04%  0.05%   0 Net Periodic
 13        3248      6351    511   0.00%  0.00%  0.00%   0 Net Input
 14         168     32790      5   0.00%  0.00%  0.00%   0 Compute load avgs
 15      152408      2731  55806   0.98%  0.12%  0.07%   0 Per-minute Jobs

CPU utilization for five seconds – CPU utilization for the last five seconds. The first number indicates the total, the second number indicates the percent of CPU time spent at the interrupt level.

PID – The Process ID

Runtime (uS) – CPU time the process has used, expressed in microseconds.

Invoked – The number of times the process has been invoked.

uSecs – Microseconds of CPU time for each process invocation.

5Sec – CPU utilization by task in the last five seconds.

1Min – CPU utilization by task in the last minute.

5Min – CPU utilization by task in the last five minutes.

TTY – Terminal that controls the process.

Process – Name of Process.

What is important at this point is to verify the CPU utilization for five seconds, if the value has a significant amount of increase or is having a 100% utilization then something is really wrong.

• show process cpu history – displays in ASCII graphical form the total CPU usage on the router over a period of time: one minute, one hour, and 72 hours, displayed in increments of one second, one minute, and one hour, respectively. Maximum usage is measured and recorded every second; average usage is calculated on periods over one second.

The following is a sample output of the one-hour portion of the output:

router#show processes cpu history

!--- One minute output omitted
 

     6665776865756676676666667667677676766666766767767666566667
     6378016198993513709771991443732358689932740858269643922613
100
 90
 80         *  *                     * *     *  * *  *
 70  * * ***** *  ** ***** ***  **** ******  *  *******     * *
 60  #***##*##*#***#####*#*###*****#*###*#*#*##*#*##*#*##*****#
 50  ##########################################################
 40  ##########################################################
 30  ##########################################################
 20  ##########################################################
 10  ##########################################################
    0....5....1....1....2....2....3....3....4....4....5....5....
              0    5    0    5    0    5    0    5    0    5

               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%


!--- 72-hour output omitted

• The Y-axis of the graph is the CPU utilization.
• The X-axis of the graph is the increment within the period displayed in the graph; in this instance, it is the individual minutes during the previous hour. The most recent measurement is on the left end of the X-axis.
• The top two rows, read vertically, display the highest percentage of CPU utilization recorded during the increment.

In the above example, the CPU utilization for the last minute recorded is 66 percent. The router may have reached 66 percent only once during that minute, or it may have reached 66 percent multiple times; the router records only the peak reached during the increment and the average over the course of that increment.

For the issue above, Dean checked out all four existing routers and verified all their CPU utilization, it turned out to be that one of the router is having a high utilization. The High utilization of the router is correlated to an over utilization of the T1.  He then checked of the NAT translation and found out one of the servers is sending lots of traffic to the internet causing multiple NAT translation thus causing the utilization of the router to increase and pointing the issue of the T1 being over utilized and inside network being slow. We had Isolated the said server and blocked its access on the internet, removing the one-to-one NAT translation and denying the ports 80 and 21 for the said server.

It should be noted that high CPU utilization, by itself, does not indicate a problem with your device.

As a rough guideline, only consistently high CPU utilization over an extended period of time indicates a problem. Further, these commands are more relevant in the process of figuring out what went wrong rather than being indicators that all is not fine.

Need some advice on your network? You may check out some technical stuff at Techblog, Intelletrace Site or Tweet askintelletrace.