Global Sources
EE Times-India
Stay in touch with EE Times India
 
EE Times-India > Networks
 
 
Networks  

Optimize Linux to meet carrier-grade requirements

Posted: 01 Oct 2003     Print Version  Bookmark and Share

Keywords:Embedded 

PDF document

By John Mehaffey

High-availability Architect

MontaVista Software Inc.

E-mail: mehaf@mvista.com

In control theory, it is a basic

tenet that you cannot control

what you do not measure. For

carrier-class systems with ex-

pectations of continuous avail-

ability,thismeansthatyoumust

monitor the resources used by

yourapplicationstoensurethat

theyareavailablewhenneeded,

and that the system does not

become overutilized.

Carrier Grade Linux, de-

fined here as a Linux distribu-

tionthatmeetsthepriority-one

attributesrequiredbytheOpen

Source Development Lab

(OSDL) Carrier Grade Linux

Requirements Definition ver-

sion 1.0, requires that a re-

source-monitoring subsystem

be provided that minimally

monitors disk usage, network

usage, CPU and memory utili-

zation as well as process status.

The resource-monitoring

subsystem provided by Carrier

Grade Edition provides two

monitoring packages with dif-

ferent capabilities. The pack-

agesarecalledremotemonitor-

ing (RMON) and resource

monitor. Each will be discussed

with regard to its strengths and

weaknesses, and examples of

howtouseRMONareprovided.

RMONprovidesthecapabil-

itywithinthekerneltomonitor

resources in real-time. For ex-

ample, RMON could be used to

monitor a resource like the

number of forks that have oc-

curred since system reboot. To

monitorresources,RMONuses

counters,gauges,statisticsand

thresholds. A counter com-

puteseventsthataremonotoni-

cally increasing, like the num-

ber of received Ethernet pack-

ets. A gauge is a container for a

value that can go up and down,

like the current number of

tasks. A statistic is a value that

is calculated from a gauge, for

example, packets per second. A

threshold is the value of a sta-

tistic or gauge at which action

is taken. For example, when a

statistic reaches a threshold

value, an RMON device will re-

turn some data from a read()

call on the device.

The following gauges and

counters are provided as part of

Carrier Grade Edition.

Ethernet, on a per-interface

basis or an aggregate basis:

7 Received packets;

7 Received bytes;

7 Receive errors;

7 Transmitted packets;

7 Transmitted bytes;

7 Transmit errors.

System resources:

7 Current free memory pages;

7 Currentnumberofprocesses;

7 Current number of zombies;

7 Numberofprocessescreated;

7 Current system load.

You may also create your

own resources from user space

that may be monitored by other

user-spaceorkernelcode.Afull

API is available inside the ker-

nel that can do anything the

user-space API can do, so

threads inside the kernel may

monitor any resources.

Using RMON

To use RMON, you need to

reconfigurethekerneltoenable

the resource monitors. To en-

able RMON, you need to enable

support in the kernel and re-

build your kernel. The RMON

support menu pick is in the ker-

nel-debuggingandrecoverysec-

tion of the OS.

Once RMON has been en-

abledinthekernel,youmayadd

resource-monitoring functions

to your driver source code. A

completedescriptionoftheAPI

for RMON is in the linux/in-

clude/linux/rmon.h include

file.

RMON specifics

The RMON API defines two

types of basic monitors: counts

and gauges.

Counts are monotonically

increasing values, such as the

number of bytes written to a

disk or the number of Ethernet

packets received. A count has

an active and a snapshot value,

the active value the one cur-

rently being counted and the

snapshot value being a previ-

ous value of the counter that

was saved. The active value

may be copied to the snapshot

value upon command and op-

tionally set to zero in one

Optimize Linux to meet carrier-grade requirements

atomic operation (so that no

counts will be lost).

Gauges are values that go up

and down, such as the number

ofprocessesinthesystemorthe

amount of memory currently

being used by the system.

Gauges may have set maximum

and minimum values (although

they may not be meaningful for

some gauges). Like counts,

they have an active and a snap-

shot value; thus snapshots may

be taken on command, but un-

like counts, the active value

cannot be set to zero.

Statistics are monitors that

may be attached to counters

and gauges. They allow for

some preprocessing of the raw

gauges and counters.

Counters have the following

statistics:

7 Rate: This is a running aver-

age of the number of incre-

ments per period of time for

a value. The value reported is

theaveragevalueoverthepe-

riod of time given, averaged

over the period of time mul-

tiplied by the number of

samples. For instance, if you

settheperiodto1,000msand

the number of samples to 10,

this will report the average

value per second over the last

10s.

7 Leaky bucket: This decre-

ments the value of the count

by a set amount periodically.

This allows you to more easily

highlight counters that may

have spiky usage patterns.

Gauges have the following

statistics:

7 High-water mark: The maxi-

mumvalueofthegaugesince

the last snapshot.

7 Low-water mark: The mini-

mumvalueofthegaugesince

the last snapshot.

A statistic looks just like a

gauge or a counter from a read-

ing and snapshot point of view.

A monitor may have multiple

instances, which allows a

single monitor to have mul-

tiple counters in it. For in-

stance, if an "Ethernet bytes

received" monitor is created, it

may have an instance for each

Ethernet card in the system.

There is only one monitor, but

it has multiple instances that

are each queried and counted

independently.

Thresholds can track moni-

tors and signal when they go

beyond certain set values. The

threshold is activated when the

value goes beyond the set range

(called the "on" value). An-

other set of ranges is given

(called the "off" values), and

the threshold must go inside

the off value for the threshold

tobedeactivated.Thisprovides

hysteresis to avoid reporting a

high number of threshold

events when the monitor value

is close to the threshold and

goes up and down.

Implementing a monitor

Toimplementamonitorinyour

driver,youneedonlymakeafew

simple calls to RMON routines.

The code fragments to imple-

ment a counter monitor for

FDDI packets received are

shown in Figure 1.

You must create your basic

Figure 1: The code fragments implement a counter monitor for FDDI packets received.

monitor and an instance of that

monitor. The instance identi-

fiermustbeauniqueintegerfor

each instance. The code as-

sumes there is only one FDDI

card in the system. In general,

you would need to implement

an instance per card instead of

an instance per driver.

The program in Figure 2

reads the monitor routine

implemented above to check

that the packet rate does not

exceed a predefined threshold.

Resource Monitor

Resource Monitor provides an

alternative monitoring sub-

system to RMON that offers

consistent programmatic ac-

cess to statistics, thresholds

and watermarks for any instru-

mented subsystem in the Linux

OS. The architecture of Re-

source Monitor defines four

primary components:

Consumers: Linux applications

that use the programmatic

access.

Resource Monitor Library: The

interfaces used by consumers

for monitoring access.

Resource Monitor daemon

(resourcemonitord): The run-

time process that manages con-

sumer-monitoringrequestsand

provides continuous monitor-

ing of statistics.

Subsystem Monitor libraries:

The subsystem-specific access

methods used by the daemon to

query statistical values.

Resource Monitor is de-

pendent on the Linux event-

logging system, which provides

clientapplicationswiththeabil-

ity to register for system events.

Itgenerateseventrecordswhen-

ever statistic monitors trigger

on their rule evaluations, and it

is designed to use the interfaces

exposed by the event-logging

system. Resource Monitor also

provides a means of querying

thestatisticsavailableinthesys-

tem (called discovery).

Resource Monitor does have

limitations. First, Resource-

Continued

/* See if the monitor already exists, and delete it if so. */

err = rmon_find_monitor(rmon_fd, "fddi_rx_packetrate", &target_mon);

if (err == 0) {

/* The monitor already exists, destroy it so we can create a clean

version.. */

err = rmon_destroy_monitor(rmon_fd, target_mon.basic.id);

if (err == -1) {

perror("rmon_destroy_monitor");

exit(1);

}

}

/* Find the monitor we will be watching, the received fddi

packet rate. */

err = rmon_find_monitor(rmon_fd, "fddi_rx_packets", &target_mon);

if (err == -1) {

perror("rmon_find_monitor");

exit(1);

}

/* Now create the rate monitor. */

err = rmon_create_rate_monitor(rmon_fd,

"fddi_rx_packetrate",

target_mon.basic.id,

1000, /* 1 second */

60, /* Averaged over 60 seconds. */

&packetrate_mon);

if (err == -1) {

perror("rmon_create_rate_monitor");

exit(1);

}

/* Create a threshold to watch the rate monitor. The thresholds

are destroyed when the monitoring program closes the fd, so we

always need to create it. */

err = rmon_create_threshold(rmon_fd,

packetrate_mon,

-1, /* Monitor all interfaces. */

-1, -1, /* We ignore the "off" values. */

n-9, n, /* On at n, off at n-9 */

0, /* cb_data is not used. */

&packetrate_thresh);

/* Now wait for data from the monitor (threshold exceeds). This

call blocks on a read, then calls the given handler when data

comes in, then returns. */

for (;;) {

handle_threshold_read(rmon_fd, &handler, NULL);

}

}

monitord polls the resources.

Asaresult,ResourceMonitoris

not "real-time." Second, the

EXT2 file system-specific li-

braries and headers package is

required for universally unique

identifier support.

Using Resource Monitor

The Resource Monitor, when

initiallyinstalled,hasnodefault

statistics to monitor, so you

need to add these to your driver

source code or user-application

code. A complete description--

with required files--of the API

for Resource Monitor is in-

cluded with the Resource

monitordpackage.Youwillalso

need to configure and run

Resourcemonitord to use Re-

source Monitor. This provides

theRMONcapabilityforsystem

statistics made available

through subsystem libraries.

Resource Monitor's use of Unix

domain sockets limits monitor-

ing to local resources (i.e., re-

sources on the same node).

[Communication Systems Design]

Figure 2: This code sequence reads the monitor to check that the packet rate does not exceed a defined threshold.





Comment on "Optimize Linux to meet carrier-grade..."
Comments:  
*  You can enter [0] more charecters.
*Verify code:
 
 
Webinars

Seminars

Visit Asia Webinars to learn about the latest in technology and get practical design tips.

 

Go to top             Connect on Facebook      Follow us on Twitter      Follow us on Orkut

 
Back to Top