These are mostly OLD notes from the real-time process of designing and
building the wulfware suite.  They are in no particular order, and one
day I may make them go away as they're in both the old CVS repo(s) and
the new SVN repo.  However, there may be some stuff of interest therein
in the meantime.



wulfstat is a program designed to run on a tty interface and provide
detailed monitoring information on a collection of networked cluster
nodes or workstations.  It gathers the information from xmlsysd, a daemon
that runs on the nodes or workstations, collects information from /proc
files or systems calls, and returns it in an xml-based format via a tcp
socket connection.

5/17/02 -- added support for <hostrange> and <iprange> tags, fixed various
bugs.  Checked in tagged as llist_2.  And maybe as llist_3 -- since I've
also broken out routines for parsing the <host>, <hostrange> and <iprange>
tags AND for adding them (separately) out of linked lists that can be
manipulated for purposes of building a wulfhosts file to be saved.

5/20/02 -- I went ahead and incremented minor revision by .1.  At this
particular instant, the program "works" including all the aforementioned
tags, and I can even verify that I'm reading in the tag contents.
HOWEVER, I haven't yet gotten any code in fill_values to work to parse
out the returns for each monitored PID, and suspect that a merry time
will be had working all this out.  The primary issue here is that we
have to fill the linked list of running PIDS EACH PASS through, either
updating an existing struct or deleting a dead struct and starting a new
one.  I don't know whether this will be efficient enough -- free/malloc
can be fairly costly in time.  We'll see.  With a fairly slowly varying
list of entities, it may be fast enough.

I must confess: I maybe possibly shoulda written the entire program to
get values directly from the xml structure and never moved them into an
intermediate position.  If I had done this, I wouldn't have to worry at
ALL about allocating intermediate storage, only about efficiently
extracting values from the linked list of xml structs (which are
basically linked lists).  The only exceptions probably should have been
things in <stat> and <net>, where there are derived rates.

Is it worth a complete 0.3.0 rewrite to implement this?  We'll see.

Much time passes... up to 10/8/02.

OK, we've learned several useful tricks in the meantime.  One of them is
pthreads, which is a lightweight shared-memory thread library that is
vastly better than fork/exec in any flavor as far as ease of use is
concerned.  The other is how to actually use the windowing features in
ncurses, which we alas failed to do before.

SO we're going to try a short-term rewrite of the tty interface with the
following design goals:

  a) A separate window for the various critical elements of the program,
in particular, the header line(s), the menu lines, the bottom line and
the DEBUG WINDOW! should all be in windows of their own.  So should the
actual output!  We should also use "window borders" (see man box_set)
rather than doing this the hard way to separate windows.
  b) Implement scrolling rather than F/B keys to get up and down in a
host list.  ncurses supports the scrolling of a virtual window -- so
should we.
  c) Simplify the debug window and debugging output.  Basically, as long
as the debug window is scrollable, it can just be a dumb tty with no
line control.  Also, keep in mind resize_term and wresize -- it is
perfectly reasonable to require a MINIMUM of (say) 20 lines for regular
operation, 30 lines for debugging operation, and exactly 80 columns for
the xterm display.
  d) Life would be simpler yet if I think ahead to how wulfstat will
need to morph to become gwulfstat.

# $Id: DESIGN 65 2002-03-13 21:06:59Z rgb $
#
# See copyright in copyright.h and the accompanying file COPYING
#========================================================================

                       XML System Daemon

The XML system daemon is being designed as a replacement for procstatd,
an early way to provide lightweight access to key system statistics for
remote cluster or LAN monitoring.

Basically, the XML system daemon will be designed to provide 

  a) a lightweight wrapper function, based on libxml, that can
encapsulate arbitrary fields from /proc-based files.  The XML will
transparently encode the path and field indices of the data returned. At
this time read-only access will be provided; this is NOT intended to
function as a daemon-based proc control interface.

  b) functions to produce xml-encoded access to selected systems
information calls such as time and users that might be useful to systems
or cluster managers.

  c) a daemon-based command interface that can be used to select
specific path/field or supported systems data call descriptors for for
return in an xml-encoded packet produced by the xml interface.  As the
project develops, commands for selection based on glob or regexp
criteria may be added.  Initially this will be procstatd's old ascii 
command interface only.

This greatly simplifies and hence improves upon the functionality of
procstatd -- basically a small set of routines configures the daemon and
opens the selected proc paths, and then reads, xml encodes, and returns
the requested data in packet form through a standard socket interface.
All presentation and processing is the responsibility of the calling
routine.

This simplification means that trivial modifications of e.g. ps or
uptime or even top can be used to monitor remote hosts as easily as
local ones via a socket interface.  It will also permit the construction
of both web, gui, and even (via a suitable DTD) text applications for
converting return data into "reports" or visual displays of entire
cluster or lan proc/system data.

A swiss-army-knife tty monitoring tool (wulfstat) is provided so that
the daemon can be immediately useful until I finish a Gtk-based GUI app
with more features.

Further design notes and modifications will be documented below as
appropriate.  The principle one of interest are the rules to be followed
by folks seeking to add (e.g. sensors) support to the daemon for their
local hardware.  PLEASE follow these rules, as they are key to modifying
the daemon in a way that will not break applications like wulfstat AND
that can use wulfstat's core parsing routines to fairly easily extract
the stuff that you add.

  rgb  3/13/02

========================================================================
                 <xmlsyd> Language Specification

The language is very simple.  Note the packet must start with a
Content-Length specifier (from http 1.1) followed by an empty blank line
so that the message parser knows how many bytes to read from the stream.

The document root is <xmlsysd> and must
wrap the whole message.

There are two toplevel tags, <system> and <proc>.  <system> wraps fields
derived from systems calls, e.g. gettimeofday(), hostname() or non-proc
files like wtmp (users).  <proc> wraps all fields encapsulating data
parsed from files under /proc.

In general I have tried to create a tag structure that -- up to a point
-- recapitulates an xpath-like map:

  /proc/stat <-> <proc><stat>
  /proc/net/dev <-> <proc><net><dev>

This breaks down, of course, because /proc looks like it was designed by
a myriad of scraggly monkeys loaded on Jolt Cola and banging randomly on
a keyboard... no, that is too kind.  Some parts of it look like they
were designed by >>deranged<< scraggly monkeys.  Seriously, /proc is
crying out for a fundamental makeover into something that CAN be neatly
echod in xml.

In any event, one can see that within e.g. /proc/stat I TRY to assign a
tag per line parsed, and to tag the fields parsed out of the line with
sensible (and accurate) names.  Of course its format is completely
different from /proc/meminfo, which is TOTALLY different from
/proc/net/dev, and don't get me started on sensors (the /proc API
designed by SADISTIC deranged monkeys who may or may not be scraggly).

The fundamental design principles are:

  a) TRY to create a rational representation of /proc, feeling free to
ignore irrelevant (to monitoring, in your opinion) fields or to impose
order on chaos as needed.
  b) VERY IMPORTANT!  Every Tag/XPath Must Be Unique!  For example, the

<proc>
 <stat>
   <cpu>
    ...(content tags)
   </cpu>

can occur several times, for cpu0, cpu1.. on an SMP system.  We
hence differentiate the cpus (AND get unique xpaths) by adding an
"id" attribute:
<proc>
 <stat>
   <cpu id="0">
     <user>234235</user>
    ...(content tags)
   </cpu>
   <cpu id="1">
    ...(content tags)
   </cpu>

which can be accessed uniquely via xpaths like

  /xmlsysd/proc/stat/cpu[@id="0"]/user

Similarly there are many network devices (with different names and
types!) that occur inside /proc/net/dev, so we use a toplevel
  ...
   <interface id="0" devtype="eth" devid="0">

to tag eth0, for example.  That way

  /xmlsysd/proc/net/dev/interface[@devtype="eth" @devid="0"/receive/bytes

makes sense.  If we differentiated via only e.g.

  <interface>
    <name>eth0</name>
    <receive>
      <bytes>12091</bytes>

we'd have to "look ahead" to figure out which of the <interfaces> to
descend into to retrieve rx bytes.  Trust me, this is a real pain and
we don't want to do it on the client application side.

There are more sensible rules for xml design than I can capture right
here and now, so you should likely read a book or two (or ASK me) before
trying anything major that you then have to redo.  For example, I
learned the hard way not to create tags like <eth0> or tags with values
given in attributes so you shouldn't have to.

A good rule of thumb is: only create new tags that wulfstats "xtract()"
or "xtract_attribute()" functions can access (if necessary, adding a new
enumerated type to the return list) and MODIFY THIS FUNCTION as needed
at the SAME time.  The latter is often a very instructive step...

  c) Some quantities that we might want to present are rates.  In order
create a rate (on the client side) we need to know EXACTLY when the
data was read from proc.  Many file tags therefore carry a timestamp
attribute set right before opening the proc file for reading, e.g.

  <stat tv_sec="1016048061" tv_usec="789464">

so that two successive reads can form the time difference (accurate to
a few microseconds), the value difference, and form a rate.  The rate
formed won't be horribly accurate if the time delta is less than 1 sec
(many things monitored are very bursty and short time rates are not good
predictors of average rates) but we need the timestamp(s).  Besides,
the top to bottom timestamp delta gives you a very good idea of how long
the message took to assemble on the monitored host and hence the relative
load produced by running the daemon.

  d) Don't Break Things.  There are two things you need to be VERY
careful not to break.  One is existing tags and paths.  PLEASE submit
major changes in format back to me so I can incorporate them, explaining
why the changes are necessary, but remember that if the changes are
really big (change the paths completely) all the clients will break.
There are tags I >>plan<< to add (e.g. <disk> from /proc/stat) but
adding it shouldn't break <cpu> or <intr>.  The other is memory
management.  libxml is very tricky in that you have to free all sorts of
things used to construct or parse the xml document using ITS constructs,
while still remembering to free your own.  Even being careful, my first
cut of this daemon leaked memory like a sonuvabitch until I squashed a
nasty little bug.  It is a Bad Thing to run a leaky daemon intended to
run for months at a time and hundreds of thousands of message cycles...

Below you can see a typical return message (without compression and with
"pretty" indentation)  from the daemon as of this moment.  Note that
information redundancy is OK, and even tagname reuse is OK as long as
paths remain unique (including attributes).

Good Luck,

  rgb   (rgb@phy.duke.edu)



%< Snip Snip ============= Sample return xmlsysd message =============
Content-Length: 4028

<?xml version="1.0"?>
<xmlsysd>
  <system>
    <time tv_sec="1016048061" tv_usec="788933">2:34:21 pm</time>
    <identity>
      <hostname>golem</hostname>
      <hostip>192.168.1.140</hostip>
    </identity>
    <users tv_sec="1016048061" tv_usec="789442">3</users>
  </system>
  <proc>
    <stat tv_sec="1016048061" tv_usec="789464">
      <cpu id="0">
        <user>316976</user>
        <nice>773</nice>
        <sys>92659</sys>
        <tot>49861016</tot>
      </cpu>
      <page>
        <in>317279</in>
        <out>1860644</out>
      </page>
      <swap>
        <in>1</in>
        <out>0</out>
      </swap>
      <intr>54537416</intr>
      <ctxt>8610762</ctxt>
      <processes>52819</processes>
    </stat>
    <meminfo tv_sec="1016048061" tv_usec="790457">
      <memory>
        <total>327102464</total>
        <used>263335936</used>
        <free>63766528</free>
        <shared>393216</shared>
        <buffers>109690880</buffers>
        <cached>108212224</cached>
      </memory>
      <swap>
        <total>1077469184</total>
        <used>102400</used>
        <free>1077366784</free>
      </swap>
    </meminfo>
    <net>
      <dev tv_sec="1016048061" tv_usec="798104">
        <interface id="0" devtype="lo" devid="">
          <name>lo</name>
          <ip>127.0.0.1</ip>
          <host>localhost</host>
          <receive>
            <bytes>97083646</bytes>
            <packets>141291</packets>
            <errs>0</errs>
            <drop>0</drop>
            <fifo>0</fifo>
            <frame>0</frame>
            <compressed>0</compressed>
            <multicast>0</multicast>
          </receive>
          <transmit>
            <bytes>97083646</bytes>
            <packets>141291</packets>
            <errs>0</errs>
            <drop>0</drop>
            <fifo>0</fifo>
            <collisions>0</collisions>
            <carrier>0</carrier>
            <compressed>0</compressed>
          </transmit>
        </interface>
        <interface id="1" devtype="eth" devid="0">
          <name>eth0</name>
          <ip>192.168.1.140</ip>
          <host>golem.rgb.private.net</host>
          <receive>
            <bytes>762887796</bytes>
            <packets>2036397</packets>
            <errs>0</errs>
            <drop>0</drop>
            <fifo>0</fifo>
            <frame>0</frame>
            <compressed>0</compressed>
            <multicast>0</multicast>
          </receive>
          <transmit>
            <bytes>326138029</bytes>
            <packets>1860317</packets>
            <errs>0</errs>
            <drop>0</drop>
            <fifo>0</fifo>
            <collisions>0</collisions>
            <carrier>0</carrier>
            <compressed>0</compressed>
          </transmit>
        </interface>
      </dev>
      <sockstat tv_sec="1016048061" tv_usec="799216">
        <used>45</used>
        <tcp>
          <inuse>13</inuse>
          <orphan>0</orphan>
          <timewait>0</timewait>
          <alloc>13</alloc>
          <mem>1</mem>
        </tcp>
        <udp>
          <inuse>6</inuse>
        </udp>
        <raw>
          <inuse>0</inuse>
        </raw>
        <frag>
          <inuse>0</inuse>
          <memory>0</memory>
        </frag>
      </sockstat>
    </net>
    <loadavg tv_sec="1016048061" tv_usec="799388">
      <load1>0.21</load1>
      <load5>0.05</load5>
      <load15>0.02</load15>
    </loadavg>
    <cpuinfo tv_sec="1016048061" tv_usec="799554">
      <processor id="0">
        <vendor_id>GenuineIntel</vendor_id>
        <family>6</family>
        <model_num>8</model_num>
        <model_name>Celeron (Coppermine)</model_name>
        <clock units="MHz">801.824</clock>
        <cachesize units="KB">128</cachesize>
      </processor>
    </cpuinfo>
    <sysvipc tv_sec="1016048061" tv_usec="799947">
      <msgbufs>0</msgbufs>
      <msgtot>0</msgtot>
      <sembufs>0</sembufs>
      <semtot>0</semtot>
      <shmbufs>2</shmbufs>
      <shmtot>786432</shmtot>
    </sysvipc>
    <version>2.4.9-21</version>
  </proc>
</xmlsysd>
quit
rgb@golem|T:103>exit
Script done on Wed Mar 13 14:34:25 2002

A particular need that has surfaced is to add a pci query -- basically
to run lspci in a shell and build a static table when xmlsysd first
comes up -- so that the PCI hardware available on a system (at least the
networking hardware, but possibly more) can be remotely queried.

This should be easy enough to do one of two ways -- actually run lspci
(this puts the burden of maintaining all tables on the pciutils
maintainers) or to steal lspci's internals.  If this were a command that
was run per query, the latter would clearly be the correct course of
action, but for a one-off command with a complex and dynamic table
backing it, it seems for once smarter to shell it out and use lspci.

lspci even seems to directly support this usage, see "lspci -m" for a
parseable list of strings per device line:

rgb@lilith|B:1069>lspci -m
00:00.0 "Host bridge" "Intel Corp." "440BX/ZX/DX - 82443BX/ZX/DX Host
bridge" -r03 "" ""
00:01.0 "PCI bridge" "Intel Corp." "440BX/ZX/DX - 82443BX/ZX/DX AGP
bridge" -r03 "" ""
00:03.0 "CardBus bridge" "Texas Instruments" "PCI1420" "Dell Computer
Corporation" "00b0"
00:03.1 "CardBus bridge" "Texas Instruments" "PCI1420" "Dell Computer
Corporation" "00b0"
00:07.0 "Bridge" "Intel Corp." "82371AB/EB/MB PIIX4 ISA" -r02 "" ""
00:07.1 "IDE interface" "Intel Corp." "82371AB/EB/MB PIIX4 IDE" -r01
-p80 "" ""
00:07.2 "USB Controller" "Intel Corp." "82371AB/EB/MB PIIX4 USB" -r01 ""
""
00:07.3 "Bridge" "Intel Corp." "82371AB/EB/MB PIIX4 ACPI" -r03 "" ""
00:08.0 "Multimedia audio controller" "ESS Technology" "ES1983S
Maestro-3i PCI Audio Accelerator" -r10 "Dell Computer Corporation"
"00b0"
00:10.0 "PCI bridge" "Actiontec Electronics Inc" "Mini-PCI bridge" -r11
"" ""
01:00.0 "VGA compatible controller" "ATI Technologies Inc" "Rage
Mobility M3 AGP 2x" -r02 "Dell Computer Corporation" "00b0"
08:04.0 "Ethernet controller" "Intel Corp." "82557/8/9 [Ethernet Pro
100]" -r08 "Actiontec Electronics Inc" "EtherExpress PRO/100B (TX)
(MiniPCI Ethernet+Modem)"
08:08.0 "Communication controller" "Lucent Microelectronics" "WinModem
56k" -r01 "Actiontec Electronics Inc" "LT WinModem 56k (MiniPCI
Ethernet+Modem)"

Looks like we could get by with fields 2 (e.g. "Ethernet controller"), 3
"Intel Corp" and 4 "82557/8/9 [Ethernet Pro 100].  It would also be
lovely to establish a map between pci device and network device, but it
looks like that won't happen unless somebody gives me a clew.

In addition, at the moment, I need to decruft things, debug things, and
gradually add features.  The usual.

Help is always welcome.

One GREAT way to help is by adding get_proc_sensors.c components for any
sensors you actually have handy to test with.  Don't forget to
contribute them along with a snippet of their returned xml so I can work
them into wulfstat as well as the daemon.


                            wulflogger

wulflogger is a command line variant of the wulfstat program.  It is
written to use exactly the same wulfhosts file (and indeed, most of the
same libwulf library used to parse wulhosts, establish and maintain
connections to hosts, initialize and update host value structures) as
wulfstat.  It polls all the connected hosts after a user-adjustable
delay and then spits a table of values associated with the monitored
quantity to stdout, from which it can be e.g. piped into a file for long
term logging, piped into a graphics utility for graphing or turning into
a web display, or parsed from inside e.g. a perl script with a simple
split command into an array or hash of per-host values and their
associated host-local timestamp.  It was written by request to
specifically support tools like rrd (round robin database) but I expect
it to find usage in many other venues.  The fully GPL code can also
easily be modified so that it logs quantities selected by the user if
you don't like my prebuilt displays by using this program as a template.

Eventually it might even be worthwhile to create a variant that permits
an output format to be e.g. read in from a file and a "universal"
version built where all displays are custom displays and can be
customized without modifying the code, but this won't happen until
somebody really needs it as it is a bit of a chore.  Volunteers
(especially volunteers who are good programmers:-) are always welcome to
contribute this as a major modification if you really need it faster
than I can free time to do it.

To build wulflogger you MUST have libwulf, the xmlsysd/wulfhosts API
library.  If you obtained this from a tarball, chances are good that it
has a working version of libwulf (which is really a separate package)
already present so that it is easy to build.  The same is true for the
source RPM distribution tarball, of course.  The library does NOT at
this time have to be installed separately on the system unless you wish
to do development.  wulfstat, wulflogger, and libwulf can all be
obtained from:

  http://www.phy.duke.edu/~rgb/Beowulf/beowulf.php

or

  http://www.phy.duke.edu/brahma/Resources/resources.php

To build from the tarball, unpack the sources and enter "make".  Chances
are pretty good it will just build.  If it fails, you probably need to
install libxml2 or some other library package -- look at the error
messages.  If you want to install it somewhere from the tarball, you'll
want to look at and probably modify macros in the Makefile so that it
goes where you want it to go, then enter (possibly as root) "make
install".  If you hack on it much, especially if you hack on the libwulf
library, you'll likely want to enter "make clean;make" to ensure that
the library rebuilds.

If you prefer a packaged solution (and you should) there are general
instructions in the Makefile itself for building an RPM using the
"make rpm" make target.  There is also a nifty target for installing the
tgz and rpm's and some documentation on a website.  I wouldn't advise
hacking this stuff too much unless you know what you are doing.  Or you
can (always) rebuild the source rpm, which is by far the preferred way
to proceed on an rpm-based system as it ensures that the binary and
library are linked to current system libraries.

wulflogger is in beta, but it is in fairly advanced beta -- it works
pretty well, people are using it, and nearly all its features work as
well as they are supposed to work.  As far as I can tell it neither
leaks memory nor crashes egregiously.  Its documentation is still a bit
crude, but that will improve over time, as will some examples of its
application in e.g. perl scripts or rrd scripts.

Please report any bugs, feature requests, or lavish praise to
rgb@phy.duke.edu, flames to /dev/null (where I don't consider
constructive criticisms or bug reports flames).  If you DO develop any
nifty scripting applications that use wulflogger, consider contributing
them back to me for inclusion in the package so others can use them as
well.

One caveat -- at this time libwulf.a is still full GPL, so you cannot
include it/link it to commercial closed source applications -- the
inheritance clause applies.  If this bothers you, feel free to talk to
me and we can at least discuss a LGPL.

This is a very short project description and plan for webwulf (now
wulf2html)

  a) Script(s) that build html table(s) using wulflogger
   i) arguments must mirror wulflogger to a point
  ii) adjustable update delay
 iii) selectable output file/location for html generated
  iv) loops (forever) updating html table with some time delay, runs
      in background on a single (selected) system.
   v) arguments all read in from configuration file, not command line,
      so that it can be installed as a "configurable application"

  b) website "kit" in php that can be installed in a suitable location
with links for various views and/or clusters.  The primary pages should
include meta tag for automatically recycling the table with selected
time delay (which may be slower than the wulflogger update and should
probably never be any less).

  c) /etc/init.d support for starting canned webwulf script a) from
specified configuration(s).

These are LONG run plans.  In the SHORT run, I just want to build a
moderately nifty script application a la a) and maybe a simple wrapper
php page a la b) to use right now.

Note that you should be able to find a simple header example in the
documents directory, usually /usr/share/doc/wulfweb-something.


rgb  (obfuscated)
at  (obfuscated)
phy  (obfuscated)
duke  (obfuscated)
edu (obfuscated)
