documentation/specification/expected_data.txt

XML via UDP - expected data from hosts
======================================


<<< Provisional, pjm2@kent.ac.uk >>>


Purpose of this document
------------------------

This document shall help to provide a separate party with
the knowledge required to use their own implementation of
a piece of host monitoring software.  In particular, this
document details the data that is to be expected from a
host.  This document is additional to
http://www.i-scream.org/cgi-bin/docs.cgi?doc=specification/xml_via_udp.txt


Background
----------

Hosts are expected to periodically send UDP packets to the
central monitoring system.  Such packets may contain various
pieces of information about the host, such as how much free
memory is remaining on the host, etc.  Some of this data is
deemed to be 'essential', some is 'expected' and the rest
is not necessary, but shall not be ignored by the central
monitoring system server.


Specification
-------------

The UDP packet must contain a complete and well-formed
piece of XML markup, describing the data that the host is
submitting.  In the event of a piece of data containing
an invalid character, such as a "<", it is the responsibilty
of the host to format this in a valid manner, for example
by replacing all occurrences of "<" with "&lt" before it is
sent in a UDP packet to the central monitoring system server.

Alternatively, the data may be wrapped up in special CDATA
tags.  For example, if we wish to send the following text:

    <>&!"

surrounded by <test_chars> tags then we can safely send this
by wrapping it up to produce the following XML snippet:

    <test_chars><![CDATA[<>&!"]]></test_chars>

The bare minimum that a host should send is the following: -

    <packet>
    </packet>

Note that every XML tag must also have a matching closing
tag.

However, such a packet is rather useless when it is
received by the server - it does not provide any useful
information.  As such, a packet will be rejected by the
server if it does not specify the machine_name, its
I.P. address (ip), a packet sequence number (seq_no)
and a date.  Note that the date is a long integer
representing the number of seconds since the epoch.
All of these are expected to be specified as an attribute
of the packet tag.  If they are not specified here, then
the packet will be rejected.

Thus, to ensure that a packet has a chance of being
acted upon, a host must specify at least four attribute
values - machine_name, ip, date and seq_no.  These must
be formed in the style shown below.

    <packet machine_name="raptor" ip="aaa.bbb.ccc.ddd"
            date="93456245245" seq_no="1234567">
    </packet>

Note that the server may run additional 'plugin' tools
that are designed to filter/reject packets under certain
conditions.  Please check with the server administrator
if you wish to know about the specific configuration of
your central monitoring system.

The I.P. address and machine_name could be found from
the UDP packet itself, but these remain essential so
as to provide resiliance from stray packets that do not
contain any valid or useful data.

Remember that malformed XML data would be rejected by the
central monitoring system immediately, without acting upon
it.


Classification of XML data in our system
----------------------------------------


ESSENTIAL: -

The following values are ESSENTIAL in each packet.  Any
packet without these will be rejected.  They MUST be
specified as attributes of the root node ("packet"): -

    ip
    seq_no
    date
    machine_name


EXPECTED: -

The following values are EXPECTED in each packet.  All
packets are RECOMMENDED to specify these values, as the
server is designed to permanently store such data in an
easy and quick to access manner.  It is not essential to
specify such values, but it does limit the usefulness of
the central monitoring system if these values are not
specified: -

    mac_address (attribute)
    os_name (attribute)
    os_version (attribute)
    cpus *
    load *
    memory *
    swap *
    disk *
    users (attribute)

* - These parameters are specified with tags of the same
    name.  They contain nested XML tags.  See later in
    this document for more details about that!

All of the values marked as being attributes above must be
specified as an attribute of the root tag ("packet")


OTHER: -

Any other arbitrary values may be specified by the host.
The only provision here is that they may not share the
same name as another tag or attribute within the same
heirarchical level.  The server will be able to handle
such clashes, however, it cannot be guaranteed which
of the duplicate parameters would gain priority.


Detail of a complete 'expected' XML packet via UDP
--------------------------------------------------

I shall start by showing an example of a complete
'expected' packet.  I shall then explain the contents
of it.

Here is an example of an XML packet, containing all of
the 'essential' and 'expected' data: -

<packet ip="" seq_no="" date="" machine_name=""
        mac_address="" os_name="" os_version=""
        user="">
    <cpus>
        <cpu0></cpu0>
        <cpu1></cpu1>
        <cpu2></cpu2>
    </cpus>
    <load>
        <load1><load1>
        <load5><load5>
        <load15><load15>
    </load>
    <memory>
        <free></free>
        <total></total>
    </memory>
    <swap>
        <free></free>
        <total></total>
    </swap>
    <disk>
        <p0>
            <free></free>
            <total></total>
            <mounted></mounted>
            <label></label>
        </p0>
        <p1>
            <free></free>
            <total></total>
            <mount></mount>
            <label></label>
        </p1>
    </disk>
</packet>

Note that linefeeds and spaces would normally not be present.
These are only shown above for structure clarity.

Note that all of the essential attributes have been specified
as attributes of the root node.  The expected attribute
values are also attributes of the root node.

The <cpus> tag begins a list of cpu load information.
Each subtag herein must be of the form cpux, where x is
the number of the processor in multiprocessor systems.
Note that this numbering starts from zero.

The <load> tag begins a list of system load information.
The load tag is only expected to contain 3 subtags, named
"load1", "load5" and "load15".  These represent the average
load 1 minute ago, 5 minutes ago and 15 minutes ago
respectively.

The <memory> tag begins a list of system memory information.
This tag is only expected to contain two subtags, as shown
in the example above.  These two subtags specify how much
memory the system has free and how much memory the system
has in total.

The <swap> tag is identical in nature to the <memory> tag.

The <disk> tag begins a list of partition information.
Each subtag must be of the form px, where x is the number
of the disk partition that is unique for that particular
system.  Note that the numbering of px is expected to
start from zero.  Each <px> tag contains the following
subtags: -
    <free> - the free partition space in bytes.
    <total> - the total partition size in bytes.
    <mount> - the mount point, eg, "/var".
    <label> - the label of the physical disk.

The <disk_total> tag begins a list of total disk space,
similar in implementation to the <disk_free> tag.

Any other values and attributes may be specified in the
packet, providing they are not specified in the same
heirarchical level as another attribute or value with
the same name.


Flexibility and Minimising bandwidth
------------------------------------

The following issues still stand: -

The means of submitting host data via UDP containing XML
markup is provided so that future customisation is easily
possible.  It would be possible to easily tailor a custom
piece of host monitoring software to provide exactly what
data is desired for adequate monitoring.

Some of the XML markup demonstrated above contains a lot
of rendundant features.  For example, it is not necessary
to lay the contents out neatly (although this certainly
helps visualise the contents).

The amount of data sent within each UDP packet may be (in
some cases, vastly) reduced by using some of the ideas
described below: -

  - Remove unnecessary linefeeds and 'white space'

  - If a single piece of data is to be represented, it
    will usually occupy less space if it is stored as
    an attribute to a tag, rather than within a pair
    of tags.

  - Comments within the XML may be useful for testing
    purposes, however, the server ignores all comments
    so these can be removed to reduce packet sizes.

Taking the above into account, here is an example of some
ideally formed XML to be contained within a UDP packet: -

    <packet machine_name="raptor" ip="aaa.bbb.ccc.ddd"
    ><freespace><drive1>23677</drive1><drive2>23534</d
    rive2><drive3>10390</drive3></freespace></packet>

Notice how all unnecessary 'white space' and linefeeds have
been removed.  The comment has also been removed.  Values
such as "machine_name" and "ip" have both been stored as an
attribute of the root node ("packet") as this results in
a smaller packet size.


About
-----

This document was written by Paul Mutton [pjm2@kent.ac.uk]
for use by the team working on a 3rd year Computer Science 
project called "i-scream". More details can be found on the 
project website - http://www.i-scream.org

 
Revision:	1.7
Committed:	Sun Aug 1 10:39:55 2004 UTC (20 years, 3 months ago) by tdb
Content type:	text/plain
Branch:	MAIN
CVS Tags:	HEAD
Changes since 1.6:	+4 -4 lines
Log Message:	Catch a lot of old URL's and update them. Also remove a couple of old files that aren't used.
#	User	Rev	Content
1	tdb	1.6	XML via UDP - expected data from hosts
2			======================================
3
4
5	tdb	1.7	<<< Provisional, pjm2@kent.ac.uk >>>
6	tdb	1.6
7
8			Purpose of this document
9			------------------------
10
11			This document shall help to provide a separate party with
12			the knowledge required to use their own implementation of
13			a piece of host monitoring software. In particular, this
14			document details the data that is to be expected from a
15			host. This document is additional to
16	tdb	1.7	http://www.i-scream.org/cgi-bin/docs.cgi?doc=specification/xml_via_udp.txt
17	tdb	1.6
18
19			Background
20			----------
21
22			Hosts are expected to periodically send UDP packets to the
23			central monitoring system. Such packets may contain various
24			pieces of information about the host, such as how much free
25			memory is remaining on the host, etc. Some of this data is
26			deemed to be 'essential', some is 'expected' and the rest
27			is not necessary, but shall not be ignored by the central
28			monitoring system server.
29
30
31			Specification
32			-------------
33
34			The UDP packet must contain a complete and well-formed
35			piece of XML markup, describing the data that the host is
36			submitting. In the event of a piece of data containing
37			an invalid character, such as a "<", it is the responsibilty
38			of the host to format this in a valid manner, for example
39			by replacing all occurrences of "<" with "&lt" before it is
40			sent in a UDP packet to the central monitoring system server.
41
42			Alternatively, the data may be wrapped up in special CDATA
43			tags. For example, if we wish to send the following text:
44
45			<>&!"
46
47			surrounded by <test_chars> tags then we can safely send this
48			by wrapping it up to produce the following XML snippet:
49
50			<test_chars><![CDATA[<>&!"]]></test_chars>
51
52			The bare minimum that a host should send is the following: -
53
54			<packet>
55			</packet>
56
57			Note that every XML tag must also have a matching closing
58			tag.
59
60			However, such a packet is rather useless when it is
61			received by the server - it does not provide any useful
62			information. As such, a packet will be rejected by the
63			server if it does not specify the machine_name, its
64			I.P. address (ip), a packet sequence number (seq_no)
65			and a date. Note that the date is a long integer
66			representing the number of seconds since the epoch.
67			All of these are expected to be specified as an attribute
68			of the packet tag. If they are not specified here, then
69			the packet will be rejected.
70
71			Thus, to ensure that a packet has a chance of being
72			acted upon, a host must specify at least four attribute
73			values - machine_name, ip, date and seq_no. These must
74			be formed in the style shown below.
75
76			<packet machine_name="raptor" ip="aaa.bbb.ccc.ddd"
77			date="93456245245" seq_no="1234567">
78			</packet>
79
80			Note that the server may run additional 'plugin' tools
81			that are designed to filter/reject packets under certain
82			conditions. Please check with the server administrator
83			if you wish to know about the specific configuration of
84			your central monitoring system.
85
86			The I.P. address and machine_name could be found from
87			the UDP packet itself, but these remain essential so
88			as to provide resiliance from stray packets that do not
89			contain any valid or useful data.
90
91			Remember that malformed XML data would be rejected by the
92			central monitoring system immediately, without acting upon
93			it.
94
95
96			Classification of XML data in our system
97			----------------------------------------
98
99
100			ESSENTIAL: -
101
102			The following values are ESSENTIAL in each packet. Any
103			packet without these will be rejected. They MUST be
104			specified as attributes of the root node ("packet"): -
105
106			ip
107			seq_no
108			date
109			machine_name
110
111
112			EXPECTED: -
113
114			The following values are EXPECTED in each packet. All
115			packets are RECOMMENDED to specify these values, as the
116			server is designed to permanently store such data in an
117			easy and quick to access manner. It is not essential to
118			specify such values, but it does limit the usefulness of
119			the central monitoring system if these values are not
120			specified: -
121
122			mac_address (attribute)
123			os_name (attribute)
124			os_version (attribute)
125			cpus *
126			load *
127			memory *
128			swap *
129			disk *
130			users (attribute)
131
132			* - These parameters are specified with tags of the same
133			name. They contain nested XML tags. See later in
134			this document for more details about that!
135
136			All of the values marked as being attributes above must be
137			specified as an attribute of the root tag ("packet")
138
139
140			OTHER: -
141
142			Any other arbitrary values may be specified by the host.
143			The only provision here is that they may not share the
144			same name as another tag or attribute within the same
145			heirarchical level. The server will be able to handle
146			such clashes, however, it cannot be guaranteed which
147			of the duplicate parameters would gain priority.
148
149
150			Detail of a complete 'expected' XML packet via UDP
151			--------------------------------------------------
152
153			I shall start by showing an example of a complete
154			'expected' packet. I shall then explain the contents
155			of it.
156
157			Here is an example of an XML packet, containing all of
158			the 'essential' and 'expected' data: -
159
160			<packet ip="" seq_no="" date="" machine_name=""
161			mac_address="" os_name="" os_version=""
162			user="">
163			<cpus>
164			<cpu0></cpu0>
165			<cpu1></cpu1>
166			<cpu2></cpu2>
167			</cpus>
168			<load>
169			<load1><load1>
170			<load5><load5>
171			<load15><load15>
172			</load>
173			<memory>
174			<free></free>
175			<total></total>
176			</memory>
177			<swap>
178			<free></free>
179			<total></total>
180			</swap>
181			<disk>
182			<p0>
183			<free></free>
184			<total></total>
185			<mounted></mounted>
186			<label></label>
187			</p0>
188			<p1>
189			<free></free>
190			<total></total>
191			<mount></mount>
192			<label></label>
193			</p1>
194			</disk>
195			</packet>
196
197			Note that linefeeds and spaces would normally not be present.
198			These are only shown above for structure clarity.
199
200			Note that all of the essential attributes have been specified
201			as attributes of the root node. The expected attribute
202			values are also attributes of the root node.
203
204			The <cpus> tag begins a list of cpu load information.
205			Each subtag herein must be of the form cpux, where x is
206			the number of the processor in multiprocessor systems.
207			Note that this numbering starts from zero.
208
209			The <load> tag begins a list of system load information.
210			The load tag is only expected to contain 3 subtags, named
211			"load1", "load5" and "load15". These represent the average
212			load 1 minute ago, 5 minutes ago and 15 minutes ago
213			respectively.
214
215			The <memory> tag begins a list of system memory information.
216			This tag is only expected to contain two subtags, as shown
217			in the example above. These two subtags specify how much
218			memory the system has free and how much memory the system
219			has in total.
220
221			The <swap> tag is identical in nature to the <memory> tag.
222
223			The <disk> tag begins a list of partition information.
224			Each subtag must be of the form px, where x is the number
225			of the disk partition that is unique for that particular
226			system. Note that the numbering of px is expected to
227			start from zero. Each <px> tag contains the following
228			subtags: -
229			<free> - the free partition space in bytes.
230			<total> - the total partition size in bytes.
231			<mount> - the mount point, eg, "/var".
232			<label> - the label of the physical disk.
233
234			The <disk_total> tag begins a list of total disk space,
235			similar in implementation to the <disk_free> tag.
236
237			Any other values and attributes may be specified in the
238			packet, providing they are not specified in the same
239			heirarchical level as another attribute or value with
240			the same name.
241
242
243			Flexibility and Minimising bandwidth
244			------------------------------------
245
246			The following issues still stand: -
247
248			The means of submitting host data via UDP containing XML
249			markup is provided so that future customisation is easily
250			possible. It would be possible to easily tailor a custom
251			piece of host monitoring software to provide exactly what
252			data is desired for adequate monitoring.
253
254			Some of the XML markup demonstrated above contains a lot
255			of rendundant features. For example, it is not necessary
256			to lay the contents out neatly (although this certainly
257			helps visualise the contents).
258
259			The amount of data sent within each UDP packet may be (in
260			some cases, vastly) reduced by using some of the ideas
261			described below: -
262
263			- Remove unnecessary linefeeds and 'white space'
264
265			- If a single piece of data is to be represented, it
266			will usually occupy less space if it is stored as
267			an attribute to a tag, rather than within a pair
268			of tags.
269
270			- Comments within the XML may be useful for testing
271			purposes, however, the server ignores all comments
272			so these can be removed to reduce packet sizes.
273
274			Taking the above into account, here is an example of some
275			ideally formed XML to be contained within a UDP packet: -
276
277			<packet machine_name="raptor" ip="aaa.bbb.ccc.ddd"
278			><freespace><drive1>23677</drive1><drive2>23534</d
279			rive2><drive3>10390</drive3></freespace></packet>
280
281			Notice how all unnecessary 'white space' and linefeeds have
282			been removed. The comment has also been removed. Values
283			such as "machine_name" and "ip" have both been stored as an
284			attribute of the root node ("packet") as this results in
285			a smaller packet size.
286
287
288			About
289			-----
290
291	tdb	1.7	This document was written by Paul Mutton [pjm2@kent.ac.uk]
292	tdb	1.6	for use by the team working on a 3rd year Computer Science
293			project called "i-scream". More details can be found on the
294	tdb	1.7	project website - http://www.i-scream.org
295	tdb	1.6
296