ViewVC Help
View File | Revision Log | Show Annotations | Revision Graph | Root Listing
root/i-scream/projects/cms/documentation/specification/expected_data.txt
(Generate patch)

Comparing projects/cms/documentation/specification/expected_data.txt (file contents):
Revision 1.1 by pjm2, Tue Nov 28 09:00:33 2000 UTC vs.
Revision 1.7 by tdb, Sun Aug 1 10:39:55 2004 UTC

# Line 1 | Line 1
1 < XML via UDP - expected data from hosts
2 < ======================================
3 <
4 <
5 < <<< Provisional, pjm2@ukc.ac.uk >>>
6 <
7 <
8 < Purpose of this document
9 < ------------------------
10 <
11 < This document shall help to provide a separate party with
12 < the knowledge required to use their own implementation of
13 < a piece of host monitoring software.  In particular, this
14 < document details the data that is to be expected from a
15 < host.  This document is additional to
16 < http://www.i-scream.org.uk/cgi-bin/docs.cgi?doc=specification/xml_via_udp.txt
17 <
18 <
19 < Background
20 < ----------
21 <
22 < Hosts are expected to periodically send UDP packets to the
23 < central monitoring system.  Such packets may contain various
24 < pieces of information about the host, such as how much free
25 < memory is remaining on the host, etc.  Some of this data is
26 < deemed to be 'essential', some is 'expected' and the rest
27 < is not necessary, but shall not be ignored by the central
28 < monitoring system server.
29 <
30 <
31 < Specification
32 < -------------
33 <
34 < The UDP packet must contain a complete and well-formed
35 < piece of XML markup, describing the data that the host is
36 < submitting.  In the event of a piece of data containing
37 < an invalid character, such as a "<", it is the responsibilty
38 < of the host to format this in a valid manner, for example
39 < by replacing all occurrences of "<" with "&lt" before it is
40 < sent in a UDP packet to the central monitoring system server.
41 <
42 < Alternatively, the data may be wrapped up in special CDATA
43 < tags.  For example, if we wish to send the following text:
44 <
45 <    <>&!"
46 <
47 < surrounded by <test_chars> tags then we can safely send this
48 < by wrapping it up to produce the following XML snippet:
49 <
50 <    <test_chars><![CDATA[<>&!"]]></test_chars>
51 <
52 < The bare minimum that a host should send is the following: -
53 <
54 <    <packet>
55 <    </packet>
56 <
57 < Note that every XML tag must also have a matching closing
58 < tag.
59 <
60 < However, such a packet is rather useless when it is
61 < received by the server - it does not provide any useful
62 < information.  As such, a packet will be rejected by the
63 < server if it does not specify the machine_name, its
64 < I.P. address (ip), a packet sequence number (seq_no)
65 < and a date.  Note that the data is a long integer
66 < representing the number of seconds since the epoch.
67 < All of these are expected to be specified as an attribute
68 < of the packet tag.  If they are not specified here, then
69 < the packet will be rejected.
70 <
71 < Thus, to ensure that a packet has a chance of being
72 < acted upon, a host must specify at least four attribute
73 < values - machine_name, ip, date and seq_no.  These must
74 < be formed in the style shown below.
75 <
76 <    <packet machine_name="raptor" ip="aaa.bbb.ccc.ddd"
77 <            date="93456245245" seq_no="1234567">
78 <    </packet>
79 <
80 < Note that the server may run additional 'plugin' tools
81 < that are designed to filter/reject packets under certain
82 < conditions.  Please check with the server administrator
83 < if you wish to know about the specific configuration of
84 < your central monitoring system.
85 <
86 < The I.P. address and machine_name could be found from
87 < the UDP packet itself, but these remain essential so
88 < as to provide resiliance from stray packets that do not
89 < contain any valid or useful data.
90 <
91 < Remember that malformed XML data would be rejected by the
92 < central monitoring system immediately, without acting upon
93 < it.
94 <
95 <
96 < Classification of XML data in our system
97 < ----------------------------------------
98 <
99 <
100 < ESSENTIAL: -
101 <
102 < The following values are ESSENTIAL in each packet.  Any
103 < packet without these will be rejected.  They MUST be
104 < specified as attributes of the root node ("packet"): -
105 <
106 <    ip
107 <    seq_no
108 <    date
109 <    machine_name
110 <
111 <
112 < EXPECTED: -
113 <
114 < The following values are EXPECTED in each packet.  All
115 < packets are RECOMMENDED to specify these values, as the
116 < server is designed to permanently store such data in an
117 < easy and quick to access manner.  It is not essential to
118 < specify such values, but it does limit the usefulness of
119 < the central monitoring system if these values are not
120 < specified: -
121 <
122 <    mac_address (attribute)
123 <    os_name (attribute)
124 <    os_version (attribute)
125 <    cpus *
126 <    load *
127 <    memory *
128 <    swap *
129 <    disk_free *
130 <    disk_total *
131 <    users (attribute)
132 <
133 < * - These parameters are specified with tags of the same
134 <    name.  They contain nested XML tags.  See later in
135 <    this document for more details about that!
136 <
137 < All of the values marked as being attributes above must be
138 < specified as an attribute of the root tag ("packet")
139 <
140 <
141 < OTHER: -
142 <
143 < Any other arbitrary values may be specified by the host.
144 < The only provision here is that they may not share the
145 < same name as another tag or attribute within the same
146 < heirarchical level.  The server will be able to handle
147 < such clashes, however, it cannot be guaranteed which
148 < of the duplicate parameters would gain priority.
149 <
150 <
151 < Detail of a complete 'expected' XML packet via UDP
152 < --------------------------------------------------
153 <
154 < I shall start by showing an example of a complete
155 < 'expected' packet.  I shall then explain the contents
156 < of it.
157 <
158 < Here is an example of an XML packet, containing all of
159 < the 'essential' and 'expected' data: -
160 <
161 < <packet ip="" seq_no="" date="" machine_name=""
162 <        mac_address="" os_name="" os_version=""
163 <        user="">
164 <    <cpus>
165 <        <cpu1></cpu1>
166 <        <cpu2></cpu2>
167 <        <cpu3></cpu3>
168 <    </cpus>
169 <    <load>
170 <        <load1><load1>
171 <        <load5><load5>
172 <        <load10><load10>
173 <    </load>
174 <    <memory>
175 <        <free></free>
176 <        <total></total>
177 <    </memory>
178 <    <swap>
179 <        <free></free>
180 <        <total></total>
181 <    </swap>
182 <    <disk_free>
183 <        <p1></p1>
184 <        <p2></p2>
185 <        <p3></p3>
186 <    </disk_free>
187 <    <disk_total>
188 <        <p1></p1>
189 <        <p2></p2>
190 <        <p3></p3>
191 <    </disk_total>
192 < </packet>
193 <
194 < Note that linefeeds and spaces would normally not be present.
195 < These are only shown above for structure clarity.
196 <
197 < Note that all of the essential attributes have been specified
198 < as attributes of the root node.  The expected attribute
199 < values are also attributes of the root node.
200 <
201 < The <cpus> tag begins a list of cpu load information.
202 < Each subtag herein must be of the form cpux, where x is
203 < the number of the processor in multiprocessor systems.
204 < Note that this numbering starts from zero.
205 <
206 < The <load> tag begins a list of system load information.
207 < The load tag is only expected to contain 3 subtags, named
208 < "load1", "load5" and "load10".  These represent the average
209 < load 1 minutes ago, 5 minutes ago and 10 minutes ago
210 < respectively.
211 <
212 < The <memory> tag begins a list of system memory information.
213 < This tag is only expected to contain two subtags, as shown
214 < in the example above.  These two subtags specify how much
215 < memory the system has free and how much memory the system
216 < has in total.
217 <
218 < The <swap> tag is identical in nature to the <memory> tag.
219 <
220 < The <disk_free> tag begins a list of free disk space
221 < information.  Each subtag must be of the form px, where
222 < x is the number of the disk partition that is unique for
223 < that particular system.  The value represents how much
224 < space is free on that partition.  Note that px numbering
225 < starts from zero.
226 <
227 < The <disk_total> tag begins a list of total disk space,
228 < similar in implementation to the <disk_free> tag.
229 <
230 < Any other values and attributes may be specified in the
231 < packet, providing they are not specified in the same
232 < heirarchical level as another attribute or value with
233 < the same name.
234 <
235 <
236 < Flexibility and Minimising bandwidth
237 < ------------------------------------
238 <
239 < The following issues still stand: -
240 <
241 < The means of submitting host data via UDP containing XML
242 < markup is provided so that future customisation is easily
243 < possible.  It would be possible to easily tailor a custom
244 < piece of host monitoring software to provide exactly what
245 < data is desired for adequate monitoring.
246 <
247 < Some of the XML markup demonstrated above contains a lot
248 < of rendundant features.  For example, it is not necessary
249 < to lay the contents out neatly (although this certainly
250 < helps visualise the contents).
251 <
252 < The amount of data sent within each UDP packet may be (in
253 < some cases, vastly) reduced by using some of the ideas
254 < described below: -
255 <
256 <  - Remove unnecessary linefeeds and 'white space'
257 <
258 <  - If a single piece of data is to be represented, it
259 <    will usually occupy less space if it is stored as
260 <    an attribute to a tag, rather than within a pair
261 <    of tags.
262 <
263 <  - Comments within the XML may be useful for testing
264 <    purposes, however, the server ignores all comments
265 <    so these can be removed to reduce packet sizes.
266 <
267 < Taking the above into account, here is an example of some
268 < ideally formed XML to be contained within a UDP packet: -
269 <
270 <    <packet machine_name="raptor" ip="aaa.bbb.ccc.ddd"
271 <    ><freespace><drive1>23677</drive1><drive2>23534</d
272 <    rive2><drive3>10390</drive3></freespace></packet>
273 <
274 < Notice how all unnecessary 'white space' and linefeeds have
275 < been removed.  The comment has also been removed.  Values
276 < such as "machine_name" and "ip" have both been stored as an
277 < attribute of the root node ("packet") as this results in
278 < a smaller packet size.
279 <
280 <
281 < About
282 < -----
283 <
284 < This document was written by Paul Mutton [pjm2@ukc.ac.uk]
285 < for use by the team working on a 3rd year Computer Science
286 < project called "i-scream". More details can be found on the
287 < project website - http://www.i-scream.org.uk
288 <
289 <
1 > XML via UDP - expected data from hosts
2 > ======================================
3 >
4 >
5 > <<< Provisional, pjm2@kent.ac.uk >>>
6 >
7 >
8 > Purpose of this document
9 > ------------------------
10 >
11 > This document shall help to provide a separate party with
12 > the knowledge required to use their own implementation of
13 > a piece of host monitoring software.  In particular, this
14 > document details the data that is to be expected from a
15 > host.  This document is additional to
16 > http://www.i-scream.org/cgi-bin/docs.cgi?doc=specification/xml_via_udp.txt
17 >
18 >
19 > Background
20 > ----------
21 >
22 > Hosts are expected to periodically send UDP packets to the
23 > central monitoring system.  Such packets may contain various
24 > pieces of information about the host, such as how much free
25 > memory is remaining on the host, etc.  Some of this data is
26 > deemed to be 'essential', some is 'expected' and the rest
27 > is not necessary, but shall not be ignored by the central
28 > monitoring system server.
29 >
30 >
31 > Specification
32 > -------------
33 >
34 > The UDP packet must contain a complete and well-formed
35 > piece of XML markup, describing the data that the host is
36 > submitting.  In the event of a piece of data containing
37 > an invalid character, such as a "<", it is the responsibilty
38 > of the host to format this in a valid manner, for example
39 > by replacing all occurrences of "<" with "&lt" before it is
40 > sent in a UDP packet to the central monitoring system server.
41 >
42 > Alternatively, the data may be wrapped up in special CDATA
43 > tags.  For example, if we wish to send the following text:
44 >
45 >    <>&!"
46 >
47 > surrounded by <test_chars> tags then we can safely send this
48 > by wrapping it up to produce the following XML snippet:
49 >
50 >    <test_chars><![CDATA[<>&!"]]></test_chars>
51 >
52 > The bare minimum that a host should send is the following: -
53 >
54 >    <packet>
55 >    </packet>
56 >
57 > Note that every XML tag must also have a matching closing
58 > tag.
59 >
60 > However, such a packet is rather useless when it is
61 > received by the server - it does not provide any useful
62 > information.  As such, a packet will be rejected by the
63 > server if it does not specify the machine_name, its
64 > I.P. address (ip), a packet sequence number (seq_no)
65 > and a date.  Note that the date is a long integer
66 > representing the number of seconds since the epoch.
67 > All of these are expected to be specified as an attribute
68 > of the packet tag.  If they are not specified here, then
69 > the packet will be rejected.
70 >
71 > Thus, to ensure that a packet has a chance of being
72 > acted upon, a host must specify at least four attribute
73 > values - machine_name, ip, date and seq_no.  These must
74 > be formed in the style shown below.
75 >
76 >    <packet machine_name="raptor" ip="aaa.bbb.ccc.ddd"
77 >            date="93456245245" seq_no="1234567">
78 >    </packet>
79 >
80 > Note that the server may run additional 'plugin' tools
81 > that are designed to filter/reject packets under certain
82 > conditions.  Please check with the server administrator
83 > if you wish to know about the specific configuration of
84 > your central monitoring system.
85 >
86 > The I.P. address and machine_name could be found from
87 > the UDP packet itself, but these remain essential so
88 > as to provide resiliance from stray packets that do not
89 > contain any valid or useful data.
90 >
91 > Remember that malformed XML data would be rejected by the
92 > central monitoring system immediately, without acting upon
93 > it.
94 >
95 >
96 > Classification of XML data in our system
97 > ----------------------------------------
98 >
99 >
100 > ESSENTIAL: -
101 >
102 > The following values are ESSENTIAL in each packet.  Any
103 > packet without these will be rejected.  They MUST be
104 > specified as attributes of the root node ("packet"): -
105 >
106 >    ip
107 >    seq_no
108 >    date
109 >    machine_name
110 >
111 >
112 > EXPECTED: -
113 >
114 > The following values are EXPECTED in each packet.  All
115 > packets are RECOMMENDED to specify these values, as the
116 > server is designed to permanently store such data in an
117 > easy and quick to access manner.  It is not essential to
118 > specify such values, but it does limit the usefulness of
119 > the central monitoring system if these values are not
120 > specified: -
121 >
122 >    mac_address (attribute)
123 >    os_name (attribute)
124 >    os_version (attribute)
125 >    cpus *
126 >    load *
127 >    memory *
128 >    swap *
129 >    disk *
130 >    users (attribute)
131 >
132 > * - These parameters are specified with tags of the same
133 >    name.  They contain nested XML tags.  See later in
134 >    this document for more details about that!
135 >
136 > All of the values marked as being attributes above must be
137 > specified as an attribute of the root tag ("packet")
138 >
139 >
140 > OTHER: -
141 >
142 > Any other arbitrary values may be specified by the host.
143 > The only provision here is that they may not share the
144 > same name as another tag or attribute within the same
145 > heirarchical level.  The server will be able to handle
146 > such clashes, however, it cannot be guaranteed which
147 > of the duplicate parameters would gain priority.
148 >
149 >
150 > Detail of a complete 'expected' XML packet via UDP
151 > --------------------------------------------------
152 >
153 > I shall start by showing an example of a complete
154 > 'expected' packet.  I shall then explain the contents
155 > of it.
156 >
157 > Here is an example of an XML packet, containing all of
158 > the 'essential' and 'expected' data: -
159 >
160 > <packet ip="" seq_no="" date="" machine_name=""
161 >        mac_address="" os_name="" os_version=""
162 >        user="">
163 >    <cpus>
164 >        <cpu0></cpu0>
165 >        <cpu1></cpu1>
166 >        <cpu2></cpu2>
167 >    </cpus>
168 >    <load>
169 >        <load1><load1>
170 >        <load5><load5>
171 >        <load15><load15>
172 >    </load>
173 >    <memory>
174 >        <free></free>
175 >        <total></total>
176 >    </memory>
177 >    <swap>
178 >        <free></free>
179 >        <total></total>
180 >    </swap>
181 >    <disk>
182 >        <p0>
183 >            <free></free>
184 >            <total></total>
185 >            <mounted></mounted>
186 >            <label></label>
187 >        </p0>
188 >        <p1>
189 >            <free></free>
190 >            <total></total>
191 >            <mount></mount>
192 >            <label></label>
193 >        </p1>
194 >    </disk>
195 > </packet>
196 >
197 > Note that linefeeds and spaces would normally not be present.
198 > These are only shown above for structure clarity.
199 >
200 > Note that all of the essential attributes have been specified
201 > as attributes of the root node.  The expected attribute
202 > values are also attributes of the root node.
203 >
204 > The <cpus> tag begins a list of cpu load information.
205 > Each subtag herein must be of the form cpux, where x is
206 > the number of the processor in multiprocessor systems.
207 > Note that this numbering starts from zero.
208 >
209 > The <load> tag begins a list of system load information.
210 > The load tag is only expected to contain 3 subtags, named
211 > "load1", "load5" and "load15".  These represent the average
212 > load 1 minute ago, 5 minutes ago and 15 minutes ago
213 > respectively.
214 >
215 > The <memory> tag begins a list of system memory information.
216 > This tag is only expected to contain two subtags, as shown
217 > in the example above.  These two subtags specify how much
218 > memory the system has free and how much memory the system
219 > has in total.
220 >
221 > The <swap> tag is identical in nature to the <memory> tag.
222 >
223 > The <disk> tag begins a list of partition information.
224 > Each subtag must be of the form px, where x is the number
225 > of the disk partition that is unique for that particular
226 > system.  Note that the numbering of px is expected to
227 > start from zero.  Each <px> tag contains the following
228 > subtags: -
229 >    <free> - the free partition space in bytes.
230 >    <total> - the total partition size in bytes.
231 >    <mount> - the mount point, eg, "/var".
232 >    <label> - the label of the physical disk.
233 >
234 > The <disk_total> tag begins a list of total disk space,
235 > similar in implementation to the <disk_free> tag.
236 >
237 > Any other values and attributes may be specified in the
238 > packet, providing they are not specified in the same
239 > heirarchical level as another attribute or value with
240 > the same name.
241 >
242 >
243 > Flexibility and Minimising bandwidth
244 > ------------------------------------
245 >
246 > The following issues still stand: -
247 >
248 > The means of submitting host data via UDP containing XML
249 > markup is provided so that future customisation is easily
250 > possible.  It would be possible to easily tailor a custom
251 > piece of host monitoring software to provide exactly what
252 > data is desired for adequate monitoring.
253 >
254 > Some of the XML markup demonstrated above contains a lot
255 > of rendundant features.  For example, it is not necessary
256 > to lay the contents out neatly (although this certainly
257 > helps visualise the contents).
258 >
259 > The amount of data sent within each UDP packet may be (in
260 > some cases, vastly) reduced by using some of the ideas
261 > described below: -
262 >
263 >  - Remove unnecessary linefeeds and 'white space'
264 >
265 >  - If a single piece of data is to be represented, it
266 >    will usually occupy less space if it is stored as
267 >    an attribute to a tag, rather than within a pair
268 >    of tags.
269 >
270 >  - Comments within the XML may be useful for testing
271 >    purposes, however, the server ignores all comments
272 >    so these can be removed to reduce packet sizes.
273 >
274 > Taking the above into account, here is an example of some
275 > ideally formed XML to be contained within a UDP packet: -
276 >
277 >    <packet machine_name="raptor" ip="aaa.bbb.ccc.ddd"
278 >    ><freespace><drive1>23677</drive1><drive2>23534</d
279 >    rive2><drive3>10390</drive3></freespace></packet>
280 >
281 > Notice how all unnecessary 'white space' and linefeeds have
282 > been removed.  The comment has also been removed.  Values
283 > such as "machine_name" and "ip" have both been stored as an
284 > attribute of the root node ("packet") as this results in
285 > a smaller packet size.
286 >
287 >
288 > About
289 > -----
290 >
291 > This document was written by Paul Mutton [pjm2@kent.ac.uk]
292 > for use by the team working on a 3rd year Computer Science
293 > project called "i-scream". More details can be found on the
294 > project website - http://www.i-scream.org
295 >
296 >

Diff Legend

Removed lines
+ Added lines
< Changed lines
> Changed lines