documentation/specification/spec-realtime.txt

I-Scream Specification Outline (Realtime side only)
===================================================

ajm4, 30/10/2000
tdb1, 30/10/2000

System Component Startup
************************

CORE
----
The core of the system provides little or no functionality
to the operation of the system, but instead oversees the
running.  At startup this should be the first component to
intansiate.  It essentially acts as a central logging and 
configuration distrubution site, the "central" in
centralised monitoring system.  It may also be running the
ORB or some components related to it.

On startup the first thing it should do is read in any
configuration files, start the logging interface then 
prepare to bring the system online.  This is done by
starting the various components.  If however the system
configuration states that particular components are
operating in "distributed" mode, then it blocks until
the various key components have registered that they are 
online.

Client Interface
----------------
The Client Interface is essentially just one component with 
a series of lists within it. When run it should, obviously, 
create an instance of the Client Interface, and then bind 
this to the ORB and register with the naming service.

It can then read its configuration in from the CORE and get
a hook on the logging service that the CORE provides.

It then needs to construct the "local clients". These 
clients communicate with the system using the same interface
as the external clients, but they are tailored to specific 
purposes, such as E-Mail alerts, and SMS alerts. The Client 
Interface then listens on a "well known" address for clients 
to request a connection.

Filter
------
The filter is broken down into three main subcomponents. 

  - Filter Manager
      The Filter Manager is responsible for managing which 
      filters are used by the hosts. The Filter Manager is 
      available at a "well known" location which is pre-
      programmed into the hosts. The Filter Manager is 
      responsible for creating and managing the other 
      components of the filter system.
  
  - Main Filter
      The Main Filter is the single point that links back 
      into the CORE of the system. It will connect to the 
      DBI and the CLI to deliver data.
  
  - Filters
      There can be multipler Filters, and these are the 
      "front line" to the hosts. They all link back to the 
      Main Filter to send data into the system. It is 
      possible to run these Filters on any machine, allowing 
      management of data flow.

At startup a Filter Manager object is activated at the "well 
known" location (probably a given machine name at a 
predefined port). The Filter Manager will create an instance 
of the Main Filter, and any Filters under it's control. It 
should also bind itself to the ORB and register with the 
naming service. 

It can then read its configuration in from the CORE and get
a hook on the logging service that the CORE provides.

Through some mechanism the other Filters, elsewhere on the
network, will register with the Filter Manager. The Filter
Manager will need to tell each Filter the location of the
Main Filter upon registering. The Filter Manager will then
be in a position to receive connections from hosts and pass
them off to Filters.

System Running State
********************

CORE
----
Once the various components are running then the core is
essentially idle, logging information and handling
configuration changes.

Client Interface
----------------
In the running state the Client Interface is always 
listening for clients on the "well known" address. When a 
connection is received it is passed in to the main Client 
Interface and the client is queried about which hosts it 
wishes to receive information about. This is then stored in 
an internal "routing table" so the Client Interface knows 
which hosts to send the information on to. This routing 
table is constructed with this form;

  host1: client1 client2 client5
  host2: client2
  host3: client3 client4
  host4: client1 client3

This design is such that when a piece of information is 
recieved from a host the Client Interface can immediately 
see which clients wish to receive this data, without too 
much searching.

The "local clients" function just like any other client, 
although they are local, in that they will wish to receive 
information about hosts they are interested in. However, 
they will contain a lot more logic, and be required to work 
out who wants to be alerted about what, and when. They will 
also be responsible for sending the alert.

Filter
------
When a host first loads up it knows where to locate the 
Filter Manager because it's located at a "well known" 
location. The host will fire up a TCP connection to the 
Filter Manager to announce itself. The Filter Manager will 
use some method (logically) to allocate a Filter to the 
host. The Filter Manager should base this decision on 
various factors, such as the load on the selection of 
filters, and possibly the location in relation to the host. 
The host will then be directed to this Filter for all 
further communications.

As the system runs the host will send data with (maybe) UDP 
to the Filter (that it's been allocated to). This choice has 
been made because it puts less onus on the host to make the 
connection, rather the data is just sent out. However, to 
ensure that the data isn't just disappearing into the depths 
of the network a periodic heartbeat will occur (at a 
predefined interval) over TCP to the Filter. This heartbeat 
can be used as a form of two-way communication, ensuring 
that everything is ok, and if required, to send any 
information back to the host. This heartbeat must occur 
otherwise the server may infer the host has died.

This could link in to alerting. An amber alert could be 
initiated for a host if the server stops receiving UDP 
packets, but an red alert be raised if the heartbeat doesn't 
occur.

If, for some reason, the Filter were to disappear the host 
should fall back on it's initial discovering mechanism - ie. 
contacting the Filter Manager at it's "well known" location. 
The host should report that it's lost it's Filter (so the 
Filter Manager can investigate and remove from it's list of 
Filters), and then the Filter Manager will reassign a new 
Filter to the host. Communication can then continue.

The idea of plugins to the Filters has been introduced. 
These plugins will implement a predefined plugin interface, 
and can be chained together at the Filter. Using the 
interface we can easily add future plugins that can do 
anything from parsing new data formats, to implementing 
encryption algorithms. The Filter will pass incoming data to 
each plugin in turn that it has available, and then finally 
pass the data on to the Main Filter. The Filter need not 
have any real knowledge about the content of the data.
Revision:	1.4
Committed:	Tue Oct 31 16:12:25 2000 UTC (24 years ago) by ajm
Content type:	text/plain
Branch:	MAIN
CVS Tags:	PROJECT_COMPLETION, HEAD
Changes since 1.3:	+38 -12 lines
Log Message:	Added information about the CORE and what services it provides, together with information about how the other components make use of it.
#	User	Rev	Content
1	ajm	1.1	I-Scream Specification Outline (Realtime side only)
2			===================================================
3
4			ajm4, 30/10/2000
5	tdb	1.2	tdb1, 30/10/2000
6	ajm	1.1
7			System Component Startup
8	tdb	1.2	************************
9	ajm	1.1
10			CORE
11	tdb	1.2	----
12	ajm	1.4	The core of the system provides little or no functionality
13			to the operation of the system, but instead oversees the
14			running. At startup this should be the first component to
15			intansiate. It essentially acts as a central logging and
16			configuration distrubution site, the "central" in
17			centralised monitoring system. It may also be running the
18			ORB or some components related to it.
19
20			On startup the first thing it should do is read in any
21			configuration files, start the logging interface then
22			prepare to bring the system online. This is done by
23			starting the various components. If however the system
24			configuration states that particular components are
25			operating in "distributed" mode, then it blocks until
26			the various key components have registered that they are
27			online.
28	ajm	1.1
29			Client Interface
30	tdb	1.2	----------------
31	tdb	1.3	The Client Interface is essentially just one component with
32			a series of lists within it. When run it should, obviously,
33			create an instance of the Client Interface, and then bind
34	ajm	1.4	this to the ORB and register with the naming service.
35
36			It can then read its configuration in from the CORE and get
37			a hook on the logging service that the CORE provides.
38
39			It then needs to construct the "local clients". These
40			clients communicate with the system using the same interface
41			as the external clients, but they are tailored to specific
42	tdb	1.3	purposes, such as E-Mail alerts, and SMS alerts. The Client
43			Interface then listens on a "well known" address for clients
44			to request a connection.
45	ajm	1.1
46			Filter
47	tdb	1.2	------
48			The filter is broken down into three main subcomponents.
49
50			- Filter Manager
51			The Filter Manager is responsible for managing which
52			filters are used by the hosts. The Filter Manager is
53			available at a "well known" location which is pre-
54			programmed into the hosts. The Filter Manager is
55			responsible for creating and managing the other
56			components of the filter system.
57
58			- Main Filter
59			The Main Filter is the single point that links back
60			into the CORE of the system. It will connect to the
61			DBI and the CLI to deliver data.
62
63			- Filters
64			There can be multipler Filters, and these are the
65			"front line" to the hosts. They all link back to the
66			Main Filter to send data into the system. It is
67			possible to run these Filters on any machine, allowing
68			management of data flow.
69
70			At startup a Filter Manager object is activated at the "well
71			known" location (probably a given machine name at a
72			predefined port). The Filter Manager will create an instance
73	tdb	1.3	of the Main Filter, and any Filters under it's control. It
74			should also bind itself to the ORB and register with the
75	ajm	1.4	naming service.
76
77			It can then read its configuration in from the CORE and get
78			a hook on the logging service that the CORE provides.
79
80			Through some mechanism the other Filters, elsewhere on the
81			network, will register with the Filter Manager. The Filter
82			Manager will need to tell each Filter the location of the
83			Main Filter upon registering. The Filter Manager will then
84			be in a position to receive connections from hosts and pass
85			them off to Filters.
86	ajm	1.1
87			System Running State
88	tdb	1.2	********************
89	ajm	1.1
90			CORE
91	tdb	1.2	----
92	ajm	1.4	Once the various components are running then the core is
93			essentially idle, logging information and handling
94			configuration changes.
95	ajm	1.1
96			Client Interface
97	tdb	1.2	----------------
98	tdb	1.3	In the running state the Client Interface is always
99			listening for clients on the "well known" address. When a
100			connection is received it is passed in to the main Client
101			Interface and the client is queried about which hosts it
102			wishes to receive information about. This is then stored in
103			an internal "routing table" so the Client Interface knows
104			which hosts to send the information on to. This routing
105			table is constructed with this form;
106
107			host1: client1 client2 client5
108			host2: client2
109			host3: client3 client4
110			host4: client1 client3
111
112			This design is such that when a piece of information is
113			recieved from a host the Client Interface can immediately
114			see which clients wish to receive this data, without too
115			much searching.
116
117			The "local clients" function just like any other client,
118			although they are local, in that they will wish to receive
119			information about hosts they are interested in. However,
120			they will contain a lot more logic, and be required to work
121			out who wants to be alerted about what, and when. They will
122			also be responsible for sending the alert.
123	ajm	1.1
124			Filter
125	tdb	1.2	------
126			When a host first loads up it knows where to locate the
127			Filter Manager because it's located at a "well known"
128			location. The host will fire up a TCP connection to the
129			Filter Manager to announce itself. The Filter Manager will
130			use some method (logically) to allocate a Filter to the
131			host. The Filter Manager should base this decision on
132			various factors, such as the load on the selection of
133			filters, and possibly the location in relation to the host.
134			The host will then be directed to this Filter for all
135			further communications.
136
137			As the system runs the host will send data with (maybe) UDP
138			to the Filter (that it's been allocated to). This choice has
139			been made because it puts less onus on the host to make the
140			connection, rather the data is just sent out. However, to
141			ensure that the data isn't just disappearing into the depths
142			of the network a periodic heartbeat will occur (at a
143			predefined interval) over TCP to the Filter. This heartbeat
144			can be used as a form of two-way communication, ensuring
145			that everything is ok, and if required, to send any
146			information back to the host. This heartbeat must occur
147			otherwise the server may infer the host has died.
148
149			This could link in to alerting. An amber alert could be
150			initiated for a host if the server stops receiving UDP
151			packets, but an red alert be raised if the heartbeat doesn't
152			occur.
153
154			If, for some reason, the Filter were to disappear the host
155			should fall back on it's initial discovering mechanism - ie.
156			contacting the Filter Manager at it's "well known" location.
157			The host should report that it's lost it's Filter (so the
158			Filter Manager can investigate and remove from it's list of
159			Filters), and then the Filter Manager will reassign a new
160			Filter to the host. Communication can then continue.
161
162			The idea of plugins to the Filters has been introduced.
163			These plugins will implement a predefined plugin interface,
164			and can be chained together at the Filter. Using the
165			interface we can easily add future plugins that can do
166			anything from parsing new data formats, to implementing
167			encryption algorithms. The Filter will pass incoming data to
168			each plugin in turn that it has available, and then finally
169			pass the data on to the Main Filter. The Filter need not
170			have any real knowledge about the content of the data.