documentation/specification/spec-realtime.txt

I-Scream Specification Outline (Realtime side only)
===================================================

ajm4, 30/10/2000
tdb1, 30/10/2000

System Component Startup
************************

CORE
----


Client Interface
----------------


Filter
------
The filter is broken down into three main subcomponents. 

  - Filter Manager
      The Filter Manager is responsible for managing which 
      filters are used by the hosts. The Filter Manager is 
      available at a "well known" location which is pre-
      programmed into the hosts. The Filter Manager is 
      responsible for creating and managing the other 
      components of the filter system.
  
  - Main Filter
      The Main Filter is the single point that links back 
      into the CORE of the system. It will connect to the 
      DBI and the CLI to deliver data.
  
  - Filters
      There can be multipler Filters, and these are the 
      "front line" to the hosts. They all link back to the 
      Main Filter to send data into the system. It is 
      possible to run these Filters on any machine, allowing 
      management of data flow.

At startup a Filter Manager object is activated at the "well 
known" location (probably a given machine name at a 
predefined port). The Filter Manager will create an instance 
of the Main Filter, and any Filters under it's control. 
Through some mechanism the other Filters, elsewhere on the 
network, will register with the Filter Manager. The 
Filter Manager will need to tell each Filter the location 
of the Main Filter upon registering. The Filter Manager will 
then be in a position to receive connections from hosts and 
pass them off to Filters.

System Running State
********************

CORE
----


Client Interface
----------------


Filter
------
When a host first loads up it knows where to locate the 
Filter Manager because it's located at a "well known" 
location. The host will fire up a TCP connection to the 
Filter Manager to announce itself. The Filter Manager will 
use some method (logically) to allocate a Filter to the 
host. The Filter Manager should base this decision on 
various factors, such as the load on the selection of 
filters, and possibly the location in relation to the host. 
The host will then be directed to this Filter for all 
further communications.

As the system runs the host will send data with (maybe) UDP 
to the Filter (that it's been allocated to). This choice has 
been made because it puts less onus on the host to make the 
connection, rather the data is just sent out. However, to 
ensure that the data isn't just disappearing into the depths 
of the network a periodic heartbeat will occur (at a 
predefined interval) over TCP to the Filter. This heartbeat 
can be used as a form of two-way communication, ensuring 
that everything is ok, and if required, to send any 
information back to the host. This heartbeat must occur 
otherwise the server may infer the host has died.

This could link in to alerting. An amber alert could be 
initiated for a host if the server stops receiving UDP 
packets, but an red alert be raised if the heartbeat doesn't 
occur.

If, for some reason, the Filter were to disappear the host 
should fall back on it's initial discovering mechanism - ie. 
contacting the Filter Manager at it's "well known" location. 
The host should report that it's lost it's Filter (so the 
Filter Manager can investigate and remove from it's list of 
Filters), and then the Filter Manager will reassign a new 
Filter to the host. Communication can then continue.

The idea of plugins to the Filters has been introduced. 
These plugins will implement a predefined plugin interface, 
and can be chained together at the Filter. Using the 
interface we can easily add future plugins that can do 
anything from parsing new data formats, to implementing 
encryption algorithms. The Filter will pass incoming data to 
each plugin in turn that it has available, and then finally 
pass the data on to the Main Filter. The Filter need not 
have any real knowledge about the content of the data.
Revision:	1.2
Committed:	Mon Oct 30 19:38:34 2000 UTC (24 years ago) by tdb
Content type:	text/plain
Branch:	MAIN
Changes since 1.1:	+86 -11 lines
Log Message:	Completed the section on the Filters.
#	Content
1	I-Scream Specification Outline (Realtime side only)
2	===================================================
3
4	ajm4, 30/10/2000
5	tdb1, 30/10/2000
6
7	System Component Startup
8	************************
9
10	CORE
11	----
12
13
14	Client Interface
15	----------------
16
17
18	Filter
19	------
20	The filter is broken down into three main subcomponents.
21
22	- Filter Manager
23	The Filter Manager is responsible for managing which
24	filters are used by the hosts. The Filter Manager is
25	available at a "well known" location which is pre-
26	programmed into the hosts. The Filter Manager is
27	responsible for creating and managing the other
28	components of the filter system.
29
30	- Main Filter
31	The Main Filter is the single point that links back
32	into the CORE of the system. It will connect to the
33	DBI and the CLI to deliver data.
34
35	- Filters
36	There can be multipler Filters, and these are the
37	"front line" to the hosts. They all link back to the
38	Main Filter to send data into the system. It is
39	possible to run these Filters on any machine, allowing
40	management of data flow.
41
42	At startup a Filter Manager object is activated at the "well
43	known" location (probably a given machine name at a
44	predefined port). The Filter Manager will create an instance
45	of the Main Filter, and any Filters under it's control.
46	Through some mechanism the other Filters, elsewhere on the
47	network, will register with the Filter Manager. The
48	Filter Manager will need to tell each Filter the location
49	of the Main Filter upon registering. The Filter Manager will
50	then be in a position to receive connections from hosts and
51	pass them off to Filters.
52
53	System Running State
54	********************
55
56	CORE
57	----
58
59
60	Client Interface
61	----------------
62
63
64	Filter
65	------
66	When a host first loads up it knows where to locate the
67	Filter Manager because it's located at a "well known"
68	location. The host will fire up a TCP connection to the
69	Filter Manager to announce itself. The Filter Manager will
70	use some method (logically) to allocate a Filter to the
71	host. The Filter Manager should base this decision on
72	various factors, such as the load on the selection of
73	filters, and possibly the location in relation to the host.
74	The host will then be directed to this Filter for all
75	further communications.
76
77	As the system runs the host will send data with (maybe) UDP
78	to the Filter (that it's been allocated to). This choice has
79	been made because it puts less onus on the host to make the
80	connection, rather the data is just sent out. However, to
81	ensure that the data isn't just disappearing into the depths
82	of the network a periodic heartbeat will occur (at a
83	predefined interval) over TCP to the Filter. This heartbeat
84	can be used as a form of two-way communication, ensuring
85	that everything is ok, and if required, to send any
86	information back to the host. This heartbeat must occur
87	otherwise the server may infer the host has died.
88
89	This could link in to alerting. An amber alert could be
90	initiated for a host if the server stops receiving UDP
91	packets, but an red alert be raised if the heartbeat doesn't
92	occur.
93
94	If, for some reason, the Filter were to disappear the host
95	should fall back on it's initial discovering mechanism - ie.
96	contacting the Filter Manager at it's "well known" location.
97	The host should report that it's lost it's Filter (so the
98	Filter Manager can investigate and remove from it's list of
99	Filters), and then the Filter Manager will reassign a new
100	Filter to the host. Communication can then continue.
101
102	The idea of plugins to the Filters has been introduced.
103	These plugins will implement a predefined plugin interface,
104	and can be chained together at the Filter. Using the
105	interface we can easily add future plugins that can do
106	anything from parsing new data formats, to implementing
107	encryption algorithms. The Filter will pass incoming data to
108	each plugin in turn that it has available, and then finally
109	pass the data on to the Main Filter. The Filter need not
110	have any real knowledge about the content of the data.