This section provides a brief, non-technical overview of the technologies we employ. It is just an outline of what is under the scenarios in another section. We have a separate glossary that provides thumbnails for certain buzzwords. Many of those explanatory paragraphs link to book and web resources. Some links are also at the bottom of the page.
Of all the pages in this section, this is the only one not targeted at a completely nontechnical reader. We refer you to the scenarios and to the nontechnical glossary.
Please refer to the drawing below which shows both the major functions and the code blocks that implement them. The functions in the diagram represent three main, separable research projects. Items 1 and 2, supported by 3,4, and 5, are the Federation Mechanism which is an advanced idea in the Enterprise Integration space. Items 2 and 8, supported by 9 and 10 are an advanced concept visualization environment. Items 2, 6 and 7 constitute a means of performing advanced analyses on huge information pools to seek concept clusters. Each block, as you can see, depends on the federated, abstract model, a sort of novel representation scheme. Each block is described below.
Block One: Federating Disparate Information Pools
Items 1 and 2 constitute a specific approach to Enterprise Integration, one that was developed with international guidance under ARPA sponsorship. It takes disparate data pools which are physically distributed, use different representation paradigms, and are at differing levels of correctness and completeness. This situation often exists when previously unrelated businesses affiliate to produce a complex product. The job is a hard one, to make disparate models appear unified. Also, the EI community has defined some additional properties for this federated model to act at their “level 5” capability: it needs to support fine-grained intelligent agents; each process owner (at a level of granularity they define) must be able to see both
- measures of how well the enterprise satisfies its goals, the effect its own process has on those goals, and a means of indicating how to change itself to improve those measures;
- a similar capability where one process can see how other processes contribute to the goals of the enterprise, and how combinations of itself and them can change to improve those measures. This is the goal of an agent-led local systems optimization process with presumably hundreds or thousands of "improvement agents" working in concert.
There is no way to support a Federation Mechanism like this without pulling a few tricks, and the main trick we pull is to have all relationships be “lazy,” meaning that they are evaluated only when needed and only the parts that are needed. This effectively means that instead of fusing and normalizing information types (like data elements) when they are encountered, we do it locally and only when needed. In other words, we don’t immediately take the gazzillion atomic pieces of information and evaluate each against the bazzilion conceptual relationships in our world. That’s too huge in terms of figuring. Instead, along with every atom of information, we store all the potential interpretations and relationships (and other questions about) that atom. It trades the complexity of figuring things out for the complexity of representing all the things we need to figure out maybe later. We have some tricks in Item 2 below that make this manageable.
Within this box, what we do is somewhat unusual, but it is done by mostly conventional means. Every representation method needs to have been preregistered with the standard ontology. This is the only onerous part of the process, but one expects that soon this will be done as a matter of course. What this means is that every system one can use to represent and store information has described the elements it uses and the way they behave in a standard, formal way. We call this an ontology, shown as Item (3)—a standard technique for discovering how elements might relate. The two novel things we do here are:
- Instead of translating the disparate atoms into some common format, we keep them as they are (in a conventional database) and add all the extra information for later. As we noted earlier, this trades the huge problem of figuring things out to (at least temporarily) the problem of the now massive information base;
- Ontologies work best when you have a target world. The standard use is for translating a source, or many source representation systems into another. Or alternatively, your reference system can be a body of known facts: the Cyc project, in which case the federation is an interpretation system. We use a reference that is a rich set of myth and metaphor (Item 5) as the reference world. This means that each atom not only has information about how it might relate to everything else in the source world (in our scenarios this is a business enterprise or intelligence pool) but it also has information about every possible link to elements in the metaphoric world.
The implementation is easy. You need a language with lazy types and late binding (actually the ability to store intermediate unbound states of objects), speed and links to KIF, the language of ontologies. Dylan or Python will work when lazy evaluation is kludged by exploiting the storage of partial dynamism. Haskell is more natural.
You will note that the big soup of this model is the grand central station for the three big new-capability-chunks. This is just a data pool — the web will do just fine using XML because in practice, we do not store anything that already exists, we just store labeled pointers. So every atom that exists in one of the diverse source information pools is just pointed to. Similarly, we point to ontology entities that are stored separately, in fact in many ways is treated much like just another source information pool. The “model base” itself uses mundane technology unless you consider the web/XML advanced in some way.
The ontology is a standard device. We use PSL as a foundation because it is the most formal of the major ones, is crafted as a standard, and employs situation semantics as a theory of state (very important). As with all ontologies, the normal expression is in KIF. The technology is mundane, but we have included some notes on the approach.
The Enterprise Rationalizer (4)
A function not central to our project, but critical to running the enterprise. Later on, you’ll see how we can find relationships in this soup. Some of those will be relationships which resolve outstanding federation issues. For instance, we may detect when an entry in one of the source databases or models is almost certainly incorrect. Or we can deduce that two models describe similar things, though they are unaware of the similarities, for instance we may discover that the processes for delivering newspapers has the same patterns as the processes for designing drugs. All this information should flow back to the clients, so that at the intermediate level, the federation distance is closer, and at the source level value is added. The heavy lifting is done in the next block. Item 4 is simply a report mechanism that partially evaluates lazy expressions, resolving some outstanding issues. Any evaluation technology can do this. The technology is straightforward. Haskell is used for implementation simplicity.
The Reference Metaphor Structures (5)
This is a little tricky. To support the visualization block, we use a reference based on some deep mythological, cosmological or cultural structure. This is a truly novel concept. GeoKabbalitter1 is an example of the type of thing we have found useful. The recording technology is simple, XML-based. What’s difficult is scoping the science of what adds value in the sense of cognitive structure, and what adds noise. The difficulty is that any metaphoric reference system is only leveraged after some significant acclimatization by the user, so the cost of discovering usefulness of different systems is high. We currently have no rules to predict this. But age and sex seem not to matter so much as handedness and musical ability. Attention span can be measured, and the ability to focus is a factor, but the system can be adjusted for many superficial clusters or fewer deep ones. So the representation technology is mundane, but the “fitting” technology is non-existent so far.
We structure this information according to the lattices of Item 2, but no technology is used as that work is hand-done at present.
Block Two: Creating Complex Concept Clusters
This is novel. We use an ordinary database (Item 5) to store all of our functions and lazy expressions. But the data model is built using a (simple entity-relationship based, though the object-actor paradigm works as well) methodology of a small grammar of symmetric primitives. These are sufficiently abstract to serve as their own metagrammar. The models are a form of regular, periodic (acyclic hetarchic) graphs known as concept lattices. extended for multidimensional graphs. Haskell is used.
The idea is to translate each concept atom into an element of a graph with a spatial regularity, thus creating a vectorized concept space. Since the graph has a topology, we can abstract the vectors into fiber bundles to discover clusters. A module within the clusterer manipulates the topology (symmetry) and dimensionality of the lattices to optimize clustering density. The lazy expressions survive the abstraction and are evaluated late, because the representation grammar is fully reflective at any level. Some clustering is by rule, some by agent and some directed by the human through Items 7 and 8.
We need a high order term rewriting grammar to rewrite the graph model, simplifying and evaluating expressions. Our primary goals in doing so are to resolve outstanding issues in the federation, and to inference about unknowns. In the first case, we are (perhaps partially) evaluating expressions, rewriting abstract terms; in the second case, we are evaluation the soft variables in the situation calculus. As a front end to our Haskell functions, we use something like HOPS.
Block Three: Visualizating and Directing Complex Concept Clusters
The Lattice Visualization Environment (8)
The display elements are not essential to the working of the other blocks, except to direct item 7. If all the heavy lifting is done in the components above, then all we need is a (OpenGL) 3-d display system that is capable of multimedia-type popups, labels and links. But we want to be able to have a sliding boundary between the vizualization client, so we prototype the client system in Mathematica/OpenGL. Mathematica can be used as a non-strict functional language (meaning with lazy evaluation). Here we display 3-d (or with hyperstereograms, 4-d) projections of the lattices in an immersive situation. (The view is described in the scenarios.) The only interesting technologies here are when the division between (web) server and client is deep into Item 6. That is, some of the clustering (say that done by rule and agent) is done at the server in the Haskell environment and some (say that initiated by the user) at the client. This would be very tricky if the representation of the state of the environment were not explicitly represented along with the state of the world being represented. It is not concise, but there is an XML-based state protocol to synchronize resolutions and abstractions between the two systems. Again, situation theory is employed.
One navigates the concept lattice as if it were a three dimensional fabric. Concept clusters are indicated by physical proximity. Different topological abstractions will result in different attractors for clusters, but specific cues are needed to help the navigator perceive a cluster. Often clusters have similar structure so represent a pattern of clusters of a metacluster. The conventional cues are by colors and by “constellations:” eigenvalue figures which characterize clusters. But a statistical recognition tool is used, based on Edgewarp, a visualization enhancement technology. The “focuser” simply toggles the “grid edge geometry” view rather than the lattice-color view. The idea is quite sophisticated, based on research on recognizing statistical patterns.
The Cosmological Preference Filter (10)
The ontology (Item 3) is open in the sense that it can accommodate many different world views, presumably all those likely to be used in the source information pools. An example is different views of time, whether discrete or continuous, whether causal or coincidental. Some other examples reflect more fundamental differences in worlds. The browsing user must have a world view, and therefore a commitment among these choices. But the concept representations and their clusters change depending, literally, on one’s viewpoint. Therefore the preference filter allows one to assume different perceptual garb. The technology is mundane, just a reorganization of the order in which the lazy elements are evaluated.
Links
<PREVIOUS TO SCENARIOS> <OLD HOME> <FAR OUT HOME> <NEXT TO CONCEPT LATTICES>
