I reply to Latimer's (1999) criticisms by trying to clarify some key features of the Hyperstructures model, discussing further some evidence in its favour, and explaining its ontological status. I acknowledge the specific instantiation of a hyperstructure in the neural net described by Schmidhuber (1999).
2. To start with, it is worthwhile recalling my main premise that cognitive systems evolved to deal with highly changeable environments so that moment-to-moment experiences of any object will tend to be both highly degenerate and variable -- novel in orientation, distance, speed and direction of motion, and partially occluded -- with little of the consistency required by a feature-matching system. The only information that remains consistent in such experience, I argue, is the 'deep' covariation structure, or hyperstructure, describing an object's spatio-temporal transformation over time. Although each sensory experience of an object will be unique, and often 'fuzzy', its hyperstructure will tend to remain characteristic. For example, when the millions of light points in a person walking are reduced to as few as ten (as is done experimentally with the classic point light walker) a characteristic covariation structure persists within them.
3. Accordingly, in my view, recognition/classification or other forms of predictability are only made possible by (i) the internalisation over time of the covariation hyperstructure characteristic of each experienced object (through some 'attunement' to them in internal neural networks); and (ii) the activation of one or other of these internalised hyperstructures by the 'sample' covariation structure in current input (which, again, will tend to be characteristic for the source object). This interaction excites expected 'missing' values in the internal network, corresponding with the deficiencies of the input. In this way, a 'superimage' of subfeatures (and ultimately features, parts, objects, and action schemas) is constructed. It is these constructs which lead to recognition, predictions for action, and naming.
4. In this scheme, 'local' and global' are reciprocally associated. The local properties are the changing sensory values (which, for convenience, I simply call light-points) and the global properties the covariation structure they exhibit between them. The global properties are also internalised over time and used to generate further 'local' values in the formation of internal superimages. What we actually 'see' emerges as a construct from the interaction between local and global. (As mentioned in the target article, when presented with a perfectly static visual input, shorn of normal covariation structure, we apparently don't 'see' anything.) Of course, in a system evolved for the capture of complex covariation, wherever it arises, hyperstructures will tend to form internally at various levels, corresponding with relatively rich covariation clusters in object experience. For example, the light points constituting the edge of a table will, as we move around it, exhibit a characteristic hyperstructural pattern. Over numerous experiences, that hyperstructure will become internalised. Even when partially occluded by another object, an image of that subfeature can be constructed from the kind of interaction just mentioned. The corner of the table will create another hyperstructure at this sub-featural level. However, the light-points from the edges and corners (as well as the surface) will, together, exhibit a hyperstructural pattern at an additional (featural) level, that of the table top. Again this feature can be recovered even when largely occluded, or seen only fleetingly, or from a novel direction or distance, by virtue of the internalised ('featural') hyperstructure. Similarly, a hyperstructure at the 'whole-object' level is internalised from the covariation between sensory values arising from different features (e.g., the top and the legs). And so on, through other levels of covariation experience, such as multiple objects in action schemas, and human cooperation.
5. So I argue that what is captured in the internal networks consists of 'hyperstructures of nested hyperstructures', and even 'hypernetworks'. These mature forms develop (ontogentically) so that what a neonate will 'see' first will be constellations of raw sensory values and, perhaps, simple subfeatures like lines and edges, the hyperstructures for which might be expected to develop very rapidly. Only later will the infant be capable of comprehending features, whole objects, relations between objects, and so on, in a sequence broadly like that described by Piaget (e.g., 1954)
6. The value of this nesting of hyperstructures in progressively more inclusive ones is that it further facilitates (super)image construction, and thus predictability. In some cases it may be possible to construct a whole-object (super)image from the 'bottom-up', from a glimpse of covariation at the subfeatural level, such as the corner of a table protruding from a stack of furniture. In others, the whole process may operate in reverse. For example, we can create an image of 'most likely' objects and features from very skimpy covariation between light points at the level of a 'social' scenario (e.g., a game of tennis portrayed by a few moving points on a computer screen). More likely, as with the point light walker, hyperstructural activation will be going on at, and across, several levels simultaneously. This is quite different from either 'direct' detection of 'global properties' or their convergent derivation from 'local analysis' as Latimer (para 5) suggests.
7. Note that nowhere is there an internal analogical representation (such as a template or prototype) of any subfeature, feature or object, only a hyperstructure acting as a kind of internal grammar for their 'on-line' construction, in the way that language grammar guides the completion of an incomplete word or sentence. (Indeed, I suspect that human language is a special case of such nested hyperstructures operating over acoustic stimuli). The only real primitives throughout are the raw sensory values and their changes over time: all other images, including 'features' are constructed from interaction between these and the internal 'nests' of hyperstructures built up from experience.
8. One hopes this helps meet Latimer's complaint about the model not being clearly specified. Ironically, the notion of covariation hyperstructures is quite compatible with the 'dependence relations' advocated at one point by Latimer, and I'm only sorry that this isn't clearer in the target article. As for being 'testable', I suggest that this takes two forms: first, reconciling or explaining prevailing puzzles; second, confirming novel predictions or hypotheses. As regards the first, the hyperstructures model corresponds with the sometimes puzzling architecture of the brain, consisting of richly interacting feed-forward, feed-back loops, and horizontally interconnected layers: in sum, just the arrangement for 'hypernetwork' formation. It also, in my view, explains some classical puzzles prevailing in the literature about concept structure: for example, how a concept can seemingly involve information at the 'exemplar', 'prototype', and 'theory' levels simultaneously.
9. As for novel hypotheses, we predicted that recognition of extremely degenerate inputs from 'well-learned' objects like a person walking would depend on the complexity of covariation structure in the input, and this was confirmed with the use of highly degenerate point light stimuli. We also predicted that, as children develop, their conceptual representations of common events (and their increasingly accurate predictions from them) would reflect increasingly complex covariation structure, and this was also confirmed. Another prediction arising from the model is that infants will be capable of the abstraction of complex covariation structure from a very early age. This is, indeed, my interpretation of Marcus et al's (1999) demonstration of the abstraction of complex 'algebraic rules' in seven-month-old infants. As Pinker (1999: 41) puts it in discussing that work, it 'suggests that the ability to recognise abstract patterns of stimuli that cut across their sensory content is a basic ability of the human mind'.
10. It was very helpful of Schmidhuber (1999) to draw my attention to his specific example of hyperstructural abstraction in a neural network. This, of course, reinforces the suspicion expressed in my paper that this is what connectionist nets do generally. But I don't agree with Latimer's view that this necessarily demonstrates that 'global properties...are clearly derived from the products of activation from discrete inputs' corresponding with features. My point (and that of Schmidhuber) is that these couldn't be discrete because the inputs are never the same.
11. It is, however, quite reasonable for Latimer to wonder about the ontological status of my claim about hyperstructures in relation to connectionist systems. Though largely an aside in the target paper, my point is that, if those systems claim to model aspects of human cognition, we need to be clear how they do so - a question that has been neglected in the past. Hyperstructural analysis may help direct us to critical aspects of the informational structure of experience, perhaps sometimes indicating special difficulties, or errors, and to take remedial action accordingly. For example, I have suggested elsewhere that conditions like human dyslexia and autism may stem from inadequate or erroneous hyperstructure abstraction. This has the same ontological status of a model of the kidney, say, which may inform us that it doesn't just 'cleanse' body fluids, as an input/output device, but does so through filtration and active reabsorption, which in turn helps us a great deal in understanding the kidney, and assisting its function (with dialysis machines) when necessary. Such hyperstructural analysis will of course be formidably complex - but, then, so is the cognitive system.
Piaget, J. (1954). The construction of reality in the child. New York: Basic Books.
Latimer, C. (1999). Abstract ideas, schemata and hyperstructures: plus ca change. Commentary on Richardson on Hyperstructure. PSYCOLQUY 10(040). ftp://ftp.princeton.edu/pub/harnad/Psycoloquy/1999.volume.10/ psyc.99.10.040.hyperstructure.3.latimer http://www.cogsci.soton.ac.uk/psyc-bin/newspy?10.040
Marcus, G.F., Vijayan, S., Rao, S.B., & Vishton, P.M. (1999). Rule learning by seven-month-old infants. Science, 283: 77-79.
Pinker, S. (1999). Out of the minds of babies. Science, 283: 40-41.
Richardson, K. (1999). Hyperstructures in brain and cognition. PSYCOLOQUY 10(031). ftp://ftp.princeton.edu/pub/harnad/Psycoloquy/1999.volume.10/ psyc.99.10.031.hyperstructure.1.richardson http://www.cogsci.soton.ac.uk/psyc-bin/newspy?10.031
Schmidhuber, J. (1999). Extracting predictable hyperstructure. Commentary on Richardson on Hyperstructure. PSYCOLOQUY 10.(034). ftp://ftp.princeton.edu/pub/harnad/Psycoloquy/1999.volume.10/ psyc.99.10.034.hyperstructure.2.schmidhuber http://www.cogsci.soton.ac.uk/psyc-bin/newspy?10.034