Avatars, Agency and Performance: The Fusion of Science and Technology within the Arts
Richard Andrew Salmon 2014
5. Setting the Scene:
The Performance Arts Context and Nature of the Investigative Terrain
5.1 From the engineering – to performance arts perspective
The development of the Articulated Head project could so easily have progressed predominantly from a robotics and engineering perspective.
Many aspects of the project did develop this way but the Thinking Head project owed its inception to Stelarc’s Prosthetic Head. Stelarc, who is first and foremost a performance artist, exerted his influence over the evolutionary stages of the Thinking Head Project from the original Prosthetic Head through to the Articulated Head. This influence both initiated and maintained a strong performance art dimension to the works, which might otherwise have been ignored. Indeed, the title and content of this thesis acknowledges the importance of the artistic dimension to these works, both in citing the fusion of science and technology within the arts in the title, and then presenting and testing creative new media propositions with the intention of addressing the big question: how can interaction between humans and machines be improved? (particularly from a performance arts perspective).
5.1.1 The positive engineering perspective
The big question can so easily be interpreted in terms of a massive engineering challenge. That is, how can we make the robot more sophisticated? Sophisticated enough that humans move beyond the perception of merely talking to a machine, to a perception of having an empathetic relationship with the conversational agent that inspires an awareness of the head as an agent, one that is both engaging and has agency. One can just imagine an engineer’s planning meeting wish list: give it more inputs, more sensors, more sensitive apparatus, finer movement mechanisms, more processing power, more memory, better, smaller, lighter and faster electronic and mechanical components – but this approach brings with it a multitude of complications. Teams of engineers and programmers would be required to tackle all of the developments present on the wish list. Each development installed would likely manifest in another set of behaviors presented by the machine to an interacting audience.
Some of the emergent behaviors may be accidental, un-planned and happily beneficial to the human . machine interaction in some way or another, whereas other emergent behaviors may be less helpful and require further engineering to address and suppress these unhelpful manifestations.
Most importantly, the above wish list (hypothetical or otherwise) implies that the machine is the thing that needs changing; that development of the machine [is] the key object of focus in order to answer the big question. That is, the suggestion is that through an exponentially expanding wish list and a sustained engineering approach, coupled with a trial and error development of the machines capabilities, improvements to human . machine interaction will somehow eventuate.
5.1.2 Questioning the approach.
However, an engineering approach is unlikely to be the correct one that would satisfy all of the issues raised in this thesis in their entirety. For whilst it is fair to assume that modifying the machine might bring about improvements in the flow of human . machine interaction, one must consider all aspects of the performance that is taking place in interaction, the human and the spatio-temporal environment in which the interaction is taking place as well.
Therefore, the engineering view outlined above (positive though it maybe) is too narrow to capture the domain of human interaction; it overstates the mechanistic by seeing all problems as stemming from input/out limitations. In short it over-generalizes what can be solved by brute force.
“Too large a generalisation leads to mere barrenness. It is the large generalisation, limited by a happy particularity, which is the fruitful conception.” (Whitehead, 1997, p. 176)
5.1.3 The negative perspective
The opposite perspective to the positive (but possibly over-general view above) might be a view that the Articulated Head will always be just a computer-controlled robot/avatar. That is, given that any convincing avatar performance is so drastically constrained and limited by the current state of technology in areas such as automatic speech recognition, and in how our abilities to glean, analyse, store and represent useful data regarding aspects of the audience (e.g., a person’s sex, mood, dress, health or even altitudinal disposition) might be mimicked, and given that sensing technologies and databases are likely to remain under developed, then it could be concluded that these restrictions effectively presented insurmountable obstacles to “conditioning affect and engagement” so that the venture is unworthy of effort as it is hopelessly futile. This somewhat negative, some might say cynical (though some might also say realistic) view would definitely lead to “mere barrenness” for this investigation because not to attempt to establish how one might achieve enhancement of performance in this human . machine interaction, simply fails to address the big question.
5.1.4 Of positive thought, critical evaluation and leaps of faith
Both the positive and negative views towards improvement of human . machine interaction projected above held some merit in terms of fact and prognosis, though both have also ignored very important aspects of the context within which the big question is asked. For example, the positive view projected towards machine development completely ignores possible contextual development in the environment surrounding the robot and also fails to acknowledge the importance of the human in the interaction environment. The positive view simply represents a leap of faith that everything will be solved by technological developments; yet these developments may well lead to a snakes and ladders period of protracted work that ultimately fails to make progress because some of the new and intended improvements made to the machine may actually cause more problems than they solve in terms of performance (a case of one step forwards two steps back!). On the other hand, the negative view, whilst possibly projecting an accurate assessment of current technological achievements and the barriers to development, completely ignores the future technological horizons that may occur. Indeed, if one never makes the attempt then one will never know the precise nature of the problems that need to be overcome.
A critical surveyor of possible project developments provides a more tempered view of both the current status of the project and the scope for future developments. This tempered view is strategically useful for formulating an overview for project management. Whilst a positive disposition might be attractive and make one popular amongst the project development team in the short term, and faith might occasionally lead to fruitful conceptions, it is much more likely that carefully considered critical evaluation will lead to successful outcomes and by so doing make more concrete steps forward in determining how human . machine interaction can be improved in this and similar interactive environments.
5.1.5 A melioristic yet tempered view towards development
Broadly speaking, a melioristic view towards development of the machine and interactive environment in which the human plays a very major part offers a viable pathway to adornment, embellishment and agency along with the possibility that the machine’s performance may have a conditioning affect that will help interacting and engagement with its audience. This positive view emphasizes many small but developmental steps in the direction of audience engagement and so should ultimately enhance performance within the domain of this human . machine interaction.
However, before charging ahead with even small developmental steps, it is important to first formulate a clear overview of the interactive environment under investigation and then subsequently determine which project developments might prove to be fruitful in terms of improving human . machine interaction. This thesis is more concerned with trying to establish what developmental steps make sense and why, than it is in making the developmental steps and testing them, (i.e. the concern is to engender a process of emergent discovery rather than hypothesis testing).
Furthermore, this investigation is less concerned with giving credence to current technological barriers and constraints, choosing instead to place more focus on the development of arguments for determining which technological barriers must be overcome in order to achieve improvement of this human . machine interaction. Thus, this investigation projects a positive view towards future developments in human . machine interaction, based on a tempered critical analysis of human centered empirical Video Cued Recall Interview evidence collected. The evidence then leads to a carefully considered and critically evaluated recommendation for future developments (including technological leaps of faith) that are expected to prove successful in terms of improving this human . machine interaction.
The emergent theories and ideas that stream from this investigative work are based on sensible constructed and substantiated arguments that are linked with happy peculiarities of the interactive environment under investigation and are expected to lead to fruitful conceptions when implementing the recommendations with similar interactive exhibits.
5.1.5.1 The happy particularity with regards to this investigation
The principal “happy particularity”(Whitehead, 1997, p. 176) with regards to this investigation, which was identified through analysis of empirical research data collected, presented and discussed in more detail in Section 7 under Observation 1 – The anthropomorphic stance, was that the audience interacting with the Articulated Head appeared (without exception) to be predisposed to adopting an anthropomorphic stance toward interactions, when first approaching the Articulated Head. The audience appeared predisposed to the perception of the avatar as a conscious entity, indeed they acted and engaged with the Articulated Head as if it was an existential being. I will elaborate on the anthropomorphic stance later, before this I will describe elements in the environment that might help condition this.
5.1.6 The influence of the spatio-temporal environment
To help illustrate the importance of a spatio-temporal environment and its influence in performance more clearly, I am asking the reader to consider the following scenario as something potentially parallel to the Articulated Head’s interactive performance:
5.2 The analogous performing arts scenario: The Stage
My colleague, a lecturer of Performing Arts and a thespian might play Long John Silver in Treasure Island on the theatre stage, where his convincing performance, the costumes and the stage set, touch the audience in such a way as to make them forget, for a short time, that what they are actually watching is really just an act – the performance draws the audience into the story in such a way that it is real – he is a pirate for the time in which the performance is taking place, I argue that so too must the robotic exhibit’s performance, costume and staging touch its audience and seductively draw them in. Just as the audience might leave the theatre following the performance of Treasure Island, saying, the actor was a convincing pirate, so too must the performance of the Articulated Head elicit such a complimentary commentary. If I find such a reflection from the audience in the Video Triggered Recall Interviews after interaction with the robot has taken place, I would deem it a success.
For a human to assimilate the characteristics and narrative of another human’s story, and to project this convincingly on stage, takes considerable discipline and practice, along with attention to detail to the art of acting, costume and set design. To convince a human that the avatar, the Articulated Head, was a conscious entity, one which acted with autonomy and intent would be a similarly difficult illusion to create – and one especially difficult to sustain.
The goal of this study and accompanying creative work (see section 6), was not necessarily to make the avatar more humanlike, nor was it to make the Articulated Head overtly theatrical but to find ways in which it could produce the practice of agency so that, at the very least, it appeared to act with autonomy and intent. This is the crux upon which this study evolved, a pivot that led to both the direction and the research methodology employed as well as shaped the development of the creative additions that were proposed and tested.
An audience perception of the avatar as a conscious entity, which acted with autonomy and intent was conceptually wrapped in the necessity to persuade the audience that the avatar possessed consciousness.
Drawing parallels between the theatre stage and the performance of the Articulated Head was both intentional, and not insignificant to this project.
However good an actor my colleague might be, the removal of his costume,
his wooden leg and the sound of its thumping on the stage floor, all the elements of the stage set, the boat, the island, sand, palm trees, the sounds of water and wind and the treasure chest itself, would likely weaken the pirate illusion and inhibit the audience’s propensity for suspension of disbelief in the character being presented to them by the actor.
That is, environmental auditory and visual contextualizing cues, when carefully constructed in relation to the performance of Long John Silver as detailed above, not only condition affect and engagement but also act as catalysis to the perception of a ‘real’ narrative consciousness within the actors portrayal of Long John Silver – the part and performance, as perceived within the minds and consciousness of the audience watching it.
The environmental cues, when carefully constructed in relation to the avatar, the Articulated Head, should similarly have been able to not only condition affect and engagement, but should also have been able to act as catalysis to the perception of a ‘real consciousness’ of the avatar within the minds and consciousness of its audience, in the proceeds of their interactions.
Careful attention is drawn to the critical conceptual point made above, as it represents a clear rationale and motive in relation to the proposed creative new media additions.
5.2.1 Illusion and Conscious Perception
Illusion and conscious perception are brought to the fore. After all, with reference to our ‘theatre stage scenario’ detailed above, it is very clear upon reflection that all the elements of the stage set, the boat, the island, sand, palm trees, the sounds of water and wind – are all essentially falsely situated within the wider context of our existence and the layers of our experience.
It is likely that most of the stage elements are not even ‘real’; the majority of the scene may be represented as a painted backdrop. This however does not stop these elements from exerting a contextualizing influence upon the act, or from being brought to bear upon the minds and consciousness of the audience watching the show. Neither do these contextualizing elements detract from the centre of attention on the stage, being the actor in this case, Long John Silver. In fact, very much the opposite, they are the catalyst, which cause the catalysis of the perception of a real ‘other’ within the consciousness of the audience to be realised. Therefore, the contextualizing influence of auditory and visual cues surrounding the avatar and its audience should also have proved to be just as powerful a tool, without detracting from the star of the show, being the Articulated Head at the centre of the interactive installation
5.2.2 The importance of what is said and what is not
As previously stated, the importance of the human in the human . machine interaction is of central concern throughout this investigation. It is the human that is the rationale for why this study prioritises human experience and asks the end user (humans) what they actually think and feel during interactions, what matters to them and indeed what does not. Furthermore, there is as much to be gleaned from what participants interacting with the Articulated Head did not say in Video Cued Recall Interviews as there is to be gleaned from what they did say. That is, if ‘something’, an observable characteristic present during interaction (a possible object of consciousness), was not mentioned as being a point of focus or raised in discussion of interactions, it may still play a role in the interpretive data analysis. Just because the participant did not mention it, that does not mean it was irrelevant. Indeed some unmentioned aspect of the interactive environment in which the interactions took place may be highly significant.
5.3 Investigative framework: Evidence, not hypothesis led
Any self-respecting and professional chief crime scene investigator would not enter into an inquiry with any assumptions about the perpetrator of the crime (even if there were some highly likely suspects) without first considering all the evidence and clues at their disposal very carefully, this investigation has similarly not been entered into with any specific hypothesis or assumptions about how human . machine interaction is perceived or can be improved. The investigation, its findings and its conclusions are research data [evidence and clue] led, not researcher or hypothesis led. There was no attempt to prove a predefined theory or belief. This investigation has been conducted from the outset with this, and only this particular case in mind, the human . machine interaction that took place between the Articulated Head and an interacting audience in relatively unconstrained public exhibition spaces.
A section of text is included electronically as E-Appendix 10: The Analogous Road Traffic Accident Investigative Scenario. The E-Appendix contains some thoughts related to the nature of the investigative terrain encountered in this investigation. The treatment of circumstantial evidence and the impact that this might have on the investigations findings in relation to understanding and establishing how to improve the interactive environment is also considered.
5.4 Philosophical and practical justification of approach
Art influences science and technology just as science and technology influence art. The Articulated Head was an interactive installation artwork that incorporated a menagerie of technologies, many of which displayed functionality that was at the cutting edge of scientific technological development at the time. Popular perceptions and expectations of state-of- the-art science and technology can be exaggerated beyond reality, driven by ideas presented in science-fiction popular culture and other media.
5.4.1 The current state of play: Popular perceptions
In recent years, there has been an increasing focus in the media on the development of robots and robotic-type devices. For example, an article in the August 2011 copy of the National Geographic Magazine (NGM, 2013, p.
66) dedicated to the latest developments in humanoid robots, covered the work of some well-known roboticist such as Professor Hiroshi Ishiguro with his work on human-like robots at Osaka University, and the work of the LifeNaut project in Vermont with the robotic head called Bina48.
Description: rofessor Hiroshi Ishiguro (right) sits next to the Geminoid HI-2 robot, a tele-operated robot that looks exactly like himself in Kyoto, Japan. Ishiguro is director of the Intelligent Robotics Laboratory at Osaka University.
Figure 5-1 Professor Hiroshi Ishiguro (right) sits next to the Geminoid HI-2 robot
Professor Hiroshi Ishiguro (right) sits next to the Geminoid HI-2 robot, a tele-operated robot that looks exactly like himself in Kyoto, Japan. Ishiguro is director of the Intelligent Robotics Laboratory at Osaka University.
Figure 5-2 Bina48 an early demonstration of “mindfile” transfer at Lifeanaut.com.
The LifeNaut project and its social robot Bina48 are featured in a Vermont local newspaper. This is an excellent article about “mindfiles” and the purpose of the Lifenaut Project: (Robot blurs biological boundaries)
The National Geographic article mentioned projects a widely held popular public view about the current state of robotic development. The Magazine article opens with the statement:
“Robots are being created that can think, act and relate to humans. Are we ready?” (NGM, 2013, p. 66)
These are indeed bold proclamations, which seem to be ubiquitous amongst sci-fi fans and robotics researchers alike. One might somewhat cynically suggest that the increase in the frequency of such statements in these two groups is no accident, with the latter group being a subset of the former. On this view, media narrative and hyperbole, have led to an outlook on the likely progress of science and technology in relation to robotics that is ridiculously over optimistic (especially when viewed in relation to actual achievements). The latter of the two groups, appears to exaggerate the current prospective forecast of humanoid robot capabilities in relation to concepts such as thinking and building relationships with humans, simply because it makes a lot of sense to paint a very rosy picture, especially when trying to secure research grant funds from those who are probably less well informed about robots and are inclined to believe the publicity.
5.4.2 The role of the Arts in Robotics and vice versa
The above is not to say that the active imaginations of sci-fi fans, TV and film scriptwriters and alike have had only negative impact upon robotics development. Indeed, much the opposite may be the case; the Arts can and do lead scientists and technologists towards new visions and aspirations that are sometimes realized. Certainly the direction of scientific and technological developments is influenced and guided by artist’s aspirations and portrayals. There are numerous examples of artist aspirations and portrayals influencing scientific and technological developments from history, in Leonardo Da Vinci’s depiction of a vertical flying machine in the 1480s, which many now attribute as the first documented design of the modern day helicopter; –
Figure 5-3 Leonardo Da Vinci’s depiction of a vertical flying machine
– to more recent artistic influences on modern scientists and technologists such as the inspirational influence that Star Trek episodes (CBS-Entertainment, 2012) are said to have had upon NASA(Wilson, (n.d.)) scientists investigating rocket propulsion and space travel, and the statements of people like Martin Cooper (“Martin Cooper – History of Cell Phone,” 2011) who is attributed as the inventor of the cell phone. Cooper says that his inspiration for invention of the cell phone came directly from Star Trek Communicators.
Figure 5-4 A Star Trek Communicator device
5.4.3 The Tryst: The Science/Technology/Arts Nexus
It is here, that the Fusion of Science, Technology within the Arts proliferates. It is also here, in this interdisciplinary tryst, that strong philosophical debates surrounding constructionism, determinism, reductionism, interpretivism, epistemology, methodology, phenomenology, ontology, anthropology – (the list of ‘isms’ and ‘ology’s’ could be extended almost ad infinitum) transpires.
5.4.3.1 The contrasting methods debate
Escalation of philosophical debate in this Arts/Science/Technology nexus comes as no surprise because the seemingly opposing philosophical worldviews of practitioners and researchers in these domains become convergent – thus leaving scholars no choice but to contemplate the relevance and currency of converging philosophical and methodological viewpoints that are normally unceremoniously rejected by their own doctrine.
The point of the above observation is to emphasize the potential interdisciplinary nature of the Science/Technology/Arts weltanschauung, and to acknowledge the tensions between accepted methods of investigation in these differing domains. The purpose of this acknowledgement is to indicate how consideration of the differing worldviews and the differences between the scope of questions relate to human . machine interaction. Included in this is a consideration of the methods used to get at these questions. These points are relevant to the research detailed herein because of the scope of the big question: How can interaction between humans and machines be improved?
5.4.3.2 Human . machine interaction: Micro versus Macro
It is my view that the study of human . machine interaction (HMI) interaction, or human . computer interaction (HCI) as it is often referred to, fundamentally takes place at an intersection occupied by areas of Science, Technology and the Arts. Computer Sciences, Usability, Interactive Design, Electronic Design, Sound and Vision, Psychology and Behavioural Science, and a range of other fields of study all have a role to play in this intersection.
Psychology clearly falls on the human side of the human . machine relationship whereas Computer Science clearly falls on the machine side of this relationship. However, there are several fields that traverse both sides of
this relationship, Interactive Design, Sound and Vision, are three such examples that feature strongly in this thesis.
With regard to the methods adopted by some of the study area groups mentioned above in relation to the human . machine interaction under investigation here:
. Computer Science might ask questions such as: What impact does expansion of the conversational agents AIML coded vocabulary have? Does expansion of the conversational agents vocabulary correspond with a direct improvement in human . machine interaction? They might then select a specific number of words or text string responses to increase the Chatbot vocabulary by – and then set out to test the effectiveness of the expanded vocabularies introduction on the human . machine interaction in collaboration with an experimental psychologist by using an experimental testing paradigm of one form or another.
. The Experimental Psychologist might ask a question such as; has the new code improved human . machine interaction by extending the scope of dialogue exchange taking place between the Articulated Head and its audience? To find evidence of this they might test participant’s interaction, first without the new code – and then with, to see if factors such as conversational longevity in the interaction had been increased – or they may use targeted participant questionnaires asking participants to rate their engagement against a scale after interaction, to see if an increased level of engagement was reported by research participants.
Whilst each of the exercises mentioned above would probably contribute something useful towards knowledge about one particular micro dynamic aspect of the human . machine interaction and how it might incrementally be improved, that contribution would be very small in relation to the enormity of the big question. Analysis of the data collected may very well appear significant when represented in isolation but might actually be relatively insignificant or possibly (in some cases) irrelevant once adjustments for addressing some concern related to some macro dynamic aspects of the interactive environment has been considered.
This research project was conducted broadly speaking from an interactive designer perspective that closely resembles the view below.
. The interactive designer by contrast to the approaches adopted by the computer scientist and experimental psychologist mentioned above, might ask questions such as: what are the overarching features (macro dynamics) governing the way in which interaction between the Articulated Head and its human audience take place? How do these features contribute positively or otherwise to the flow of interactions? And how can the interactive design be rearranged to improve the flow of human . machine interaction?
5.4.3.3 Emergent Phenomena: Context and Causality: Micro versus Macro.
For the purposes of clarifying a particular point in this thesis – it seems best to declare my personal view that there is no clear divide between the domains of the Arts and Sciences other than (perhaps) a predisposition to be bias in favour of cause over effect or vice versa. I feel it necessary to declare this stance because I approached this research project with a strong performance, media arts and technology background – but with much less experience in the social and behavioural sciences such as psychology. The institute from within which this project was managed has a very strong experimental psychology academic tradition and I have encountered many experimental psychologists as a result.
Experimental psychologists in particular, subscribe predominantly to a reductionist methodology and as such often question other philosophical and methodological approaches such as those used in this investigation. In light of the practical considerations governing the methodological approach to this research to those who would question it, although some phenomena can be explained by reducing them to a more fundamental phenomena and breaking it into constituent parts (especially when these phenomena are perceived as emanating from a non-biological complex system), subscription to the philosophical view point that any complex system is simply the sum of its parts, especially with reference to biological beings seems somewhat harder to justify. Emergent phenomena do arise that can transcend the components revealed by any reductionist investigation – such that even if these phenomena do not escape an explanation due to a scientist’s determination to prove their cause, often that explanation seems superficial in relation to the wider context within which the phenomena transpired. That is not to say that causality is not important, it is – but this investigation is not about cause – but is about effect; it is focused on how human . machine interaction can be improved.
The minutiae of interactions are as transient, dynamic, versatile and spontaneous as the human mind that participates in them, and whilst the experimental psychologist might be predisposed to try and explain how the human mind works by employing reductionist methods, this investigation is not trying to do that. Trying to explain the cause of phenomena by reducing them down into constituent parts (what the scientist sees as the more fundamental phenomena that explain the epiphenomena) and then using an experimental psychology testing paradigm to find out facts in relation to the minutiae of interactions taking place in this investigation, would require a vast team of researchers and a protracted period of time, possibly several years, in order to collect and analyse sufficient data to draw sustainable conclusions. Therefore a more practical and strategic approach was adopted, one that had some chance of finding useful leads to possible answers to the questions posed in this investigation.
Focusing on the macro dynamics of interactions taking place within this context of human . machine interaction in the complex system under investigation, makes much more sense because the system and the interactive environment in all its complexity, seems to be irreducible (especially in the timeframe afforded this study) – and is inherently irreducible due to the human component, the relatively unconstrained and unencumbered environment in which the interactions were taking place, and variations between the human subjects involved in these interactions.
The human . machine interaction under investigation here, and the cause but (perhaps) not the effect of all the emergent phenomena in these interactions are enormously complex and possibly beyond the current scope of mankind to explain accurately in all their micro dynamically psychological, interpretative, physical, biological, chemical, and electrically detailed processes by reductionist methods. Furthermore, even if the reductionist- testing paradigm were able to describe every aspect of the minutiae of emergent phenomena in these interactions over a protracted period of time, its employ would have been inappropriate at this stage of the investigation into how one can improve human . machine interaction.
Why? Well, if epiphenomena are explainable by more fundamental constituent phenomena, as subscribers to the reductionist philosophy would claim, then it is reasonable to infer that epiphenomena wield no unpremeditated agency upon the phenomena that explain them. Similarly, it is reasonable to suggest that investment in understanding the macro dynamic arrangements of this human . machine interaction should take precedence over micro dynamic explanations of causality of phenomena. It is strategically wise to focus upon the macro dynamic design of interactive environments first – especially as any move to make macro dynamic rearrangements to the interactive environment in order to improve the flow of interactions, might easily render investigation of some aspect of the micro dynamic interaction superfluous to the investigation in the changes wake.
Thus a micro over macro dynamic focus to this investigation would represent a simple, yet fairly clear case of ‘can’t see the wood for the trees’.
Section 6 that follows, imparts the rationale, artistic and technical execution details for the creative new media propositions. Section 7 presents the research data and findings in relation to a theme based analysis. Section 8 then examines key concepts that help to explain findings from the theme based analysis and inform design refinements in relation to the key research questioning strands. Conclusions to this investigation are presented in Section 9 with a blueprint of emergent recommendations as to how human . machine interaction can be improved. Section 10 concludes this thesis with further discussion and a view of the future surrounding the Science, Technology and Arts Nexus in relation to Robotics and human centered Interactive Design.