© Stefano Nolfi, 2021 | How to cite this book | Send your feedback | Collaborate
Behavior is a dynamical process that unfolds in time and originates from the continuous interaction between the robot’s body, the robot’s brain, and the environment. In the previous two chapters we stressed the fact that behavior is not generated by the brain of the robot only. The body of the robot and the environment play an important role as well. Moreover, the behavior of the robot is not simply the result of the contribution of the three constituting components (the brain, the body and the environment). It is the result of their interactions.
In this chapter we will focus on the dynamical system nature of behavior. The term dynamical systems indicates systems that change over time. It is a wide class that includes pendulums, planetary systems, weather, growing plants, human behaviors, stock markets, and human languages. More specifically, we will illustrate the implications of the dynamical system nature of behavior for robotics. Dynamical systems display a series of important properties. Exploiting these properties is essential to produce effective behaviors.
Dynamical system theory is the set of ideas and mathematical tools that describe how dynamical systems change over time (Strogatz, 2001). It originated as a branch of physics and then becomes an interdisciplinary subject. The field started in 1600, when Newton invented differential equations, discovered his laws of motion, and combined them to explain planetary motions.
Unfortunately, the analytical method used by Newton can be applied only to very special cases. For this reason, the second crucial progress in dynamical system theory was the introduction of the qualitative method discovered by Pointcaré that can be applied to any dynamical system.
The invention of high-speed computers, that enabled the possibility of simulation studies, represented the third crucial step.
One of the first notable application of dynamical system theory was made by Edward Norton Lorenz who modelled rolling fluid convection in the atmosphere with a set of 3 non-linear differential equations. By simulating how the state of the variables governed by the equations vary over time, Lorenz observed a chaotic dynamic (Figure 5.1). The state of the system continues to oscillate in an irregular aperiodic fashion without never setting down to an equilibrium or to a periodic state. The line making up the curve, representing how the state of the system varies over time, never intersects itself and loops around forever spending a variable time on each wing. Moreover, Lorenz observed that the behavior produced was highly influenced by the initial state. Minor differences in the initial state produced totally different behaviors later on. This implies that the system is inherently unpredictable since tiny errors in the measure of the initial state have large effects in the long term. Finally, Lorenz observed that the chaotic dynamic exhibited by his system included a form of structure. Indeed, as shown in Figure 5.1, the plot of the state of the system over time produce a butterfly-shaped trajectory.
As it will later become clear, these peculiar properties do not characterize the rolling fluid convection only. They are general properties characterizing many different systems.
In this section I will briefly summarize the essential concepts and terminology of dynamical systems theory. For a more detailed account see Strogatz (2001).
To describe the behavior of a dynamical system we should first identify the mutable parts or properties. This permits to describe the system with a set of state variables χ, which indicate the current state of the varying components. The dynamical law specifies how the state variables change over time. The dynamical law can be represented by a set of differential equations or iterated functions.
For example, in the case of a pendulum the variables that change over time are the angle (θ) and the angular velocity (θ̇) of the pendulum. The dynamical law that determines how the state of the pendulum changes over time can be modelled with the following differential equation (see Tedrake, 2019):
where g is the acceleration produced by gravity, l is the length of the pendulum, and b is a damping factor For simplicity we assume that the rope is massless.
By solving the differential equation, we would be able to calculate the state of the pendulum at any arbitrary time t+n on the basis of the state of the pendulum at time t. However, the presence of a non-linearity in the equation, i.e. the sin component, prevents the possibility to solve the equation analytically. This, however, does not prevent the possibility to analyze the geometrical and/or topological structure of the dynamics, as we will see below.
The state space of the system includes all the values that the variables can assume. A phase portrait is a geometrical representation of the dynamics of the system in the state space. It can include a curve that visualize how the state of the system varies over time from an initial state (as in Figure 5.1). Alternatively, it can include the vectors (represented with arrows) that display how the state of the system changes in a single time step from any possible initial state (Figure 5.3). The phase portrait permits to analyze qualitatively the behavior of the system. For example, it permits understanding whether the system converge to a stable state, which remains constant from a certain point on, or to a chaotic attractor, as in the case of the rolling fluid convection system illustrated above.
The long-term behavior of a dynamical system can diverge or converge on a specific set called attractor, i.e. a state or a set of states toward which a system tends to evolve from all starting conditions or from a large set of different starting conditions. In the case of equilibrium point attractors, the state of the system converges toward a specific state. This is the case of the dumped pendulum that converges on the state [θ = 0, θ̇ = 0], i.e. on a resting position in which the pendulum lies down still. The fact that the long-term dynamics of the dumped pendulum converges on this equilibrium point attractor is clearly shown by the phase portrait (Figure 5.3). In the case of limit cycle attractors, the state of the system converges over a closed trajectory in the state space that repeats over and over (for example the closed curve shown in red in Figure 5.4). Finally, in the case of strange attractors the system converges over a fractal structure like that shown in Figure 5.1.
The convergence is caused by the fact the trajectories passing through nearby states are oriented toward the attractor. The area of the state space that converge on the attractor is called basin of attraction. The basin of attraction implies that the system will converge on the attractor, after a transient phase, from any initial state situated in the basin of attraction. In the case of equilibrium points and limit cycle attractors, the system converges on the same states independently of the initial conditions. In the case of strange attractors, instead, the system is sensitive to the initial condition in the long term. This implies that any two arbitrarily close alternative initial points in the state space can lead to states that are arbitrarily far apart in the long term.
In general, the state space of a dynamical system can contain multiple attractors surrounded by associated basin of attractions. These basins are separated by unstable manifolds called separatrices. This implies that similar states located far from the separatrices are situated in the same basin of attraction and converge on the same attractor. Instead, initial states located near the separatrices can belong to different basins of attraction and converge on different attractors.
Finally, the dynamic of a system is influenced by parameters. For example, the dynamic of a pendulum is influenced by the value of gravity, the dynamics of a robot is influenced by the battery level. This means that the behavior of the system and the phase portrait of the system change when a parameter is modified. A parametrized dynamical law defines a family of dynamical systems, with any particular flow corresponding to a single set of parameters.
In general, dynamical systems display structural stability, i.e. the dynamics of the system changes only slightly as a result of small variations of parameters. The value of the limit sets and the shape of basins of attraction can vary slightly but the number and type of attractors remains stable. However, at certain parameter values, dynamical systems can show structural instability. This means that minimal changes in the value of the parameters can produce drastic qualitative changes. These qualitative changes are referred as bifurcations.
A simple example illustrating the sensitivity to parameters is the logistic map system introduced by Robert May (1976) that has a state space characterized by a single dimension (x) and is governed by the following iterated map equation:
Figure 5.5 shows the long-term state or states produced by the system for different values of the m parameter. When the parameter m is in the range [1.0, 3.0], the system converges on an equilibrium point attractor (see Figure 5.5). When m is in the range [3.0, 3.44949], the system converges on a limit cycle attractor in which the state oscillates between two values (from most of the initial conditions). When m is in the range [3.44949, 3.56995], the system converges on a limit cycle attractor formed by 4, 8, 16, 32, and more values. Finally, when m is > 3.56995, the system converges on a strange attractor.
This ends this brief introduction to dynamical systems. In the next sections we will discuss the implications of the dynamical system nature of behavior for robotics.
The dynamical system tools permit to clarify in a more precise way in which sense the behavior of a robot is not simply a manifestation of the robot’s brain but is rather the result of the interaction between the brain of the robot, its body, and the environment.
For this purpose, it is useful to distinguish between fully activated and under-actuated robots. The former can be commanded to follow arbitrary trajectories in configuration space by controlling all the degrees of freedom and thanks to the availability of unlimited energy resources. The latter instead cannot perform as the former due to the lack of actuators with respect to the degrees of freedom and/or due to the presence of constraints and limits on the actuators.
As an example of a fully actuated systems let’s consider a pendulum with a high-powered actuated joint. The system is fully actuated since the number of actuators equals the number of DOFs and since the actuator does not have limitations, e.g. power limitations. For a system of this type the effect of gravity, and consequently the dynamics arising from the interaction between the pendulum and the environment, can be cancelled. This can be obtained by using the motor to apply a torque equivalent to the gravitational force acting on the pendulum, i.e. equivalent to the tangential component of gravity (assuming a rigid rod). Once the dynamics originating from the interaction with the environment is cancelled, controlling the pendulum become trivial. To move the pendulum in a given direction, the controller should simply apply an additional torque in the right direction until the pendulum assumes the desired target position. In a system of this kind, the behavior is generated from the robot controller. The interaction between the system and the environment does not play any role since the effect of gravity is neutralized and since the system does not interact with external objects.
As an example of highly-actuated system we can consider ASIMO®, the humanoid robot released by Honda in 1996 that represented the state of the art of walking robots for the following 20 years. Although ASIMO displayed impressive walking capabilities, its behavior looked unnatural. This was due to the fact that, similarly to the fully actuated pendulum described above, ASIMO uses a lot of torque to cancel the natural dynamics and to follow a planned trajectory. Indeed, the robot consumes at least an order of magnitude more energy than a human to walk. I used the term highly-actuated instead than fully actuated since robots are always characterized by a certain level of under-actuation. Controlling all DOF is practically impossible and relying on unlimited energy is unfeasible in most of the cases. Still, for the reason described above, ASIMO constitutes a highly-actuated system.
Under-actuated systems, on the contrary, are energy-efficient. They do not waste energy in the attempt to cancel the natural dynamics originating from the interaction with the environment. Examples of underactuated systems are the passive or quasi-passive robots described in Section 3.2. Most observers would agree that the walking style of these machines is more natural than ASIMO’s.
Another example of under-actuated systems is a low-powered actuated pendulum. The limitation of the actuator prevents the possibility to cancel out the natural dynamics of the system which is illustrated by the phase portrait of a passive pendulum (Figure 5.3). The actuator can only be used to vary slightly the direction and the module of the vectors illustrated in the phase portrait (Figure 5.6).
In the case of underactuated systems, therefore, the role of the brain consists in altering as slightly as possible the vector field that originate naturally from the interaction between the robot and the environment. The goal is that to vary the natural behavior of the systems within the limits imposed by constraints. In other words, the role of the brain is to orchestrate the robot/environmental dynamics in a way that enables the production of functional useful behavior.
The long-term stability and the sensitivity to critical conditions of dynamical systems has also important implications for robotics.
The possibility to produce behaviors that are robust to environmental variations is essential for robots since the physical world is full of uncertainties and almost no parameter, dimension, or property tends to remain exactly constant. We will discuss this aspect in more details in Chapter 7.
Dynamical systems provide a structural solution to the problem of generating behaviors that are effective and that are robust to perturbations: attractors with associated basins of attraction. Attractors cannot exist in isolation. Their existence depends on the presence of transient trajectories that converge to the attractor forming the associated basin of attraction. The basin of attraction thus plays both the roles mentioned above: ensures that the system converges to a desired steady state dynamic and ensures that the system converges back to that dynamics after deviations caused by variations or perturbations.
To clarify the concept, let’s consider a counter-example, i.e. a system exhibiting a not structurally stable behavior (Tani, 2016). For example, a frictionless spring-mass system (Figure 5.7, left) described by the following equations:
where x is the one-dimensional position of a mass m, v is its velocity, and k is the spring coefficient. A system of this kind will start to produce a periodic oscillatory behavior with an amplitude that depends on the initial extension of the spring. This behavior, however, is not structurally stable. Indeed, if we apply a force to the mass of the oscillator, the amplitude of the oscillation will change immediately and will not converge back on the original oscillation amplitude. The reason why the behavior of the system is unstable is that it does not corresponds to a limit-cycle attractor surrounded by an associated basin of attraction. Indeed, the phase portrait on the system on the 2-dimensional state space constituted by the x and v variables does not show a steady state set surrounded by a basin of attraction. It rather shows a set of concentric circles lacking any convergent flow (Figure 5.7, right).
We already encountered examples of functional behaviors corresponding to fixed points or limit cycle attractors documented by phase portrait analysis: the behavior of the foraging robot and of the robot navigating in the T-maze illustrated in Section 4.4 and 4.7. Presumably, also the behavior generated by the passive walker illustrated in Section 3.2 is the result of a limit cycle attractor that drives the system toward a specific sequence of states that repeats over and over. Indeed, although a phase-portrait analysis of this system is missing, the ability to spontaneously recover from divergences occurring during motion can be attributed to the presence of a basin of attraction that brings the system back to the limit cycle.
The second important structural property that is relevant for robotics is flexibility, namely the possibility to drastically modify the behavior exhibited by the system in specific contextual situations requiring the production of qualitatively different behaviors. The combination of stability and flexibility over time can produce a form of chaotic itineracy characterized by quasi-stable behaviors interrupted by sudden transition leading to other different quasi-stable behaviors (Iizuka & Ikegami, 2004). The possibility to realize drastic behavioral changes is enabled, as we have seen, by the possibility to generate bifurcations in the system dynamics through variations of critical parameters. We will see an example of phase transition in behavior in Section 5.7.
Finally, the last fundamental property of dynamical system (which has implications for robotics) is the possibility to generate dynamics with multi-level and multi-scale organizations. As an example of organization of this type we can consider human beings, a dynamical system formed by heterogeneous entities (molecules, cells, individuals, social groups) spanning at widely different levels of organization. The elements forming each level are independent from other elements of the level and from other levels, to some extent, but are also part of the same dynamical system.
From the perspective of this book, a relevant issue is the fact that the behavior exhibited by a robot manifests itself at different levels of organization. It usually displays a hierarchical multi-level and multi-scale organization that includes an initial level, formed by elementary behaviors extending for limited time periods, and by higher organization levels, including higher and higher level behaviors extending for longer time periods (Nolfi, 2009). The behaviors forming each level are partially independent (West-Eberhard, 2003; Carvalho and Nolfi, 2016). They play different functions and extend over different periods of time. However, they are not completely independent. They are part of the overall behavior produced by the robot and they are not separated by clearly identifiable boundaries. As we will see in the following chapters, the multi-level and multi-scale organization of behavior plays important functions. For example, it permits to generate a large number of behaviors by recombining in a compositional manner a smaller set of elementary behaviors and facilitates the development of complex behaviors.
An example that illustrates the importance of displaying behaviors organized in functional specialized sub-behaviors and the mechanism that can regulate the transition between sub-behaviors is constituted by the cleaning robot described in Carvalho and Nolfi (2016).
The experiment involves a simulated MarXbot robot (Bonani et al., 2010) that is evolved for the ability to vacuum clean the floor of an in-door environment. The robot is provided with 8 infrared proximity sensors, which encode the average activation state of eight groups of three adjacent sensors, a time sensor that encodes the amount of time passed since the start of a cleaning episode, 3 internal neurons, and 2 motors neurons that control the desired speed of the two wheels. The environment consists of an indoor area including a large central room and a series of corridors and smaller rooms (Figure 5.9). The fitness corresponds to the fraction of 20x20cm non-overlapping areas visited by the robot at least once during a cleaning episode. At the beginning of each episode, the length of the walls surrounding the environment are varied randomly up to 10% and the robot is placed in a randomly selected position and orientation within the large room. Episodes are terminated after 7500 steps corresponding to 6m and 15s.
As reported by the authors, robots lacking the time sensor evolved sub-optimal solutions characterized by the production of a single exploration behavior (Figure 5.9, top). This behavior is generated by alternating obstacle-avoidance phases near obstacles and straight motion phases far from obstacles. These robots fail to clean part of the peripheral areas. The robots provided with the time sensors, instead, develop a better solution that involves the production of two qualitatively different behaviors: a spiraling behavior to clean the large central room, and a wall following behavior to clean the peripheral portions of the environment (Figure 5.9, bottom). This behavior actually resembles that produced by the first versions of the iRobot Roomba® vacuum cleaning robot.
In the case of the robot shown in Figure 5.9 (top) the role played by the evolved brain can be summarized with two control rules that make the robot turn left or move straight when the frontal infrared sensors are activated or not, respectively. By moving straight far from obstacles, the robot keeps experiencing similar observations and consequently keeps moving straight until it approaches an obstacle. At that point, the activation of the frontal infrared sensors elicits the second rule that makes the robot turn left until its frontal side is free from obstacles. This, in turn, re-triggers the execution of the first control rule that generates a straight movement behavior, again.
In the case of the robot shown in Figure 5.9 (bottom), instead, the role of the evolved brain can be summarized with two control rules that are responsible for the spiraling and wall-following behaviors. The first control rule makes the robot move forward by turning left with an angle that decreases progressively as time passes. This rule thus produces a spiraling behavior characterized by larger and larger spirals that finally become a move-forward behavior in which the robot also slightly turns on the right. This final behavior combined with the second control rule (which makes the robot turn slightly on the left with an angle proportional to the activation of the right infrared sensors), enables the robot to produce a wall following behavior during the second part of the episode.
The transition from the spiraling and to the wall-following behavior is regulated by the state of the time sensor, which depends on the time passed, and by the state of the infrared sensors, which depends on the relative position and orientation of the robot in the environment. Indeed, the transition occurs within an appropriate time window when the right infrared sensors are activated. This ensures that the transition occurs when the robot is located near a wall with the wall on its right, i.e. in a position from which the wall-following behavior can be initiated. Moreover, this ensures that the transition occurs at the right time, i.e. after the robot spent enough time cleaning the central portion of the environment and has still sufficient time left to clean the peripheral portions of the environment.
We can indicate the state that triggers the execution of a behavior, and consequently that produces the transition to the new behavior, with the term affordance (Gibson, 1979; see also Chemero, 2011). Affordances can be described as states, extracted from observations, which indicate the opportunity to exhibit a certain behavior. From a dynamical system perspective, as we said above, affordances consist of critical values of parameters that produce a bifurcation in the robot/environmental dynamics.
Affordances can be provided naturally by the environment without the intervention of the robot or can be co-generated by the robot through actions and/or internal processing. For example, the activation of the frontal infrared sensors “affording” an obstacle avoidance behavior occurs naturally. Every time the robot approaches or is approached by an obstacle, i.e. every time the robot should avoid an obstacle, the robot experiences the state of its frontal infrared sensors activated. Instead, the activation of the right infrared sensors that “afford” the wall-following behavior in the robot shown in Figure 5.9 (right) is co-generated by the robot through the exhibition of the left-spiraling behavior. Indeed, this behavior ensures that the robot later experiences the observation that triggers the wall-following behavior.
Read section 13.8 and follow the instructions included in the Exercise 7 to plot the phase portrait of an under-actuated pendulum at different stages of the evolutionary process.
Bonani M., Longchamp V., Magnenat S., Retornaz P., Burnier D., Roulet G. et al. (2010). The marXbot, a miniature mobile robot opening new perspectives for the collective-robotic research. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4187–4193.
Carvalho J.T. & Nolfi S. (2016). Behavioural plasticity in evolving robots. Theory in Biosciences: 135(4): 201–216
Chemero A. (2011). Radical Embodied Cognitive Science. Cambridge, MA: MIT Press.
Gibson J. (1979). The Ecological Approach to Visual Perception. Boston: Houghton-Mifflin.
Iizuka H. & Ikegami T. (2004). Adaptability and diversity in simulated turn-taking behavior. Artificial Life, 10(4): 361-378.
Lorenz E.N. (1963). Deterministic nonperiodic flow. Journal of Atmospheric Sciences, (20): 130-141.
May R.M. (1976). Simple mathematical models with very complicated dynamics. Nature, 261 (5560): 459-467.
Nolfi S. (2009). Behavior and cognition as a complex adaptive system: Insights from robotic experiments. In C Hooker (Ed.), Handbook of the Philosophy of Science. Volume 10: Philosophy of Complex Systems. General editors: Dov M. Gabbay & Paul Thagard and John Woods. Elsevier.
Strogatz S. (2001). Nonlinear Dynamics and Chaos. Western Press
Tani J. (2016). Exploring Robotic Minds: Actions, Symbols, and Consciousness as Self-Organizing Dynamic Phenomena. New York: Oxford University Press.
Tedrake R. (2019). Underactuated Robotics: Algorithms for Walking, Running, Swimming, Flying, and Manipulation (Course Notes for MIT 6.832). Downloaded on November 2019 from underactuated.mit.edu.
West-Eberhard M.J. (2003) Developmental plasticity and evolution. Oxford University Press, New York