Memorias de investigación
Adaptive and Immersive Interfaces to improve Situational Awareness in Multi-Robot Missions

Research Areas
  • Artificial intelligence,
  • Robotics,
  • Virtual reality,
  • Human-robot interphase

Missions involving multiple robots have experienced an unprecedented growth over the last years. The explanation behind this fact may be that using a robot fleet is more effective, efficient, flexible and fault tolerant than employing a single robot for the task. However, this kind of missions entail a set of challenges related to human factors. Among these challenges, the most remarkable include managing workload, keeping situational awareness and managing stressful situations. This PhD thesis aims at developing a new generation of interfaces that allow single operators to control robot fleets. For this purpose, data mining and machine learning tools are applied to discover relevant information and provide it to operators, whereas virtual and augmented reality technologies are used to create immersive interfaces that improve their situational awareness. The thesis can be split into four blocks: ? Mission modeling: The objective is to generate mission models able to show what is happening at any moment and select relevant information. In this thesis, the use of process mining is proposed as a way to automatically model the missions through the data generated by them. Specifically, the use of Petri nets is proposed to determine the state of the mission, and decision trees are employed to predict its evolution. Among all the discovery algorithms of process mining that have been tested, the inductive miner provided the best results in the context of multi-robot missions. - Operator modeling: The objective is to generate operator models to enable a prediction of their behavior during the mission, in order to provide them with the most adequate information at any moment. The classification of operators is proposed according to their mission control strategies, and it is followed by the prediction of their preferences, taking into account the data of previous missions. The first step is performed by applying clustering algorithms (Partition Around Medoids provided the best classifications), whereas the second one is developed by using probability distributions. - Adaptive interfaces: Once the mission and operator models are obtained, the objective of this block is to discover the relevant information from the mission data and provide it to the operators through the interface. In this section of the thesis, the use of neural networks is proposed as a tool to transfer functions from operators to interfaces. Specifically, the experiments performed demonstrate that these models allow the interface to evaluate the relevances and risks of robots mirroring the way a human operator would. - Immersive interfaces: Once the relevant information is selected, according to the mission?s state and operator preferences; the objective of this last block is to transmit this information to the operators, while controlling their workload and keeping their situational awareness. In this thesis, the development of immersive interfaces is proposed based on virtual reality, which introduces the operators in the scenario where the robots are working. The experiments performed on this phase demonstrate that these interfaces improve the situational awareness of operators without increasing their workload. All the developments of the thesis have been validated through experiments with operators, who have monitored and commanded robot fleets (formed by ground, aerial and manipulator robots) to perform realistic missions (surveillance and intervention in disaster areas) in varied and relevant scenarios (both outdoors and indoors).
Mark Rating
Sobresaliente cum laude

Research Group, Departaments and Institutes related
  • Creador: Centro o Instituto I+D+i: Centro de Automática y Robótica (CAR). Centro Mixto UPM-CSIC
  • Departamento: Automática, Ingeniería Eléctrica y Electrónica e Informática Industrial