In this work the following three basic research questions are discussed: (1) can significant effects of modality efficiency and input performance on the selection of input modalities in multimodal HCI be disclosed by unified experimental investigations? (2) Can a utility-driven computational model of modality selection be formed based on empirical data? (3) Can the compiled model for modality selection be utilized for the practical application in the field of automated usability evaluation? Initially, foundations of decision-making in multimodal HCI are discussed, and the state of the art in automatic usability evaluation (AUE) is described. It is shown that there are currently no uniform empirical results on factors influencing modality choice that allow for the creation of a computational model. As part of this work two AUE tools, the MeMo workbench and CogTool, are extended by a newly created computational model for the simulation of multimodal HCI. Aiming at answering the first research question, the empirical part of the thesis describes three experiments with a mobile application integrating touch screen and speech input. In summary the results indicate that modality efficiency and input performance are important moderators of modality choice. The second research question is answered by the derivation of a utility-driven model for input modality choice in multimodal HCI based on the empirical data. The model provides probability estimations of modality usage, based on different levels of the parameters modality efficiency and input performance. Four variants of the model that differ in training data are tested. The analysis reveals a considerable fit for models based on averaged modality usage data. Answering the third research question it is illustrated how the modality choice model can be deployed within AUE tools for simulating multimodal interaction. The multimodal extension as well as the practical utilization of MeMo is depicted, and it is described how unimodal CogTool models of touch screen and speech based interaction can be rendered into multimodal models. A comparison of data generated by simulations with the AUE tools with predictions of the derived modality selection algorithm verifies the correct integration of the model into the tools. The practical application discloses the usefulness of the modality choice model for the prediction of the number of steps and the total time spent to solve specific tasks with multimodal systems. The practical part is concluded by a comparison of Memo and CogTool. Both tools are classified, and an assessment on a subjective basis as well as on the the basis of the quality of predictions is conducted.

Download this book @ https://depositonce.tu-berlin.de/handle/11303/5653