Multimodal sensory integration processes – MultiSense

schematic: multisensory perception

Fig. 1: Interaction of the different sensory modalities during the holistic perception of food.

Woman watching a screen which shows food

Fig. 2: Evaluation of attention with the aid of an eye tracker.

Fig. 3: Analysis of the emotional reaction using mimic analysis.

Project term: 1.9.2013 to 31.8.2018

Project funding: Fraunhofer-Gesellschaft; funded under the internal Attract program for industry-oriented development work


Odors are perceived in a multisensory environment

The sensory evaluation of food and its packaging (as well as other articles of everyday use; see below) is traditionally carried out using a unidimensional evaluation that focuses on smell and/or taste, sometime also including texture or haptic sensations. During our everyday perception of food, however, other sensory parameters play a significant role, especially in combination. Consumers are rarely confronted with only one sensory stimulus at a given time, but rather perceive orchestrated multisensory stimuli. An example of this is the smell of freshly brewed coffee, which is often accompanied by the sound of the coffee machine or the sight of the coffee packaging. Thus, concurrent visual and auditory stimuli, together with potential trigeminal components (burning, prickling, stinging), haptic and temperature perceptions, influence our perception of food. When evaluating this perception it is therefore necessary to examine how the different senses interact (cf. Fig. 1). Our approach is to use multisensory stimulation scenarios to characterize the sensory properties of products and examine the response it elicits in consumers.

Emotional reactions during food perception

In addition to these diverse sensory stimuli, the perception of food is strongly linked to our emotions. Hedonic and emotional processes are important aspects that drive food choice decisions. The MultiSense group in the Department of Sensory Analytics explores basic mechanisms of hedonic food evaluation to strengthen our understanding of acceptance, preference, and aversion of food. Such emotional processes usually proceed unconsciously, thus special methods are required to characterize consumer reactions. A repertoire of different methods is employed to evaluate sensory processing and visualize these reactions to foods.

Behavioral and physiological experiments

Physiological measurements and behavioral experiments provide insights into the salience of a sensory stimulus or multisensory stimuli:

  • Measurement of heart-rate and its variability, blood pressure, breathing frequency and volume, skin conductance response and skin temperature (to assess current state, e.g., stress, arousal, relaxation etc.)
  • Assessment of attention via eye or mouse tracking (computer-based reaction tests), or a startle response paradigm (cf. Fig. 2)
  • Analysis of reactions using mimic analysis and reaction time measurements (cf. Fig. 3)

Psychological-emotional Analyses

Psychological-emotional investigations are performed to gain knowledge on the hedonic evaluation and emotional reaction to a sensory stimulus or multisensory stimuli:

  • Psychological questionnaires and scales
  • Structured interviews
  • Analysis of the brain activation using electroencephalography (EEG) and functional magnetic resonance imaging (fMRI)

Holistic sensory perception of products

The results of these human physiological and psychological experiments are then routinely compared to the results of chemo-analytical (aroma and taste) and physico-analytical experiments (e.g. color and texture) that characterize the related olfactory and/or trigeminal compounds, and other sensory features of the product.

The holistic approach outlined above can be equally well applied to other product groups to understand consumer reactions and aid in product development or optimization, including:

  • Cosmetics, perfumery and body care (cf. Fig. 4)
  • Cleaning agents
  • Plastic-, polymer-, recyclate-based products
  • Wood, paper, cardboard products, cellulose-based materials
  • Textiles, clothing, leather, functional materials
  • Articles of daily use, toys
  • Furniture and fixtures

With the help of this holistic approach we aim to increase the overall understanding of the  unique sensory perception of a product and to gain knowledge of evaluation and decision processes that drive acceptance and preference and influence purchase decisions. The development of novel tools and techniques for procedural monitoring thereby facilitate the optimization of a product’s characteristics according to the specific needs of consumers.

Fig. 4: Interaction of the different sensory modalities during the holistic perception of cosmetics.