Dissertation defense (September 28, 2020): Ellen Paixão Silva
Student: Ellen Paixão Silva
Title: The influence of multissensory media in dyslexic children learning
Advisors: Joel André Ferreira dos Santos (advisor) Silva e Glauco Amorim (co-advisor)
Committee: Joel André Ferreira dos Santos (president), Glauco Amorim (CEFET/RJ), Gustavo Guedes (CEFET/RJ), Renata Mousinho (UFRJ) e Maria da Graça Campos Pimentel (USP)
Day/Time: September 28, 2020 / 19h
Room:
Abstract:
Dyslexia is a specific learning disorder that can affect reading and writing skills, mainly affecting the literacy process. In reading, the overload in working memory impairs the comprehension of texts, with symptoms such as word recognition errors and difficulty in decoding words being observed. Research published in the literature indicates that the formation of memory encompasses different human senses and that multisensory stimuli from multimedia content improve the performance of students with dyslexia. However, most multimedia applications stimulate only two
of the five human senses: sight and hearing. Recente research in multimedia applications seek to include stimuli to the remaining senses in the form of the so-called mulsemedia applications (multiple sensorial media). In the form of a multisensory book, this work proposes the use of multisesory content incorporating sounds, variation colors and intensity of the ambient light, wind and smell to the textual content. Multissensoru content are synchronized to reading using an eye tracker.
MBook is an application developed in this work capable of storing and presenting multisensory books. It uses the eye tracker to map the position of the eyes on the display that displays the text and instantly identify which word is being read. With this information, it is possible to check
whether that word or sequence of words has associated multisensory content and thus execute the corresponding content. MBook decouples the textual content of the book from the logic of multisensory experience. In this way, it allows authors to focus on the story and mark the locations
in the story where multisensory content can be presented, and content producers to explore existing stories to enhance the reader’s experience, including traditional multimedia content and sensory effects.
In this work we start from the hypothesis that “the use of multisensory content synchronized with reading reduces the overload in working memory necessary for reading, thus helping an individual with dyslexia to understand and memorize the content being read ”. To evaluate the hypothesis, two studies were carried out: (i) a comparative analysis between two individual case studies, one with a child with dyslexia and another with a child without dyslexia with paired profiles; and (ii) an analysis of a collective case study with five children with dyslexia. The results obtained indicate an increase in the intrinsic motivation for reading, a gain in reading speed, and an increase in text comprehension when multisensory content are included. The results obtained also indicate that MBook can be an important therapeutic tool for the treatment of students with dyslexia thus minimizing the damage caused by this disorder.
