Besides fundamental research, I am also involved in some applied research, such as SpeechView. SpeechView consists of a pair of video glasses connected to a microphone that records the speech output of an interlocutor and transmits it wireless to a computing unit running speech-to-text software.
The converted text is presented in the video glasses where the person wearing the glasses can read what the interlocutor has said. For sudden-deaf or bad-of-hearing people this system improves the intelligibility of speech significantly and can potentially increase their quality of life.
AThEME Project: Understanding the cognitive consequences of multilingual language comprehension and production
Postdoc: Bastien Boutonnet
Supervisors: Niels Schiller and Lisa Cheng
Here in Leiden, I am part of the AThEME project studying various facets of being multilingual –a great endeavour! Currently, we are planning to study multilingual interaction with a focus on foreign accent processing. We ask questions such as: Is information delivered in a non-native accent processed differently than when it is delivered in a native accent? Do people behave differently on a series of non-linguistic tasks depending on which of their several languages they use to perform the tasks? Many more will come I’m sure! I would also like to bring my interests in language-perception interactions in multilingual speakers.
Primarily, I am hoping to develop a side of my research (language processing) which has been more of a secondary goal in my previous research. I think being able to see both sides of the coin (how do we “do” language and what language does to us) is essential in order to have a more fine grained understanding of what being multilingual really entails. I am also hoping to develop some of my technical skills by implementing brain imaging techniques such as MRI and structural connectivity. I therefore strongly value collaborations between members of the various institutes and will be more than happy to share what I know and learn from others.
Estimates state that bi- or multilingualism is presumably the rule rather than the exception. We carried out studies about the activation and representation of sounds from one vs. more language inventories.
For instance, when we read texts in our second language, do we activate sounds and phonological representations of our first language as well? We have been able to show that L2 phonology is active during L1 processing and vice versa. This argues for one large phonological inventory that is non-selectively activated.
Online Processing of Speech Prosody in Mandarin
PhD student: Min Liu
Supervisors: Yiya Chen and Niels O. Schiller
In Mandarin, tone and intonation share the same acoustic correlate – F0. At the syllabic level, F0 is employed to indicate different lexical tones that differentiate lexical meanings. For instance, Tone2 is characterized as a rising F0 contour whereas Tone4 is represented as a falling F0 contour. At the sentential level, F0 is used to signal intonation types such as statement and question. Like many other languages, question intonation shows an upward trend in F0, in contrast to a downward trend in F0 for statement intonation in Mandarin.
Since both tone and intonation are realized in terms of F0, there might be a conflict between the direction of tone and that of the intonation in F0. In this study, we attempt to reveal the neuro-psychological mechanism of Mandarin speakers processing tone and intonation, to see if people can disentangle the information of tone and that of intonation. We would also like to investigate other factors that might be involved in speech prosody processing, such as, focus , semantics and so on.
Together with various colleagues from Leiden (Yiya Chen, Claartje Levelt, etc.) I have investigated the representation and processing of (linguistic) stress. In 90% of all Dutch words, the stress is on the first syllable. We thought: only the 10% of irregular words will be stored, the rest will be calculated using rules. But that's not how it works.
Within the irregular words, there are words which behave in a regular way. Thus, it may well be a combination of storing and computing lexical stress information.
English Vowel Production and Perception of Javanese and Sundanese Learners
PhD students: Arum Perwitasari
Supervisors: Dr. Yiya Chen (LUCL), Prof. dr. Niels Schiller (LUCL), Prof. dr. Marian Klamer (LUCL)
A study on a local language would be interesting in order to shape a thorough understanding on any possible impact of regional languages to English standard language learning. This study is going remedy this gap in the literature by examining vowel production and perception of Javanese and Sundanese Learners of English in order to more fully investigating the effect of L1 to TL vowel production, in the perspective of experimental and cognitive view. The study "English Vowel Production and Perception of Javanese and Sundanese Learners" comprises an attempt to a fundamental research in the speech of Second Language Learners (SLL). It is to examine deeper how SLL manage to produce and perceive L2 vowel sounds. The production and perception experiment are completed in different session.
The production study involves the participants pronouncing list of words in controlled sentences that are recorded for the analysis of quality (F1,F2 and F3) and quantity (duration). The perceptual experiment involves presenting listener with sounds and asking them to identify sound they have heard for the analysis of error rates, spatial attraction, reaction times. The sounds in the task are prepared to examine perception of SLL by seeing hand movement response.
Grammatical features (gender, number, etc.)
In a variety of studies, we are investigating how grammatical properties of a language, such as grammatical gender, number or classifiers, are represented in the language processing system.
Recently, we investigated a couple of non-western languages such as Konso (gender-number interaction) and Mandarin Chinese (classifiers). We also studied the transfer of grammatical gender properties from a first to a second language with Dutch-English bilinguals.
The interaction between lexical level prosody and sentential level prosody of Dalian Mandarin
PhD student: Yfei Bi
Supervisors: Yiya Chen and Niels O. Schiller (LUCL)
My research focuses on the lexical level and sentential level prosody and their interaction in Mandarin, especially in Dalian Mandarin, which is investigated as a testing case. For lexical level prosody, we focus on how neighbourhood density and word frequency will affect the tonal neutralization, tonal perception and production in both monosyllabic and disyllabic words from the aspects of response time, duration, F0’s contour and velocity. For sentential level prosody, we attempt to investigate the interaction between prosody and information structure (topic, focus and givenness) during speakers’ (speakers of Northern Mandarin and Dalian Mandarin) online sentence processing from the aspects of duration, F0, intensity and other acoustic features. Besides acoustic experiments, we will also perform well-controlled eye-tracking experiments to look into the perception of prosody in the two levels.
The role of lexical phonology in the production of words
Models of phonological encoding have to take many aspects of speech planning into account. For example, on the basis of speech errors and reaction times, it has been shown that segments rather than phonological features play a role in production planning. However, quite a few aspects of the model are still underspecified. More research is necessary in order to describe the processes properly.
Sound and Meaning – How are prosody and information structure interacted in tonal languages
PhD students: Lei Sun
Supervisors: Prof. dr. Niels Schiller (LUCL) and Prof. dr. Vincent van Heuven (LUCL)
I am working in a project which aims at investigating the interaction of prosody and information structure in tonal languages. Information structure refers to the way how speakers package information, e.g. they may choose to focus some parts of / the whole sentence in speech when they are answering a question or correcting wrong information in the interlocutor’s talk. Accordingly, it has been proposed that focus has narrow/broad or corrective/new types. Prosody is widely known to be employed as a tool to encode these meanings in tonal languages.
In Standard Chinese (Mandarin), focused constituent was found to have wide f0 range and longer duration, and post-focus parts were compressed or expanded depending on tonal contexts (Xu, 1999; Chen, 2010). We are interested in two general research questions:
- how is the expression of information structure notions constrained by phonological structure of tonal languages?
- how are different meanings mentioned above encoded with different prosodic cues in tonal languages and why?