prikaz prve stranice dokumenta A hybrid approach to real-time speech driven facial gesturing of virtual characters
  Download
PDF 6.32 MB
doctoral thesis
A hybrid approach to real-time speech driven facial gesturing of virtual characters
Zagreb: University of Zagreb, Faculty of Electrical Engineering and Computing, 2010. urn:nbn:hr:168:380931
Zorić, Goranka
University of Zagreb
Faculty of Electrical Engineering and Computing

Institutional repository: FER Repository

Cite this document

Zorić, G. (2010). A hybrid approach to real-time speech driven facial gesturing of virtual characters (Doctoral thesis). Zagreb: University of Zagreb, Faculty of Electrical Engineering and Computing. Retrieved from https://urn.nsk.hr/urn:nbn:hr:168:380931

Zorić, Goranka. "A hybrid approach to real-time speech driven facial gesturing of virtual characters." Doctoral thesis, University of Zagreb, Faculty of Electrical Engineering and Computing, 2010. https://urn.nsk.hr/urn:nbn:hr:168:380931

Zorić, Goranka. "A hybrid approach to real-time speech driven facial gesturing of virtual characters." Doctoral thesis, University of Zagreb, Faculty of Electrical Engineering and Computing, 2010. https://urn.nsk.hr/urn:nbn:hr:168:380931

Zorić, G. (2010). 'A hybrid approach to real-time speech driven facial gesturing of virtual characters', Doctoral thesis, University of Zagreb, Faculty of Electrical Engineering and Computing, accessed 26 November 2024, https://urn.nsk.hr/urn:nbn:hr:168:380931

Zorić G. A hybrid approach to real-time speech driven facial gesturing of virtual characters [Doctoral thesis]. Zagreb: University of Zagreb, Faculty of Electrical Engineering and Computing; 2010 [cited 2024 November 26] Available at: https://urn.nsk.hr/urn:nbn:hr:168:380931

G. Zorić, "A hybrid approach to real-time speech driven facial gesturing of virtual characters", Doctoral thesis, University of Zagreb, Faculty of Electrical Engineering and Computing, Zagreb, 2010. Available at: https://urn.nsk.hr/urn:nbn:hr:168:380931

Metadata
TitleA hybrid approach to real-time speech driven facial gesturing of virtual characters
Title (croatian)Hibridni pristup stvaranju gesti lica virtualnog lika govornim signalom u stvarnom vremenu
AuthorGoranka Zorić
MentorIgor Sunday Pandžić (mentor)
Committee memberMaja Matijašević (predsjednik povjerenstva)
Committee memberDragan Jevtić (član povjerenstva)
GranterUniversity of Zagreb
Faculty of Electrical Engineering and Computing
Zagreb
Defense date and country2010-07-09, Croatia
Scientific / art field,
discipline and subdiscipline		TECHNICAL SCIENCES
Computing
Universal decimal classification
(UDC)		004 - Computer science and technology. Computing. Data processing
Abstract
This thesis investigates automatic facial gesturing of virtual characters based only on a speech input. Facial gestures that have been included are various nods and head movements, eye blinks, eyebrow gestures and gaze, i.e. facial and head movements used in nonverbal communication that are connected to the syntactic and prosodic structure of the underlying speech rather than to semantic or emotions. The main issue in speech driven facial gesturing is an audio to visual mapping. In this thesis, a hybrid method for the real-time correlation of the speech signal with facial gestures has been proposed. An idea behind the proposed method is to include different issues that are considered important for visual prosody through three different modules. In a data-driven module facial gestures are correlated with prominent parts of the speech. A rule-based module consists of a set of rules, which among others include rules for punctuation and prolonged pauses. Additionally, in a statistics module, results of first two modules are fine-tuned and further shaped. In order to check how believable virtual characters are if animated using facial gestures obtained by the proposed method, a system for speech driven facial gesturing has been implemented, and used for a perceptual evaluation. Testing subjects generally considered facial gestures consistent and time aligned with the underlying speech, and a virtual human applying them is perceived as a natural. Moreover, the set up hypothesis has been proved: adding facial gestures driven by the speech signal to the animation of virtual characters using the proposed hybrid approach in real-time will result in more coherent and appropriate facial movements than random or no movements. Further more, the system for speech driven facial gesturing of virtual characters has been put in a context of virtual news presenters used in networked environments, since that is where the future research of this system strives next.
Abstract (croatian)
Ova doktorska disertacija istražuje automatsko stvaranje gesti virtualnog lika govornim signalom u stvarnom vremenu. Pokreti lica uključeni u ovaj rad su različiti pokreti glave i obrva, treptanje i pogled, tj. svi pokreti lica i glave koji se koriste u neverbalnoj komunikaciji povezani sa sintaksom i prozodijom govora, a ne s emocijama ili značenjem riječi. Kod generiranja animacije lica iz govora potrebno je naći korelaciju između govornog signala i pokretanja lica. Ta korelacija nije direktna, već je posljedica više elemenata, te postoje individualne i kulturološke specifičnosti. U disertaciji su istražene različite metode za preslikavanja iz audio informacije u vizualnu. Odabrano rješenje temelji se na hibridnom pristupu, pri čemu su statistički ispravni pokreti lica i glave usklađeni sa prozodijom govora u stvarnom vremenu dobiveni tehnikom strojnog učenja, korištenjem pravila te statistikom dobivenom iz baze podataka. Tehnikom strojnog učenja dobiveni su pokreti lica koji su povezani s naglašenim dijelovima govora, dok su pomoću pravila, između ostalog, dobiveni pokreti lica povezani s različitim pauzama u govoru. Ovi rezultati su dodatno oblikovani statistikom dobivenom iz baze podataka. Kako bi provjerili pokrete lica virtualnog lika, izgrađen je sustav za automatsko pokretanje lica govornim signalom zasnovan na predloženoj hibridnoj metodi, te je iskorišten za subjektivnu evaluaciju. Sudionici testiranja su doživjeli pokrete lice uglavnom kao vremenski usklađene, te konzistentne s govorom, dok je općeniti dojam da virtualni lik djeluje prirodno. Također, potvrđena je postavljena hipoteza: pokreti lica iz govora dodani animaciji virtualnog lika korištenjem predložene hibridne metode u realnom vremenu daju bolje rezultate nego slučajni pokreti ili lice bez pokreta. Tako izgrađen sustav stavljen je u kontekst virtualnih prezentatora vijesti koji se koriste u umreženim okruženjima, budući da daljnje istraživanje ove problematike ide u tom smjeru.
Keywords
Keywords (croatian)
Languageenglish
URN:NBNurn:nbn:hr:168:380931
Promotion2010
Study programmeTitle: Computer Science
Study programme type: university
Study level: postgraduate
Academic / professional title: Doktor znanosti (Doktor znanosti)
Type of resourceText
Extent85 str. : graf. prikazi
File originBorn digital
Access conditionsOpen access
Terms of useLicence In copyright icon
Created on2024-03-20 10:26:13