Multimodal construction grammar: potential and challenges
doi: https://doi.org/10.31810/rsel.53.1.6
Keywords:
construction grammar; multimodality; gesture; prosodyAbstract
This paper discusses the inclusion of multimodal information in Construction Grammar (CGx). CGx is a family of theories in Cognitive Linguistics that maintains that linguistic knowledge consists fundamentally of constructions, holistic units within a lexical-grammatical continuum, derived from usage patterns detected in communicative acts, that are constantly updated with each new communicative act. In principle, CGx represents an ideal framework for the treatment and incorporation of multimodality, since the theory does not set limits to which aspects of the communicative situation can be incorporated into these structures. If a given aspect is repeated enough and has diagnostic or predictive power, it can theoretically form part of the information included in the construction, which opens the door to the inclusion of multimodal aspects such as prosody, co-speech gestures, or even other cues such as facial expressions, body posture, and gaze direction. From this approach, the paper describes sevreal cases in which multimodal aspects such as prosody and gesturing are seen to interact and how construction grammar has modeled these cases incorporating this information. The paper ends with some challenges that multimodality poses to the current architecture of CGx.
Downloads
References
Aguirre, C. (2003). Early verb development in one Spanish-speaking child, en D. Bittner, W. U. Dressler y M. Kilani-Schoch (eds.), Development of verb inflection in first language acquisition: A cross-linguistic perspective, pp. 1-26. Berlín y Nueva York: De Gruyter Mouton
Andonova, E. y Taylor, H. A. (2012). Nodding in dis/agreement: A tale of two cultures. Cognitive Processes Suppl 1:S79-82. doi: https://doi.org/10.1007/s10339-012-0472-x
Bolinger, Dwight (1986). Intonation and its parts: Melody in spoken English. Stanford: Stanford University Press.
Bybee, J. (2006). From usage to grammar: the mind’s response to repetition. Language 82(4), 711-733.
Chomsky, N. (1986). Knowledge of language. Nueva York: Praeger
Cienki, A. (2015). Spoken language usage events. Language and Cognition 7, 499–514.
Croft, W. (2001). Radical Construction Grammar: Syntactic theory in typological perspective. Oxford y Nueva York: Oxford University Press.
Cruttenden, A. (1986). Intonation. Cambridge: Cambridge University Press.
Çetinçelik, M., Rowland, C. F., y Snijders, Tineke M. (2021). Do the eyes have it? A systematic review on the role of eye gaze in infant language development. Frontiers in Psychology Jan 8;11:589096. doi: https://doi.org/10.3389/fpsyg.2020.589096
Degutyte, Ziedune y Astell, Arlene (2021), The role of eye gaze in regulating turn taking in conversations: A systematized review of methods and findings. Frontiers in Psychology 12:616471. doi: https://doi.org/10.3389/fpsyg.2021.616471
Dimitrova, D., Chu, M., Wang, L., Özyürek, A. y Hagoort, P. (2016). Beat that word: How listeners integrate beat gesture and focus in multimodal speech discourse. Journal of Cognitive Neuroscience 28(9), 1255-1269.
Fillmore, Ch. J., Kay, P. y O’Connor, M. C. (1988). Regularity and idiomaticity in grammatical constructions: The case of let alone. Language 64(3), 501-538.
Fischer, K. y Aarestrup, M. (2021). Relationships between construction grammar(s) and genre: Evidence from an analysis of Instagram posts. Journal of Pragmatics 183, 87-104.
Goldberg, A. E. (2006). Constructions at work: The nature of generalization in language. Oxford y Nueva York: Oxford University Press.
Goldin-Meadow, S. y Brentari, D. (2017). Gesture, sign, and language: the coming of age of sign language and gesture studies. Behavioural Brain Sciences 40:E46. doi: https://doi.org/10.1017/S0140525X15001247
Hall, E. T. (1974). Handbook for proxemic research. Washington, D. C: Society for the Anthropology of Visual Communication.
Halliday, M. A. K. (1985). Spoken and written language. Victoria, NSW, Australia: Deakin University.
Hoffmann, T. (2021). Multimodal Construction Grammar: From multimodal constructs to multimodal constructions, en X. Wen y J. R. Taylor (eds.), The Routledge handbook of Cognitive Linguistics, pp. 78-92. Londres y Nueva York: Routledge.
Ibarretxe-Antuñano, I. y Valenzuela, J. (2021). Lingüística y cognición: el lenguaje desde la lingüística cognitiva. Madrid: Editorial Síntesis.
Ibbotson, P. (2013). The scope of usage-based theory. Frontiers in Psychology 4(255). doi: http://doi.org/10.3389/fpsyg.2013.00255
Ibbotson, P., Lieven, E. V. M., y Tomasello, M. (2013). The attention-grammar interface: Eye-gaze cues structural choice in children and adults. Cognitive Linguistics, vol. 24(3), 457-481. doi: https://doi.org/10.1515/cog-2013-0020
Jiménez-Bravo, M. y V. Marrero-Aguiar (2020). Multimodal perception of prominence in spontaneous speech: A methodological proposal using mixed models and AIC. Speech Communication 124, 28-45.
Kay, P. y Fillmore, Ch. (1999). Grammatical constructions and linguistic generalizations: The what’s X doing Y construction. Language 75(1), 1-33.
Kemmer, S. y Barlow, M. (2000). Introduction: A usage-based conception of language, en M. Barlow y S. Kemmer (eds.), Usage-based Models of Language, pp. 7-28. Stanford: CSLI Publications.
Kendon, A. (1967). Some functions of gaze-direction in social interaction. Acta Psychologica 26, 22-63. doi: https://doi.org/10.1016/0001-6918(67)90005-4.
Kendon, A. (2004). Gesture: Visible action as utterance. Cambridge: Cambridge University Press.
Ladefoged, P. y Johnson, K. (2011). A course in phonetics. 6ª edición. Boston, MA: Wadsworth.
Langacker, R. W. (2008). Cognitive grammar: A basic introduction. Oxford: Oxford University Press.
Lambrecht, K. (1990). What, me worry? — ‘Mad Magazine sentences’ revisited. Proceedings of the Sixteenth Annual Meeting of the Berkeley Linguistics Society 16, p. 215-228.
McAllister Byun, T. y Tessier, A. M. (2016). Motor influences on grammar in an emergentist model of phonology. Language and Linguistics Compass 10(9), 431-452.
McNeill, D. (2008). Gesture and thought. Chicago, IL: The University of Chicago Press.
Mompeán, J. A. y Valenzuela, J. (2019). Brexit means Brexit: a constructional analysis. Complutense Journal of English Studies 27, 1-37
Nota, N., Trujillo, J. P., y Holler, J. (2022). Conversational eyebrow frowns facilitate question identification: An online VR study. PsyArXiv. doi: https://doi.org/10.31234/osf.io/fcj8b
Pagán Cánovas, C. y Valenzuela, J. (2017). Timelines and multimodal constructions: Facing new challenges. Linguistic Vanguard 3(s1). doi: https://doi.org/10.1515/lingvan-2016-0087
Pagán Cánovas, C., Valenzuela, J., Alcaraz Carrión, D., Olza, I. y Ramscar, M. (2020). Quantifying the speech-gesture relation with massive multimodal datasets: Informativity in time expressions. PloS One 15(6). e0233892.
Perniss, P. (2018). Why we should study multimodal language. Frontiers in Psychology 9(1109). doi: https://doi.org/10.3389/fpsyg.2018.01109
Port, R. (2010). Rich memory and distributed phonology. Language Sciences, 32, 43-55.
Schoonjans, S. (2017). Multimodal Construction Grammar issues are Construction Grammar issues. Linguistic Vanguard 3(s1). doi: https://doi.org/10.1515/lingvan-2016-0050
Steen, F. F. y Turner, M. (2013). Multimodal Construction Grammar, en Michael Borkent, B. Dancygier y J. Hinnell (eds.), Language and the creative mind, pp. 255-274. Stanford, CA: CSLI Publications.
Thompson, S. A. (1998). A discourse explanation for the cross-linguistic differences in the grammar of interrogation and negation, en A. Siewierska y J. Jung Song (eds.), Case, typology and grammar: In honor of Barry J. Blake, pp. 307-341. Amsterdam: John Benjamins.
Tomasello, M. (2003). Constructing a language: A usage-based theory of language acquisition. Cambridge, MA: Harvard University Press.
Valenzuela, J. (2019). Señales no verbales en la comunicación: ¿problema o solución?. Ciencia Cognitiva 13(3), 71-73.
Valenzuela, J. (2017). A new multimodal construction? What do I care? International Cognitive Linguistics Conference (ICLC-14). University of Tartu (Estonia). July 11.
Valenzuela, J. y Hilferty, J. (2007). Music, modularity and syntax. International Journal of English Studies 7(2), 101-116.
Valenzuela, J. y Alcaraz-Carrión, D. (en prensa, 2023). English temporal gestures are temporal gestures.
Vigliocco, G., Perniss, P. y Vinson, D. (2014). Language as a multimodal phenomenon: Implications for language learning, processing and evolution. Philosophical Transactions of the Royal Society B 369(1651): 20130292. doi: https://doi.org/10.1098/rstb.2013.0292
Wagner, P., Malisz, Z. y S. Kopp (2014). Gesture and speech in interaction: An overview. Speech Communication 57, 209-232.
Zappavigna, M., Cléirigh, Ch., Dwyer, P. y Martin, J. R. (2009). The coupling of gesture and phonology, en M. Bednarek and J.R. Martin [eds.] New discourse on language: Functional perspectives on multimodality, identity, and affiliation, pp. 237-266. London: Continuum.
Zima, E. y Bergs, A. (2017). Multimodality and construction grammar. Linguistics Vanguard 3, s1, 2017, pp. 20161006. doi: https://doi.org/10.1515/lingvan-2016-1006
Zima, E. y B., Geert (2015). Cognitive Linguistics and interactional discourse: Time to enter into dialogue. Language and Cognition 7(4), 485-498.
Published
How to Cite
Issue
Section
Copyright (c) 2023 Revista Española de Lingüística
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
