multimodal discourse analysis

Reading papers: Cognitive linguistic and multimodal methods

Posted on February 21, 2020 by Sari Piittinen

The article I’ve read this time is “The visual basis of linguistic meaning and its implications for critical discourse studies: Integrating cognitive linguistic and multimodal methods” by Christopher Hart, published in Discourse & Society in 2016 (find it here). This is the first non-media studies article during this little article reading experiment of mine!

The disclaimer:
I don’t aim to write ‘academic reviews’ of these papers, but very informal and subjective thoughts about how these papers or chapters could influence my own work. I even avoid direct quotes of the text to also make it visually more apparent that this is not an academic text.

I was drawn to this paper because of my continued interest in multimodality and, well, the intriguing wording of ‘visual basis of linguistic meaning’. However, the ‘cognitive linguistic’ aspect of it intimidated me a little; the term ‘cognitive’ tends to make me think of all things psychological and things that happen inside a person’s mind, which is not really what I’ve been studying so far. Sure, I’ve analysed meanings produced in discourse, but I’ve tried to be careful not to step over the line to make suggestions about what people think when they are experiencing this or that. These are two different things.

I think partially because of the fear that I was stepping over into (to me) unexplored area of research it took me a lot longer to finish reading this paper than the previous ones. There’s nothing wrong with the paper or how it was written (on the contrary, it demonstrates its arguments and examples well). I’m just experiencing that good ol’ impostor syndrome and sometimes encountering something that isn’t immediately and easily understandable to me makes me question my intelligence, and that is destructive for my motivation to keep reading. Especially if I feel like it’s something that I should easily understand, which is the case here considering my educational background… Anyway. Just trying to be open about my challenges because how else to improve?

The paper discusses what cognitive linguistic critical discourse studies (CL-CDS) has to offer in combination with multimodal discourse analysis (MDA), how to explore connections between language, image and ideology, and how multimodal information is encoded by mental representations that are invoked by language use. Some of it is familiar to anyone who’s dealt with discourse studies before, but with suggested new uses and perspectives.

The paper gives a short introduction to MDA and how there’s been a need to develop a toolkit, like ones used in linguistics/grammatics, to describe the communication of ideas, attitudes and identities in non-linguistic forms. This is actually something I’ve had some methodological anxiety about, since I’m supposed to be analysing gestures, expressions, etc. in video material in the near future. It seems very difficult to find tools for describing what different non-linguistic means of expressing meaning might actually mean. If someone has a sarcastic smile on their face, am I allowed to suggest it’s a sarcastic smile, and if so, based on what reasoning? In linguistics, there are specific grammatical means and contextual information that we can use to support our arguments. I’m still struggling to understand how to do that credibly with non-linguistic communication. This paper, at least, helps orientate my thinking somewhat, for example, through the conceptualizations of image as co-text and context: how linked language usage and images match, and how previously encountered semiotic experiences reflect in simulated experiences.

One especially interesting part for me was the argument concerning how the encoding of orientational values, at the level of simulation, encourages us to experience an event as if we were in the shoes of the agent or patient (in a text). So for example, a person or group involved in an event (like being threatened by a shooter) can be positioned as the patient – as the one on the receiving end of an action – so that the viewer interprets the action as a personal threat rather than something neutrally reported. Basically, by deciding whom to portray as the agent or patient we’re also making decisions of who are to be perceived as ‘one of us’ or as ‘other’, and something like danger can be perceived as personal or impersonal. (At least that’s how I understand it.) Doing this kind of analysis in language is familiar to me, but in images, not so much. I see the benefits of using this kind of approach to the analysis of news media, in particular, but not necessarily the kind of data that I have currently collected, aside from acknowledging the positioning of participants as performers or audience and what that suggests about their perceived importance for the content. However, since we’re already speaking of simulation, I wonder if someone would one day do a close analysis of, for example, how different types of game characters are visually positioned during scenes and still images (when the player does not control the camera) and what this suggests about their ideological connotations.

I probably failed to discuss the main points that the article strove to make, but like I said, I’ve been going through some challenges and am still taking it all in. At least I managed to blog about it!

Method readings: Multimodal discourse analysis

Posted on January 17, 2020 by Sari Piittinen

A recent reading on research methodology that has turned out to be rather impactful to me is Norris’s new book, Systematically Working With Multimodal Data: Research Methods in Multimodal Discourse Analysis, published just 2019. For a good while now, I’ve been trying to find a toolkit for making sense of how meanings are communicated on videos – a systematic toolkit that would help me deal with the overwhelming material that one video can contain. While first reading it, I had somewhat mixed feelings about multimodal discourse analysis (MDA). On one hand, it was intimidating to read about the level of detail in transcription that noted someone’s hand going up and down three times when they scratched their nose – that is, I’m not sure whether the level of detail that MDA goes into in transcribing lower-level mediated actions (‘a mode’s smallest pragmatic meaning unit’) is what I’m looking for in my research. On the other hand, now that I know of such a systematic and detailed way of transcribing and noting actions in videos, it seems impossible not to want to use it for absolutely everything (although perhaps combined with some other approach). And it does help that MDA guides you to focus on multimodally transcribing only the parts that are relevant to answering your research question, since it would take too much time and might even be counter productive to transcribe the whole video. The book specifically warns you not to start immediately transcribing (especially speech because this puts the focus on spoken language), but to first to take notes, get an overall understanding of the present higher-level mediated actions (for example, Skyping is one), and then decide which parts are relevant for a detailed micro analysis.

The step-by-step guidelines offered by the book are clear and helpful, but also a bit overwhelming initially, since when the process is broken down to pieces, it seems there are endless steps to the analysis. Once one actually starts doing it, it may not turn out to be that bad, especially after some experience and having figured out, for example, which programs to use for transcription.

I’m almost certain that I’ll end up making use of MDA as suggested by Norris sooner or later; I’ve included it as part of the methodology in my recent chapter proposals and funding applications because although MDA alone doesn’t seem enough to answer the questions I aim to answer, it seems such a valuable tool for working with video data.