Things have been a bit busy lately, so I’ve had less time to devote to blogging (suggested topics always welcomed, FYI). I’ve been hard at work on several revisions of papers for journals, as well as prepping several more papers for submissions. I’m also gearing up to start running a new brainwave study here at UCSD, so that’s exciting to have underway.

I’ve also been hard at work on a major redesign/relaunch of this website. The site is getting a new look, new organization, and even a new address. If you’d like to plan ahead for the switch, the new url will be: www.visuallanguagelab.com (currently a redirect to the present site, which will then be flipped on relaunch).

Finally, I’m happy to report that my upcoming book, The Visual Language of Comics, has now entered the production stage! It will be fun to see the proofs in a few weeks. For now though, it’s exciting to see that my publisher has now created a webpage promoting the book, including a growing list of endorsements. Looks like there’s even a page on amazon for it. Let the countdown until it’s release in December begin!

For those that might be interested, one of my projects is trying to organize a book summarizing important research on visual narrative. This book will be a companion volume to my book out later this year, The Visual Language of Comics. If you may be interested in contributing, here’s a Call For Papers for it…

CFP: Interdisciplinary approaches to visual narrative

While there have been a growing number of books on comics in recent years, very few have addressed aspects of structure, particularly from theoretical, cognitive, or experimental points of view and outside the realm of literary or sociocultural theory. I am working to organize a compilation of important papers on the understanding of sequential images. Most of the chapters will be either 1) summary papers that provide extensive bibliographies that can provide an overview to students and a resource to other researchers, or 2) reprints of significant research that remain under-recognized or hard-to-find.

This Call for Papers asks for proposals for papers of two types of chapters focused particularly on research outside of English, presented for an English speaking audience:

1. Chapters that summarize, in English, advances in comic theory from non-English speaking researchers. Such chapters should be large in scope with extensive reference sections.

2. Translations into English of significant non-English comic theory (structural, cognitive, experimental, etc.) from important papers or book chapters.

Topics or chapters outside this scope may be considered, though best to contact me directly with inquiries. (Of interest may be: review papers of other types, historical development of comic “symbology”, empirically grounded discussions of differences between comics cross-culturally, etc.). Importantly, papers should be relevant not only to scholars of comics, but also to linguists, cognitive scientists, and psychologists.

Contributor Guidelines

1. Abstracts of 400-500 words accepted. Papers of 5000-9000 words, including notes and bibliography, accepted. Please also include a short biographical statement.

2. All documents should be submitted as Word or Word-compatible files. PDFs are also acceptable.

3. Submission deadline: May 15, 2013. June 15, 2013

4. Materials should be sent to me via email (Neil Cohn)

I’m often very excited when I find new research on comics, especially when it’s experimental. There is so little done that it’s a treat to find something I didn’t know about. Unfortunately, sometimes my excitement at new data turns sour when I see what was actually done…

I recently found this study (pdf) by Kinzer and colleagues that compares the comprehension and eye movements of readers for narratives in comics and video games. Their main goal is to help provide support for the use of comics and video games in educational contexts.

In this study, they presented sixth graders with either a video game version of a story or a comic created from the images of the video game. Overall, they find that participants understood the story in the video game version better than the comic version. They also found people spent more time engaged with the game than the comic.

I would take all of these findings with a grain of salt…

…because the stimuli appear to be extremely confounded because the comic versions of the story appear to be so badly created. Judging by the example in the text, the comic pages clearly were created by someone who had no real fluency in the visual language of comics. This is clear at a glance just by the example that they provide in the paper:

First off, the images make it extremely hard to tell what’s going on. Second, almost all of the balloons are placed outside of their originating panels to the extent that they completely overlap in panels forward and backward in the text. I don’t even need to know what’s in the text to know this will be confusing to a reader. This is so “illegal” in the rules of page construction that it is almost painful.

If this is their example (what is probably the best example they have), I shudder to think what other pages in the experiment look like. Seriously, if I wanted to design an experiment that had “incomprehensible comics pages” as one type of stimuli, I’d use pages like these.

It’s no wonder they found that their participants had poorer comprehension for the comic version—their stimuli are the equivalent of trying to test English comprehension while using broken English. It tells you next to nothing of interest.

There are two main points I’d like to make about this:

First, good experiments are hard to design, and having something worth saying must follow from having successfully designed an experiment that can give you good information. It pays to be critical as a creator and reader of scientific research (no matter what the topic).

Second, doing experiments using the visual language of comics is not trivial. Stimuli cannot be created by anyone, regardless of their fluency in comic creation. Just because you can throw together some images and words into panels on a page does not mean you’ve successfully created an example of “native” visual language. Believing otherwise does a disservice to yourself and to others who might read and cite your research.

Kinzer, C. K., Turkay, S., Hoffman, D. L., Gunbas, N., Chantes, P., Chaiwinij, A., & Dvorkin, T. (2012). Examining the Effects of Text and Images on Story Comprehension: An Eye-Tracking Study of Reading in a Video Game and Comic Book. In P. J. Dunston, S. K. Fullerton, C. C. Bates, K. Headley, & P. M. Stecker (Eds.), Literacy Research Association Yearbook 61 (pp. 259-275). LRA: Chicago, IL.

I am happy to announce I have a new paper published in the journal Frontiers in Cognitive Science that outlines a theory of how people navigate page layouts. A previous version of this paper was posted here several years ago, though this is now the final version. You can download a pdf of the paper here: “Navigating Comics.”

The paper describes an experiment I conducted that examined people’s intuitions for moving through page layouts. I then incorporate the results into a larger theory of the structure of page layout. Here’s the official abstract:

Like the sequence of words in written language, comic book page layouts direct images into a deliberate reading sequence. Conventional wisdom would expect that comic panels follow the order of text: left-to-right and down—a “Z-path”—though several layouts can violate this order, such as Gestalt groupings of panels that deny a Z-path of reading. To examine how layouts pressure readers to choose pathways deviating from the Z-path, we presented participants with comic pages empty of content, and asked them to number the panels in the order they would read them. Participants frequently used strategies departing from both the traditional Z-path and Gestalt groupings. These preferences reveal a system of constraints that organizes panels into hierarchic constituents, guiding readers through comic page layouts.

Cohn, Neil. 2013. Navigating Comics: An empirical and theoretical approach to strategies of reading comic page layouts. Frontiers in Cognitive Science. 4: 1-15.

Just one more pleas for helping my colleague… www.nalinineedsyou.com. Please watch, repost, and help us save the life of an amazing woman:

This is not going to be your normal post on comics and visual language. This post is a plea for help from anyone who might read this blog and anyone compelled to help save someone’s life. I summarize here much of what’s said in a blog post from my colleague Sam Sommers.

From 2006 to 2012 I was a graduate student at Tufts University. One of the professors there was Nalini Ambady, who happened to be the advisor of many of my closest friends. You probably know Nalini’s research in social psychology, though you may not have heard of her. Have you heard of Malcom Gladwell’s book Blink, which talks about the “ability to extract an enormous amount of meaningful information from the very thinnest slice of experience”? The idea of “thin slices” comes from Nalini’s work.

I am sad to say now that Nalini is battling with leukemia and is dire need of a bone marrow transplant. Because she is Indian, she has a very specific genetic marker that likely needs a match with another South Asian. Time is of the essence—she likely needs a match within 8 weeks. Her story is detailed much more at this website: http://www.helpnalininow.org

Helping save this woman’s life is easy:

1) Register today as a potential bone marrow donor in the national registry. It’s easy: if you’re between the ages of 18-44 you can simply go here: http://marrow.org/Join/Join_Now/Join_Now.aspx?promo_code=MatchNalini 

Make sure to enter the promo code “nalini” and your request for a cheek swab will be rushed to you and its processing expedited.

2) Everyone has the potential to save a life by registering on the site. In Nalini’s case, though, it’s particularly Indian donors who are likely to be a match. Accordingly, please forward this blog post or the http://www.helpnalininow.org website to any websites, email lists, or organizations with large South Asian memberships.

Please help. It takes just a few clicks of a mouse and time is running out. Your effort may help save a woman’s life. Thank you.

I am happy to announce I have a new paper published in the journal Cognitive Science that outlines my theory of sequential image understanding. I argue that approaches like “panel transitions” cannot account for the creation of meaning in sequential images, and I offer a new theory of “Visual Narrative Grammar” that better accounts for how we comprehend visual narratives.

You can download this study along with my other papers, or directly to the pdf here.

This new theory uses narrative categories similar to traditional notions of narrative (though operationalized) that are organized into hierarchic constituents. The basics of this theory are outlined, along with diagnostic methods for testing these categories and constituents. Finally, I outline how this theory can apply beyond the sequential images found in comics to the understanding of film and verbal discourse.

This paper presents the basics of this theory for the first time in a cohesive paper, though this paper actually only consists of part of the broader theory of Visual Narrative Grammar. I hope to discuss the theory in full in subsequent papers and books. I should note also that my experiments on the cognition of understanding comics (also available for download) use this theory as their basis.

Here’s the full abstract:

Narratives are an integral part of human expression. In the graphic form, they range from cave paintings to Egyptian hieroglyphics, from the Bayeux Tapestry to modern day comic books (Kunzle, 1973; McCloud, 1993). Yet not much research has addressed the structure and comprehension of narrative images, for example, how do people create meaning out of sequential images? This piece helps fill the gap by presenting a theory of Narrative Grammar. We describe the basic narrative categories and their relationship to a canonical narrative arc, followed by a discussion of complex structures that extend beyond the canonical schema. This demands that the canonical arc be reconsidered as a generative schema whereby any narrative category can be expanded into a node in a tree structure. Narrative “pacing” is interpreted as a reflection of various patterns of this embedding: conjunction, left-branching trees, center-embedded constituencies, and others. Following this, diagnostic methods are proposed for testing narrative categories and constituency. Finally, we outline the applicability of this theory beyond sequential images, such as to film and verbal discourse, and compare this theory with previous approaches to narrative and discourse.

Cohn, Neil. (2013). Visual Narrative Structure. Cognitive Science, 37 (3), 413-452 DOI: 10.1111/cogs.12016

Via this article I stumbled onto this dissertation which promotes using comics in educational contexts (a topic I am very interested in).

In one of the chapters of the thesis, it looks at the understanding of comics and includes a “neuroscience” section. Now, even when I disagree with them, I am always one to encourage research and writing on the structure and comprehension of comics (and I am wholly in support of the effort that this thesis is trying to make with regard to graphic textbooks). However, I have also railed on the invocation of neuroscience when used inappropriately.

Given my commitment to educating about comics and the brain, it behooves me to explain exactly what is wrong with a discussion like this, and especially what is misleading/wrong with an image like this:

This graphic—and the accompanying description—reflect the worst of misunderstandings and invocations about the brain. Here are among the problems:

1) It suffers from the “localization” fallacy. Contrary to what the image implies, there is not a “speech” location in the brain. Nor is there a “writing” or “vision” location in the brain. The brain does not localize full behaviors in the brain in modular locations. Rather, the brain is highly interconnected, with portions of the brain devoted to particular functions that interface to produce complex behaviors like speech, language, or vision.

For example, while the occipital cortex (back of the head) is the “primary” visual area, the visual system alone goes up through the top of the cortex (the “dorsal stream”), down into the bottom, temporal lobe (the “ventral stream”), and connections from the eyes go through the middle of the brain on its way to the occipital lobe in the first place. In other words: the visual system engages lots of different parts of the brain, not just back of the head. (…and where is the rest of the brain in this image???)

Reducing the brain into localized areas for each of these complex behaviors displays a lack of understanding about how the brain works (at least, to the extent we understand it so far) and for how these complex behaviors work (“language” alone can be subdivided into at least four different major substructures, which each have more substructures). Beyond this…

2) Discussing the brain is entirely unnecessary in a thesis like this. The overall point that this thesis is trying to make is that comics are very complex and involve numerous interacting parts that we understand almost effortlessly. This point does not rely on discussion of the brain to come across. In fact, I myself have done quite a lot of work describing the complexity of structure found in comics, some of which discusses the brain, and some does not.

This thesis does not need it, and including a poorly-understood discussion of the brain only hurts its overall point. Now, studies have shown that just including an image of the brain somehow convinces people that its point is more believable. However, unless the thesis is actually about the brain, such discussion is unnecessary and borderline dishonest (albeit unintentionally). Which leads us to…

3) If you’re going to talk about comics and the brain, at least do the basic research to discuss what work has actually been done on this topic. Granted, there are a limited number of studies that have directly examined this issue. However, they are out there, and reviews of some of these papers can be found on this very blog: Here, here, here, and here.

I myself have now done three studies looking at comics and the brain. I’m currently writing up my latest two studies (which were my dissertation), but my first experiment is online here: Full pdf article, Short “comic” version.

I’ll close by reiterating what I said in my previous post on this topic:

My point overall is this: as cool and interesting as it is, not all arguments need to be tied to the brain and cognition. And, in fact, some arguments are made weaker by doing so, since appealing to neuroscience is unnecessary at best and hand-waving at worst.

Figure out what your point is and talk about it. I’m guessing it actually has little to do with neuroscience directly.

Here’s another great recent Savage Chickens comic by Doug Savage. I’ve commented on this great strip before, and here’s another good “meta”-comic:

As we all know, hearts are symbolic of love (especially over the head), while lightbulbs over the head represent inspiration. I call “over the head” symbols like these “upfixes” since they are like affixes that are “up” (a term coined by my mentor, Ray Jackendoff).

Of course, as in the last panel, a heart and a lightbulb together show a love of lightbulbs, not a love of inspiration or ideas. This combination also does not give you inspiration about love either. What’s interesting about Savage’s observation here is that it nicely shows that you cannot combine the upfixes together. This is a first interesting trait: these upfixes have meaning on their own, but not in combination.

Related to this, the nature of the lightbulb upfix is to give you “value added” for its meaning. It no longer is just a lightbulb, but out of the relation above the head generates a new conventionalized meaning of inspiration. By adding the heart, it effectively removes this additive meaning, making the lightbulb simply a lightbulb once again.

The heart also changes meaning a bit as well. When the heart is an upfix, it describes the mental state of the person: The chicken is in love—with what, it doesn’t matter. However, in the third panel, their combination makes the heart modify the lightbulb now—it’s a love of lightbulbs—not merely reflecting a general mental state of the lover (the chicken).

Finally, this combination also changes the thought bubble. In the first two upfixes, the thought bubble mostly gives a depicted link between head and upfix. It doesn’t mean thinking about love or about inspiration, but just reinforces these signs as being mental states. However, it is mostly unnecessary. The heart or the lightbulb would retain their meaning without the thought bubble. In the combination though, the bubble now returns to it’s usual meaning as encapsulating thoughts. Having a heart and a lightbulb floating above the head wouldn’t work as an upfix, nor would it work to convey thoughts.

Altogether, this simple, quirky comic tells us a lot about the structure of these types of signs!

In Eddie Campbell’s recent article at The Comics Journal, he described several potential “rhetorical rules” that authors of comics can follow in order to make them more understandable to inexperienced readers. In a previous post, I tackled the broader issue of what factors go into limitations to understanding. In this post, I discuss what scientific research tells us about his actual stated rules.