Update: We have now processed a total of 2 million tweets (up from 1 million).

My colleague Hemant Purohit at QCRI has been working with us on automatically extracting needs and offers of help posted on Twitter during disasters. When the 2-mile wide, Category 4 Tornado struck Moore, Oklahoma, he immediately began to collect relevant tweets about the Tornado’s impact and applied the algorithms he developed at QCRI to extract needs and offers of help.

As long-time readers of iRevolution will know, this is an approach I’ve been advocating for and blogging about for years, including the auto-matching of needs and offers. These algorithms (classifiers) will also be made available as part of our Artificial Intelligence for Disaster Response (AIDR) platform. In the meantime, we have contacted our colleagues at the American Red Cross’s Digital Operations Center (DigiOps) to offer the results of the processed data, i.e., 1,000+ tweets requesting & offering help. If you are an established organization engaged in relief efforts following the Tornado, please feel free to get in touch with us (patrick@iRevolution.net) so we can make the data available to you. 

As part of QCRI’s Artificial Intelligence for Monitoring Elections (AIME) project, I liaised with Kaggle to work with a top notch Data Scientist to carry out a proof of concept study. As I’ve blogged in the past, crowdsourced election monitoring projects are starting to generate “Big Data” which cannot be managed or analyzed manually in real-time. Using the crowdsourced election reporting data recently collected by Uchaguzi during Kenya’s elections, we therefore set out to assess whether one could use machine learning to automatically tag user-generated reports according to topic, such as election-violence. The purpose of this post is to share the preliminary results from this innovative study, which we believe is the first of it’s kind.

The aim of this initial proof-of-concept study was to create a model to classify short messages (crowdsourced election reports) into several predetermined categories. The classification models were developed by applying a machine learning technique called gradient boosting on word features extracted from the text of the election reports along with their titles. Unigrams, bigrams and the number of words in the text and title were considered in the model development. The tf-idf weighting function was used following internal validation of the model.

The results depicted above confirm that classifiers can be developed to automatically classify short election observation reports crowdsourced from the public. The classification was generated by 10-fold cross validation. Our classifier is able to correctly predict whether a report is related to violence with an accuracy of 91%, for example. We can also accurately predict  89% of reports that relate to “Voter Issues” such as registration issues and reports that indicate positive events, “Fine” (86%).

The plan for this Summer and Fall is to replicate this work for other crowdsourced election datasets from Ghana, Liberia, Nigeria and Uganda. We hope the insights gained from this additional research will reveal which classifiers and/or “super classifiers” are portable across certain countries and election types. Our hypothesis, based on related crisis computing research, is that classifiers for certain types of events will be highly portable. However, we also hypothesize that the application of most classifiers across countries will result in lower accuracy scores. To this end, our Artificial Intelligence for Monitoring Elections platform will allow election monitoring organizations (end users) to create their own classifiers on the fly and thus meet their own information needs.

Big thanks to Nao for his excellent work on this predictive modeling project.

In 2010, Russian volunteers used social media and a live crisis map to crowdsource their own disaster relief efforts as massive forest fires ravaged the country. These efforts were seen by many as both more effective and visible than the government’s response. In 2011, Egyptian volunteers used social media to crowdsource their own humanitarian convoy to provide relief to Libyans affected by the fighting. In 2012, Iranians used social media to crowdsource and coordinate grassroots disaster relief operations following a series of earthquakes in the north of the country. Just weeks earlier, volunteers in Beijing crowd-sourced a crisis map of the massive flooding in the city. That map was immediately available and far more useful than the government’s crisis map. In early 2013, a magnitude 7  earthquake struck Southwest China, killing close to 200 and injuring more than 13,000. The response, which was also crowdsourced by volunteers using social media and mobile phones, actually posed a threat to the Chinese Government.

“Wang Xiaochang sprang into action minutes after a deadly earthquake jolted this lush region of Sichuan Province [...]. Logging on to China’s most popular social media sites, he posted requests for people to join him in aiding the survivors. By that evening, he had fielded 480 calls” (1). While the government had declared the narrow mountain roads to the disaster-affected area blocked to unauthorized rescue vehicles, Wang and hitchhiked his way through with more than a dozen other volunteers. “Their ability to coordinate — and, in some instances, outsmart a government intent on keeping them away — were enhanced by Sina Weibo, the Twitter-like microblog that did not exist in 2008 but now has more than 500 million users” (2). And so, ”While the military cleared roads and repaired electrical lines, the volunteers carried food, water and tents to ruined villages and comforted survivors of the temblor [...]“ (3). Said Wang: “The government is in charge of the big picture stuff, but we’re doing the work they can’t do” (4).

In response to this same earthquake, another volunteer, Li Chengpeng, “turned to his seven million Weibo followers and quickly organized a team of volunteers. They traveled to the disaster zone on motorcycles, by pedicab and on foot so as not to clog roads, soliciting donations via microblog along the way. What he found was a government-directed relief effort sometimes hampered by bureaucracy and geographic isolation. Two days after the quake, Mr. Li’s team delivered 498 tents, 1,250 blankets and 100 tarps — all donated — to Wuxing, where government supplies had yet to arrive. The next day, they hiked to four other villages, handing out water, cooking oil and tents. Although he acknowledges the government’s importance during such disasters, Mr. Li contends that grass-roots activism is just as vital. ‘You can’t ask an NGO to blow up half a mountain to clear roads and you can’t ask an army platoon to ask a middle-aged woman whether she needs sanitary napkins, he wrote in a recent post” (5).

As I’ve blogged in the past (here and here, for example), using social media to crowdsourced grassroots disaster response efforts serves to create social capital and strengthen collective action. This explains why the Chinese government (and others) faced a “groundswell of social activism” that it feared could “turn into government opposition” following the earthquake (6). So the Communist Party tried to turn the disaster into a “rallying cry for political solidarity. ‘The more difficult the circumstance, the more we should unite under the banner of the party,’ the state-run newspaper People’s Daily declared [...], praising the leadership’s response to the earthquake” (7).

This did not quell the rise in online activism, however, which has “forced the government to adapt. Recently, People’s Daily announced that three volunteers had been picked to supervise the Red Cross spending in the earthquake zone and to publish their findings on Weibo. Yet on the ground, the government is hewing to the old playbook. According to local residents, red propaganda banners began appearing on highway overpasses and on town fences even before water and food arrived. ‘Disasters have no heart, but people do,’ some read. Others proclaimed: ‘Learn from the heroes who came here to help the ones struck by disaster’ (8). Meanwhile, the Central Propaganda Department issued a directive to Chinese newspapers and websites “forbidding them to carry negative news, analysis or commentary about the earthquake” (9). Nevertheless, “Analysts say the legions of volunteers and aid workers that descended on Sichuan threatened the government’s carefully constructed narrative about the earthquake. Indeed, some Chinese suspect such fears were at least partly behind official efforts to discourage altruistic citizens from coming to the region” (10).

Aided by social media and mobile phones, grassroots disaster response efforts present a new and more poignant “Dictator’s Dilemma” for repressive regimes. The original Dictator’s Dilemma refers to an authoritarian government’s competing interest in using information communication technology by expanding access to said technology while seeking to control the democratizing influences of this technology. In contrast, the “Dictator’s Disaster Lemma” refers to a repressive regime confronted with effectively networked humanitarian response at the grassroots level, which improves collective action and activism in political contexts as well. But said regime cannot prevent people from helping each other during natural disasters as this could backfire against the regime.

See also:

 •  How Civil Disobedience Improves Crowdsourced Disaster Response [Link]

Things have been a bit busy lately, so I’ve had less time to devote to blogging (suggested topics always welcomed, FYI). I’ve been hard at work on several revisions of papers for journals, as well as prepping several more papers for submissions. I’m also gearing up to start running a new brainwave study here at UCSD, so that’s exciting to have underway.

I’ve also been hard at work on a major redesign/relaunch of this website. The site is getting a new look, new organization, and even a new address. If you’d like to plan ahead for the switch, the new url will be: www.visuallanguagelab.com (currently a redirect to the present site, which will then be flipped on relaunch).

Finally, I’m happy to report that my upcoming book, The Visual Language of Comics, has now entered the production stage! It will be fun to see the proofs in a few weeks. For now though, it’s exciting to see that my publisher has now created a webpage promoting the book, including a growing list of endorsements. Looks like there’s even a page on amazon for it. Let the countdown until it’s release in December begin!

My colleagues and I at QCRI and the Masdar Institute will be launching Verily in the near future. The project has already received quite a bit of media coverage—particularly after the Boston marathon bombings. So here’s an update. While major errors were made in the crowdsourced response to the bombings, social media can help to find quickly find individuals and resources during a crisis. Moreover, time-critical crowdsourcing can also be used to verify unconfirmed reports circulating on social media.

The errors made following the bombings were the result of two main factors:

(1) the crowd is digitally illiterate
(2) the platforms used were not appropriate for the tasks at hand

The first factor has to do with education. Most of us are still in Kindergarden when it comes to the appropriate use social media. We lack the digital or media literacy required for the responsible use of social media during crises. The good news, however, is that the major backlash from the mistakes made in Boston are already serving as an important lesson to many in the crowd who are very likely to think twice about retweeting certain content or making blind allegations on social media in the future. The second factor has to do with design. Tools like Reddit and 4Chan that are useful for posting photos of cute cats are not always the tools best designed for finding critical information during crises. The crowd is willing to help, this much has been proven. The crowd simply needs better tools to focus and rationalize to goodwill of it’s members.

Verily was inspired from the DARPA Red Balloon Challenge which leveraged social media & social networks to find the location of 10 red weather balloons planted across the continental USA (3 million square miles) in under 9 hours. So Verily uses that same time-critical mobilization approach—negative incentive recursive mechanism—to rapidly collect evidence around a particular claim during a disaster, such as “The bridge in downtown LA has been destroyed by the earthquake”. Users of Verily can share this verification challenge directly from the Verily website (e.g., Share via Twitter, FB, and Email), which posts a link back to the Verily claim page.

This time-critical mobilization & crowdsourcing element is the first main component of Verily. Because disasters are far more geographically bounded than the continental US, we believe that relevant evidence can be crowdsourced in a matter of minutes rather than hours. Indeed, while the degree of separation in the analog world is 6, that number falls closer to 4 on social media, and we believe falls even more in bounded geographical areas like urban centers. This means that the 20+ people living opposite that bridge in LA are only 2 or 3 hops from your social network and could be tapped via Verily to take pictures of the bridge from their window, for example.

The second main component is to crowdsource critical thinking which is key to countering the spread of false rumors during crises. The interface to post evidence on Verily is modeled along the lines of Pinterest, but with each piece of content (text, image, video), users are required to add a sentence or two to explain why they think or know that piece of evidence is authentic or not. Others can comment on said evidence accordingly. This workflow prompts users to think critically rather than blindly share/RT content on Twitter without much thought, context or explanation. Indeed, we hope that with Verily more people will share links back to Verily pages rather than to out of context and unsubstantiated links of images/videos/claims, etc.

In other words, we want to redirect traffic to a repository of information that incentivises critical thinking. This means Verily is also looking to be an educational tool; we’ll have simple mini-guides on information forensics available to users (drawn from the BBC’s UGC, NPR’s Andy Carvin, etc). While we’ll include dig ups/downs on perceived authenticity of evidence posted to Verily, this is not the main focus of Verily. Dig ups/downs are similar to retweets and simply do not capture/explain whether said digger has voted based on her/his expertise or any critical thinking.

If you’re interested in supporting this project and/or sharing feedback, then please feel free to contact me at any time. For more background information on Verily, kindly see this post.

 

What is the future of art? What media will it use? Computers, obviously. Information technology is very good at imitating old media: drawing programs, music programs, word processors designed for playwrights or authors. But none of these tap into the intrinsic strengths of the computer, able to do something no other medium can: simulate. Bret Victor, the man so demanding of user interfaces he left Apple, is dissatisfied with the tools available to artists that allow them to simulate. So he made his own, and gave a one-hour talk on it.

Those interested should definitely take the time to watch it, but to summarize, he demonstrates the power of simulation in creating art that is part animation and part performance, with the human and computer reacting to one another. He then lifts the curtain and show us the tools he used to simulate the characters in the scene, and it’s not code. Instead, it’s a drawing program, with lines and shapes, that he uses to define behavior. Code, he points out, is based on algebra, but his system is based on geometry. Finally, he concludes with a short performance that he built with these tools. Higher is the story of earth, from the stars to cells to civilization to space travel back to the stars.

What blew my mind about Higher is that a few years ago, I had independently created a short film on exactly that topic, with exactly the same background music (Kyle Gabler’s Best Of Times from World of Goo). Victor’s piece was far more polished, but we had both been inspired by the same music to express the same idea, the journey of life to the stars. Remember when I complained about not finding people who shared my narrative? So this is what that feels like.

What drove Victor to create his tools was the belief that art is an attempt to communicate that which cannot be put into words. By binding simulation to lingual code, we make it inaccessible and unsuitable for art and artists. Direct manipulation of the art, which is how art has been created going back to cave paintings, allows the artist to interact with and lend emotion to the art in ways not possible through code’s layer of indirection, of abstraction.

The reason artists’ needs have been neglected by developers is that, for the rest of the world, code works just fine. As I’ve previously blogged, language is one of humankind’s most powerful inventions. The direct manipulation that is liberating to the artist is confining to the engineer. Language is how we manage many layers of abstraction at once; without it we are reduced to pointing and grunting. It’s harder to communicate with a computer in code than a well-designed direct manipulation interface, but code is more powerful. In the sciences, a good result is consistent with what is already known; in art, a good piece is unexpected and shakes our established worldview. More fundamentally, the sciences observe and record some objective outside truth; art looks inward to offer one of many interpretations of the subjective human experience.

This tension that we see between science and art also shows up in schools. In a recent TED talk, Sir Ken Robinson extols diversity as a fundamental human trait, which schools attempt to erase and replace with standardization. We agree that standardization has its place, but I personally think he downplays its importance. Standardization is writing, is language; those things can’t happen without common ways of thinking. At first, children need to explore concepts and use their own terms, without a top-down lesson plan imposed by school administrators. Nevertheless, the capstone is always learning what the rest of the world calls it. That isn’t smashing creativity, but rather empowering the child to learn more about the topic from others and from reference sources. It’s creating a minimum level of knowledge common every adult member of society, which is assumed by all media. Being able to communicate  facts with others isn’t just the result of education, it’s what makes education possible in the first place. With language, groups of people can unambiguously refer to things not present, a shared imagination. Verbalization is a form of abstraction.

Let’s get back to the role of diversity in school. Students should be able to explore what interests them, but the converse is not true: some topics must be taught to everyone, even if some people do not find them interesting. This is especially true before high school. I know you’re not passionate about fractions, Little Johnny, but you need to learn them. Society expects everyone to have a minimum level of competence in every subject. Additionally, passion for a field isn’t always “love at first sight”. The future mathematician isn’t always the first in the class to get basic arithmetic.

Although the curriculum needs to be largely standardized, the pedagogy does not. The neglect of diversity in schools is most heavily felt not in what kids are or are not learning, but how they are learning it. The inflexibility imposed on lesson plans is degrading to teachers and failing our kids. Teachers should be trusted to adapt lessons to their class, and empowered with testing results they find useful, early enough to use it. Standardized testing as it exists today does not fit the bill. Every student needs to achieve the same core competencies, but the paths to doing so will be as diverse as the children themselves. A broad exposure to both methods and topics promotes the development not just of knowledge, but of personality and identity. The reason to have art in school isn’t to improve test scores but because it’s part of being human.

To be more precise, we should distinguish between “the arts” and “art”. The arts are how to create with the media classically used for art: paint, music, poetry, drama, dance, and so on. Like any other discipline, the arts require a standardized language to record and transfer this knowledge. Sometimes it’s plain English, sometimes it’s jargon, sometimes it’s symbols, but it’s still an agreed-upon abstraction. Diversity of ideas expressed in the language is inventive and healthy; diversity of the language itself is nonstandard and chaotic. With this in mind, the arts take their place at one end of a spectrum of knowledge: mathematics, natural science, social science, and history. And the arts.

But art is something entirely different. It is the personal and emotional perception of an experience that communicates without words. Art is direct and concrete; it is subjective and sublime. Much of the arts attempt to create art. Victor’s tools advance the arts; what he creates with them is art.

It’s a defensible position to say that art, because it does not rely on language as all the other fields of knowledge do, is not knowledge at all. But I’ll indulge Victor and say that not all knowledge can be verbalized. That doesn’t mean that art is beyond classification; Victor and I saw the same artistic ideas in the same piece of lyricless music. Conversely, just because something is written down doesn’t mean it’s standardized or useful knowledge. Recently, the mathematics community has been bewildered by an inscrutable set of papers which claim to prove a fundamental piece of number theory. No one can decipher them to tell if the proof is valid, and their author has not been forthcoming with an oral explanation. So in extreme cases, the analogy between language and standardization breaks down. The wordless expression is more coherent than words.

For all the knowledge that abstract language has brought us, ineffable art remains part of the human experience. It is important for our children to learn about art to become mature and thoughtful adults. It is equally important for us to provide tools that support the nonverbal side of thought, to engage the visual and auditory parts of our brains in ways words never can. These are the same failure: the refuge in abstraction, the desire to have everything neat and orderly and predictable. Art exists to explore ambiguity and paradox; it does not demand simple answers but asks complex questions.

A lot of futurists imagine a time when technology makes everything easy. There is a faith in technological convergence, where everything speaks the same language and interacts intelligently and flawlessly. But historically we see technologies become incompatible. If there’s an open standard underneath, such as email, you still get dozens of providers and clients; and if there’s not, you get the walled gardens of social media, loosely tied together by third-party “integration”. What’s important to realize is that the path of technology is not fixed. Our gadgets don’t have to make us more productive and connected; they can make us more artistic and provide privacy, if we design them so. We should stop aspiring to a monoculture of technology because, not only will it not happen for technical and economic reasons, it shouldn’t happen. Standardized technology leads to standardized thinking, especially when coupled with standardized social institutions. Creativity is  not only what drives technology further, but art and humanity as well.

The Khan Academy dashboard is meant to provide students and teachers with information that can help them target where a student is struggling, and improve. Unfortunately, the data given isn’t what’s useful to teachers, just what’s easy for computers to measure. These metrics include time spent watching and rewinding videos, time spent on different topics (broken up by videos and quizzes), and proficiency levels in exercises. But as one of the teachers I follow on Twitter points out, these programs don’t let him “SEE my students’ work so I can know HOW/WHY they got questions wrong.”

Mathmistakes.org attempts to counteract that. Math teachers send in anonymized samples of student errors that they find telling, common, or inscrutable. Teachers comment as to what they think the student is not understanding and how to fix it. In the process, newer teachers get to see the thought process of their more experienced colleagues. There are patterns to these mistakes, so can a computer be programmed to recognize them?

The most obvious example, which I doubt I’m the first to come up with, is to anticipate patterns of wrong answers. Let’s say you’re testing a physics problem where students need to plug values into an equation. (Yes, this is a naive view of physics but go with it.) Have experienced teachers compile patterns of mistakes students are likely to make: forget to square something, leave off the constant, divide instead of multiply, use a different formula, and so forth. Then the grading software picks new values to plug into the formula, and calculates all the wrong answers for these values (picking numbers so that none of the wrong answer patterns lead to the same numeric answer). Then, if the student gives any of the anticipated wrong answers, the program knows exactly what mistake the student made and can correct them. Hopefully finding a mechanistic error will provide the human teacher with a window into finding and fixing a qualitative misconception.

Let’s take a more complex, real-world example. In some computer science classes at Tufts, the programs written for homework are subjected to a battery of tests, written by both the professor and the students (and their predecessors). In one case, the assignment is to create a programming language interpreter that determines the “type” of pieces of code. For example, it needs to know that true is a boolean, 7 is an integer, and asking if true equals 7 is an error. To clarify, there’s the code the students write (called an interpreter) and then the code that it tries to type, as a test. An interpreter can fail in a number of ways: it can find the wrong type, find a type when it should raise an error, raise and error when it should find a type, stop unexpectedly with an exception, or never stop at all.

I know that’s a bit much to wrap your head around, but (1) that’s the sort of complexity we’re up against and (2) it’s not just an example, it’s a case study. I have a visualization for this data already made, as a class project. My group wanted to take this data and provide actionable reccomendations for the professor, to be able to say, “you’re not handling this properly” or “you don’t understand this very specific detail”. So we hand-built an automated classifier using what we knew about the errors. Here’s part of the visualization we came up with.

The size of each circle represents the number of students who failed at least one test with that error. The vertical position of the circle corresponds to the average number of tests passed by the students who got that error. Colors encode categories, and the horizontal spread means nothing (just a way to prevent overlap). Click on a circle and you’ll get:

Each bar is a student, identified by an anonymized hash. Their errors are grouped together, with the taller bars being the error we have selected. On the real thing (not these images), you can click on any other bar to jump to that error. Hover over the bar and move around to show each of the tests failed with that error below the circles. This highly-specific information allows the user to look at the individual tests and hypothesize the underlying cause of the error.

You can play with the interactive visualization online here.

Education isn’t a no-computers-allowed clubhouse, but software developers are must be smarter about how we approach these tools. Programmers need to work with educators and fill their needs, not just offer up whatever statistics are easy for them to collect. We have powerful tools like machine learning and visualizations, and teachers with decades of experience. We can make useful automated systems, if we stop acting as if it’s a trivial job.

And yet… all of this takes the views education as a series of questions with right and wrong answers. This is largely true in the STEM fields and even in the humanities, but not in the arts. There really is no good way to automate grading of the arts (or to grade the arts, for that matter). We need to nurture our future artists, but more importantly, we need to teach the skills necessary to appreciate the arts, and to dabble in them. As part of the human condition, we all find ourselves with emotions and ideas that we need to express, through music, painting, or writing (writes the blogger). The fact that very few people will appreciate these works is fine; what matters is the catharsis they give to their creator. That’s something no machine could ever understand.

For those that might be interested, one of my projects is trying to organize a book summarizing important research on visual narrative. This book will be a companion volume to my book out later this year, The Visual Language of Comics. If you may be interested in contributing, here’s a Call For Papers for it…

CFP: Interdisciplinary approaches to visual narrative

While there have been a growing number of books on comics in recent years, very few have addressed aspects of structure, particularly from theoretical, cognitive, or experimental points of view and outside the realm of literary or sociocultural theory. I am working to organize a compilation of important papers on the understanding of sequential images. Most of the chapters will be either 1) summary papers that provide extensive bibliographies that can provide an overview to students and a resource to other researchers, or 2) reprints of significant research that remain under-recognized or hard-to-find.

This Call for Papers asks for proposals for papers of two types of chapters focused particularly on research outside of English, presented for an English speaking audience:

1. Chapters that summarize, in English, advances in comic theory from non-English speaking researchers. Such chapters should be large in scope with extensive reference sections.

2. Translations into English of significant non-English comic theory (structural, cognitive, experimental, etc.) from important papers or book chapters.

Topics or chapters outside this scope may be considered, though best to contact me directly with inquiries. (Of interest may be: review papers of other types, historical development of comic “symbology”, empirically grounded discussions of differences between comics cross-culturally, etc.). Importantly, papers should be relevant not only to scholars of comics, but also to linguists, cognitive scientists, and psychologists.

Contributor Guidelines

1. Abstracts of 400-500 words accepted. Papers of 5000-9000 words, including notes and bibliography, accepted. Please also include a short biographical statement.

2. All documents should be submitted as Word or Word-compatible files. PDFs are also acceptable.

3. Submission deadline: May 15, 2013. June 15, 2013

4. Materials should be sent to me via email (Neil Cohn)

I’m often very excited when I find new research on comics, especially when it’s experimental. There is so little done that it’s a treat to find something I didn’t know about. Unfortunately, sometimes my excitement at new data turns sour when I see what was actually done…

I recently found this study (pdf) by Kinzer and colleagues that compares the comprehension and eye movements of readers for narratives in comics and video games. Their main goal is to help provide support for the use of comics and video games in educational contexts.

In this study, they presented sixth graders with either a video game version of a story or a comic created from the images of the video game. Overall, they find that participants understood the story in the video game version better than the comic version. They also found people spent more time engaged with the game than the comic.

I would take all of these findings with a grain of salt…

…because the stimuli appear to be extremely confounded because the comic versions of the story appear to be so badly created. Judging by the example in the text, the comic pages clearly were created by someone who had no real fluency in the visual language of comics. This is clear at a glance just by the example that they provide in the paper:

First off, the images make it extremely hard to tell what’s going on. Second, almost all of the balloons are placed outside of their originating panels to the extent that they completely overlap in panels forward and backward in the text. I don’t even need to know what’s in the text to know this will be confusing to a reader. This is so “illegal” in the rules of page construction that it is almost painful.

If this is their example (what is probably the best example they have), I shudder to think what other pages in the experiment look like. Seriously, if I wanted to design an experiment that had “incomprehensible comics pages” as one type of stimuli, I’d use pages like these.

It’s no wonder they found that their participants had poorer comprehension for the comic version—their stimuli are the equivalent of trying to test English comprehension while using broken English. It tells you next to nothing of interest.

There are two main points I’d like to make about this:

First, good experiments are hard to design, and having something worth saying must follow from having successfully designed an experiment that can give you good information. It pays to be critical as a creator and reader of scientific research (no matter what the topic).

Second, doing experiments using the visual language of comics is not trivial. Stimuli cannot be created by anyone, regardless of their fluency in comic creation. Just because you can throw together some images and words into panels on a page does not mean you’ve successfully created an example of “native” visual language. Believing otherwise does a disservice to yourself and to others who might read and cite your research.

Kinzer, C. K., Turkay, S., Hoffman, D. L., Gunbas, N., Chantes, P., Chaiwinij, A., & Dvorkin, T. (2012). Examining the Effects of Text and Images on Story Comprehension: An Eye-Tracking Study of Reading in a Video Game and Comic Book. In P. J. Dunston, S. K. Fullerton, C. C. Bates, K. Headley, & P. M. Stecker (Eds.), Literacy Research Association Yearbook 61 (pp. 259-275). LRA: Chicago, IL.