Scriptopict

Scriptopict is software that provides a way to annotate images, especially images which are meant to be read as texts, or 'scriptopicts'. It is built on and customised from VIA.

Examples:

How to use:

What is a scriptopict?

The term 'scriptopict' means 'writing picture' and refers to imagery that works as a form of writing, that is intended to be 'read'. It combines the root words 'scripto' for writing and 'pict' for picture in the same way that we derive the words 'pictoglyph' for a picture in stone, or 'pictogram' for a symbolic picture.

Some examples are the writing of Mesoamerican civilisations, European allegorical and genre painting, traditional Australian aboriginal art, the imagery in Christian churches, Japanese Ukiyo-e, or murals depicting mythic and historical events. These differ from images which are simply depictions of their subject matter, as a portrait is a depiction of a person, a still life is a picture of some objects, landscape of some scene and so on. This is of course not a clear or strict distinction. A Renaissance portrait or still life is likely to include many symbolic or allegorical aspects that are intended to be read. Indeed any image, or anything at all can be 'read' for a variety of meanings beyond what it is a direct depiction of. A picture, or any mark at all, has it's denotative and it's scriptopictic or other aspects. An image may be more or less scriptopictic. So for example, the Battle Of Kurukshetra Mural at Angkor Wat is a scriptopict, has many scriptopictic elements, or is highly scriptopictic because the viewer is meant to be able to read the story of the Battle Of Kurukshetra by looking at the mural and seeing many of it's characters and events depicted there.

Scriptopicts often depend on the reader's foreknowledge of the story being told. They are prompts to remind the viewer, or the story teller of the story and its meanings. They may include specific icons helping us identify who is who. Orpheus may be recognised by his lyre, Shiva by the crescent moon, and calendric name tags attach to figures and places in many Mesoamerican scriptopicts. This inverts the way phonetic writing works. Mayan writing was remarkable among Mesoamerican cultures because it was recognised that 'anything that could be said could be written'. The Mayans developed heiroglyphs - pictorial icons in a series standing for specific sounds to form any word. Much other writing was a combination of art and calligraphy, with meaningful elements arranged partly in series and partly non-sequentially, with a large part of the skill of the scribe being the in beauty of their writing. Meaningful elements are sometimes arranged in a series, to be read in order, and are sometimes non-linear or contain both linear and non-linear organisation. Journey narratives for example are common in remaining manuscripts with meaningful elements arranged in accordance with physical location as well as an ordered sequence around the map.

Scriptopicts might be said to allow the 'illiterate' to read, in that people who can't read phonetic text as we know it today (as a medieval peasant would not have been able to read Latin) may look at these pictorial events and understand the meanings behind them - to read the story from the pictures. Again this inverts illiteracy - someone who can read phonetic writing but is foreign to the culture is 'illiterate' in that they are unlikely to understand the full meaning of all the signs and symbols in the picture without having 'learned to read' them. We can learn to read phonetic writing and we can learn to read scriptopicts. The problem for foreignors looking at these forms of writing then is learning how to read, bearing in mind that some are intended to conceal meaning from the uninitiated or others who should not know certain information. While learning to read phonetic writing systems involves learning which symbols correspond to which sounds, learning how to read scriptopicts involves learning the stoires, mythic figures or other cultural meanings associated with symbols.

Often these works can be appreciated on many levels from visual aesthetics, to the narrative and the instructional, philosophical or spiritual implications. In the National Gallery in Canberra I once overheard an Englishman remark, "I know it's cultural and all, but I don't get it." I was at that moment appreciating both their aesthetics and enjoying them because of what I could read at the most rudimentary reading level. It seemed a shame to me that he couldn't be taught at least a little about how to read. In the same gallery was a magnificent Mughal era picture of a Prince's journey and, as beautiful as it was, it was a shame that, while I could see it was meant to tell a story, there was an encounter with a tiger here, a large celebration there, I was also illiterate. Probably this situation is largely due to the development of exhibiting institutions in the West where there has been a history of abstraction and categorisation of cultural practice and a tendency towards purification. Though this has much criticised in postmodernism there remains an institutionalised tendency to focus on visual aesthetics and remove narrative elements from visual arts, with narrative being restricted to writing and theatre, even where this abstraction was not the focus of the culture in which the artefact was produced. Objects and images tend exhibited in either art galleries which focus specifically on aesthetics or museums where only the most rudimentary, if any,of the significance of any imagery within them. The scriptopict software is intended, where appropriate, to help people read and enhance appreciation of scriptopicts, simply by enabling the easy annotation of pictorial elements and making them available on the web.

There is much about scriptopicts that is similar to geographical maps, and much about their meaning that is similar to 'mapping' in the mathematical sense of relating one symbol to another. The functionality desirable in scriptopict software is remarkably similar to mapping software - points or regions need to be layed over a base image, these elements should be clickable to display metadata and more information in a pop up or sidebar, large images should be 'tiled' to improve load time and zooming, etc.

Scriptopicts are like geographical maps. The meaningful elements are layed out in what is often called a nonlinear fashion. Rather than being 'non-linear' (1 dimensional and serial), scriptopicts are multidimensional (typically 2 dimensional in an image, allowing for infinite lines across the surface). Although there may be some intended sequence of reading, as an image the meaningful elements are before our eyes all at once and our eye may fall on them in any order and start the interpretative process in any order. Although there are phonetic texts that purposefully try to apply nonlinearity, such as Cortazar's Hopscotch, and more recent hypertext novels, or text based games, it remains necessary, to make meaning of a text to read it in the intended order - the letters do not make a word if out of order, and the sentence means something different if the words are rearranged. The way we construct meaning in phonetic writing is by linear ordering. Meaningful elements may be ordered in a scriptopict but they are layed out in more than one dimension (in an image 2 dimensions). One this multidimensional plane many lines can be drawn among meaningful elements. Mathematically the amount of connections that could be drawn between meaningful elements increases exponentially with the amount of meaningful elements, though it would be pointless to try to calculate that because what a 'meaningful element' is remains fuzzy - it is not discrete nor easily defined and so not definitively quantifiable. It's worth appreciating though that the complexity of relational or contextual meaning increases exponentially with the more meaningful elements in a scriptopict. In a scriptopict there is a network of contextual relationships along which the reader may navigate a path. Usually, these relationships extend beyond the scriptopict to cultural knowledge, memories and already existing understandings of the story, things or events depicted, since the scriptopict typically functions mnemonically or through conventional symbolism. Reading scriptopicts involves building up layers of meaning through complex relationships, like piecing together a puzzle. The experience of understanding may feel then like solving a mystery or recieving a revelation as one thing provides a clue to another, patterns become clear and we understand what once seemed confused and obscure. This is clear in spiritual text and also in ekphrasis - the Byzantine novel, Leucippe and Clitophon begins with a description of a large wall painting depicting many events. This is a framing device for the adventure that follows, as the images are all depictions of or symbolic references to the adventure. The story is 'read' or 'translated' from the image.

There are, none the less, linear paths through these multidimensional texts. This is well summed up in two of the meanings of the English word 'plot' - to plan a course, such as drawing lines across a map, and the sequence of events in a story. Often scriptopicts are intended to be read in an established order. Journey narratives a clear example. The Mapa de Cuauhtinchan No. 2 is a good example. It is a map of locations represented with images and writing and a clear path indicated by footprints from place to place telling a Chichimec migration story. Even in scriptopicts that are not intended to direct the eye in any particular sequence (except perhaps for the compositional flow), like The Anatomy Lesson, when we look at it, our eye does move from one thing to another, and our understanding of the meaning of one thing and another moves from sign to sign. The interpretative context of relation of one element to another, considering first this, then that, leads us to understand this then that, or this and that and then the other. We move from place to place across the multidimensional surface in a temporal, linear fashion - navigating a path across the image-text-map. The image-text-map becomes a metaphor for life. There is a lot of complex stuff happening in our world and our life is a temporal linear path through it. As often as not, images/texts/scriptopicts not only plot out the terrain of life, but offer a path or a guide to help us navigate it, through the stories and wisdom, the meanings we read.

Planned Improvements

About

Scriptopict and demos are an independant collaboration of William Pascoe and Kaspar Paseko, hosted on the C21CH site.

The Scriptopict software is a modification of VGG Image Annotator (VIA) software, copyright (c) 2016, Abhishek Dutta, Visual Geometry Group, Oxford University. Most of its core functionality is part of VIA software, with modifications to make it suitable for this specific purpose, and to extend and add features to it. This implementation is greatly indebted to their work as it would have taken many hours to create this functionality from scratch. Use and reuse of the software is bound by the VIA terms and conditions.