Pioneers like Michelangelo and Leonardo da Vinci might find it odd that the arts and the sciences have become separated from one another. Aesthetic composition, mathematical equations, drawing and engineering were natural bedfellows during the Renaissance. Yet separate them we did.

A pioneering Cornell Tech faculty member has devoted the final third of his career to working across these disciplines. Professor C. Richard Johnson is the Jacobs Fellow in Computational Arts and Humanities in the Jacobs Technion-Cornell Institute at Cornell Tech. His crossover work between electrical engineering and art history has impacted museums across the world.

Johnson’s love of fine art began with his first visit to an art museum when he was an exchange student in Germany in the 1970s, where he encountered paintings by masters such as Rembrandt. Just two years later, as a grad student working on a PhD in Electrical Engineering at Stanford, he chose to minor in Art History. One of his professors, Madeline Kahr, was an expert in Dutch art and encouraged him to research Vermeer’s use of the camera obscura, a predecessor of the modern camera that uses pinhole projection.

This interdisciplinary scholarship was exciting to Johnson. However, it would be 30 years before the time was right to apply his technical skills to solving real-life problems in art museums.

The Right Timing and The Right Partners
At Cornell, Johnson found a supportive environment that allowed him to help establish a new field: computational art history. As technical processes like X-raying and digital scanning became more common within museums, Johnson asked himself how he could apply his electrical engineering skills to answer art history questions from digital processing of such images.

Johnson needed to find art history collaborators who were open to sharing ideas, and he needed access to museum data — which was not always easy. To engage external experts, he had to address questions that were meaningful to them.

“I’m looking for something they already want to do, rather than trying to get them excited about something they’d never thought about before,” he says.

At the Van Gogh Museum in Amsterdam he found the perfect team at the perfect time. Johnson proposed a workshop looking at how image processing could answer questions of dating and attribution in Van Gogh’s work. While organizing this workshop, Johnson was introduced to the use of thread counting as a valuable canvas forensic technique. In 2007, the Thread Count Automation Project ( TCAP) was established.

Computational Art History in Action
Traditionally, experts who studied the canvases hand counted threads from x-radiographs of paintings on canvas to determine density from a few samples at different spots across the canvas. Johnson and his technical collaborators devised computational techniques that characterized the local thread density at spots covering the entire canvas.

´Weave maps´ could then be produced for each canvas. Due to the weaving process, these spot density values recorded with color revealed stripes. These striped patterns could be compared in order to identify matches. When a match occurred, it meant that the canvases had been cut from the same roll of cloth. This invaluable information could then be combined with other knowledge, and used to date and authenticate paintings.

The result was an enormously successful project which was reported in art history journals such as The Burlington Magazine, and in the popular media.

Since then, Johnson has worked on interdisciplinary projects with many other institutions, including the Museum of Modern Art (MoMA), the Morgan Library and Museum, the Rijksmuseum and the Metropolitan Museum of Art.

Establishing Excellence in Interdisciplinary Collaboration
Jim Coddington, former Agnes Gund Chief Conservator at MoMA, has worked closely with Johnson. The team used computational thread counting on Van Gogh paintings in MoMA’s collections, and in 2010 they established the Historic Photographic Paper Classification (HPPC) Project.

In the HPPC project, Johnson and his technical collaborators used algorithms to characterize photographic papers based on their texture. The team was then able to take unknown photographic papers and establish when and by what company they were likely to have been made, by comparing them to known papers. Again, these answers helped experts hone in on more accurate dates and attribution.

For Coddington, the key to successful interdisciplinary collaboration lies in clearly establishing which questions are being asked and in understanding why they are important. In order to do that, good dialogue is essential. Art historians, conservation experts and electrical engineers need to understand one another’s perspective and language.

“Explain it to me, as best as you can, in the terms that I’ll understand, and I’ll try to explain it to you in terms that you can best understand,” he said.

The surprises that technology can reveal in art history continue to excite Coddington. “That artist did that? What a crazy idea! Or what a brilliant idea. Or if we take an x-ray of something and we see a completely different composition underneath.”

Cornell Tech Students Working Across Disciplines
In 2015, Johnson started work on the Watermark Identification in Rembrandt’s Etchings (WIRE) project with Erik Hinterding, curator of prints at the Rijksmuseum in Amsterdam, and Andy Weislogel of the Herbert F. Johnson Museum of Art at Cornell.

To apply computing techniques to watermark identification, Johnson needed to understand how Hinterding was thinking. To date, Hinterding made identifications by eye using a book detailing over 500 known watermarks.

Johnson observed how Hinterding looked at features on the watermarks, such as the characteristics of motifs like crowns or eagles. Hinterding would narrow down the identification by asking increasingly detailed questions: Is the eagle double-headed? Is the crown positioned above or below the initials?

It was, to Johnson, similar to a computer-based decision tree model. Johnson is now working with Cornell Tech students to develop a decision tree. The students come from different departments, they work together, and discuss and share tasks.

“We’ve been going through this decision tree and trying to create it ourselves, and then asking Erik [Hinterding], well what do you think?”

Recently, the students presented their work to Hinterding, and experts from the Metropolitan Museum of Art and The Morgan Library. In the development process, they had identified a watermark that was not catalogued in Hinterding’s book. Johnson delights in these breakthrough moments.

“It’s fun teaching, let me tell you, it’s fun teaching,” he says.

Full Circle
Johnson is now finishing a collaborative book on Vermeer’s canvases, and has produced weave maps for all 34 of the artist’s paintings. It is a vast and important project which has identified six canvas pairs. This, in turn, reveals new information about Vermeer’s studio practice. And poignantly, it connects Johnson back to Madeline Kahr, the professor who first encouraged him to look at Vermeer when he was a grad student in 1970s.

Johnson attributes his achievements since then to “luck, timing, perseverance and good advice.” He has helped establish a new interdisciplinary field and worked with some of the world’s greatest museums.

“That type of engagement outside academia is really what Cornell Tech is all about,” he says.