Data Lifeboat: NEH Grant Update 1

By Ewa Spohn

And we’re off! Thanks to the Digital Humanities Advancement Grant we were awarded by the National Endowment for the Humanities, our work on the Data Lifeboat has started, in our Content Mobility program. We’ll be posting an update for you each month. 

hand-drawn sketch of a decentralization methodology, Feb 2nd, 2024

Excellent Lifeboat-related game and book brought in by Alex for our Kick-off

What’s a Data Lifeboat?

A quick recap for those not familiar with the concept, from our grant narrative:

A Data Lifeboat is an archival piece of Flickr, not all of the 50 billion images and their metadata. For example, a Lifeboat might contain all the photos tagged with “sunflower” or all the Recipes to Share group submissions. Whatever facet of the data you can think of, you could generate a Data Lifeboat for it. We envision an archival sliver richer than a mere folder of JPGs: one where you can navigate the content to explore and understand its networked context. Even better, an archival sliver that is updated if things change at flickr.com. Our goals with this project are to create several rough prototypes of the software, develop a reasonably detailed understanding of the main technical challenges, prepare a survey of critical ongoing legal issues, and establish a robust design direction for further product development.”

This idea was born from two challenges: 1) Flickr contains a multitude of shared histories, and is owned and controlled by a corporation which could decide to close the service, which—as we’ve seen in the past—can result in the destruction of cultural heritage, and 2) the Flickr archive is huge and, in its current form, impossible for any one archival institution to take on.

Flickr’s 50 billion or so photos reflect our diverse heritage, traditions, and history back to us in a unique way. The collection is also born digital, a massive advantage over conventional archives, because the photographers usually describe their pictures themselves as they publish. The pictures are also enriched by the network of social activity that surrounds them, which – again – is unique to the Flickrverse. Finally, this kind of volume is astonishing: Flickr and other platforms like it are orders of magnitude larger than our biggest cultural collections to date.

At the Foundation, we believe we must begin to treat this collection as we would our ‘traditional’ great libraries, archives, and collections. Time is of the essence: the commercial platforms that host these kinds of huge collections can (and do) disappear, effectively sinking our heritage along with them. Our hope is that a Data Lifeboat will carry Flickr images away from the possibility of a sinking ship unscathed. Our future plans include developing the idea of a “dock” in a “safe harbour” – somewhere specific for the Data Lifeboat to land and be preserved.

The scope of the grant

We’re using the NEH grant to create two identical prototype Data Lifeboats containing a selection of the Flickr Commons. This will (hopefully) be a richer archival format that allows for the exploration of content within its networked context, and one that can be updated when changes are introduced in Flickr.com. Importantly, we want to place these two Lifeboats in two different places, a proxy for our developing goal of “safe harbours” for them.

This phase of the project, making a demonstrable prototype, or prototypes, is scheduled to end mid-year.

Our crew

It’s an exciting and completely new thing, and working on it is a multidisciplinary team drawn from both the Flickr Foundation and our Flickr Commons members and advisors:

  • George Oates, Project Director, who provides strategic and design input, and financial oversight
  • Alex Chan, Tech Lead, who is developing the core of our prototypes
  • Jessamyn West, Community Manager, who leads our communication with the Flickr Commons collaborators, the project advisors, as well as broader audiences
  • Ewa Spohn, Project Manager, who ensures the team sticks to the plan. And budget

We’re excited to engage some of our Flickr Commons members directly for the first time, too. The Flickr Foundation team will be joined by staff from three of our member institutions:

And finally, our advisors, who bring a wide range of experience and knowledge and will help us make sure we build stuff for the long term:

Kick-off? Done.

We’re about to have our first all-hands meeting, although in late January we took advantage of Dietrich’s short visit to London to hold our first face-to-face workshop. 

Jenn, Ewa, George, Alex, Stef, and Dietrich (the photographer) at our kick-off

Jenn, Ewa, George, Alex, Stef, and Dietrich (the photographer) at our kick-off at HQ

Over coffee and sugary snacks, we spent two days exploring decentralized storage and how it could be applied to archiving digital content, and thinking through a possible schema for the data in a data lifeboat. 

Emerging questions

Many, many (#many) questions arose (for which we currently have no answers), for example:

  • Is a Data Lifeboat launched in response to an emergency or as part of regular housekeeping?
  • What must a Data Lifeboat contain? Could it initially just be a manifest and the images (which are large and expensive to process) are added later?
  • Who decides what is in (and out) of a Data Lifeboat and to what extent should it feel like an active selection?
  • Where are the ‘edges’ of the network surrounding a Flickr photo, and what is a holistic archive?
  • What existing digital asset management formats could (should) a Data Lifeboat be consistent with for it to be ‘docked’ successfully?

We were also very pleasantly surprised by the power of our lifeboat metaphor and how far we could stretch it to help coordinate our thinking! And thanks again to Dietrich for sharing time with us to crack the project open.

Next steps

Next up is our first all-hands meeting to bring the whole project team up to speed with the plan. That’ll be followed by a deep dive review of digital asset management systems in cultural institutions and a survey of the legal rocks that a Data Lifeboat may encounter. We think that will give us enough to allow us to define some high level requirements for the prototypes so that the development proper can start towards the end of the month.

Somewhere among all that we’re also planning a team expedition to a lifeboat museum to learn more about how lifeboats work in the physical world, but more about that in another blog post…

 

This work is supported by the National Endowment for the Humanities.

NEH logo

Our plan for 2024: Flickr Commons & Data Lifeboat & the 100-year Plan

Find out more about our nefarious schemes for the coming year…

 

When I do planning, I usually carve it up along three axes: Projects, Pipeline, and People. I want to keep our project list very short in 2024. That allows us to focus more deeply, I think, and spend time thinking and waxing and wandering a bit as we map the new terrain of our mission, to keep Flickr images visible for 100 years.

Projects

There are three main flows of project work for the team:

  1. Flickr Commons nurturing and growing
  2. Start Data Lifeboat
  3. Continue 100-year plan ideation and workshopping

Flickr Commons

Flickr Commons turned 16 years old last week. To celebrate, we launched the first instantiation of a new front door which lives at commons.flickr.org. The intent is to help Commons fans explore the different members’ collections more easily, and get a sense of recent activity across the aggregate. We hope to do another handful of releases over this year and beyond.

The other good news is that we’re nearly, finally, ready to welcome new members into the program. The software that supported new registrations and members had decayed a bit over the last decade, so, working with the company team—thanks Ruppel et al—we’ve co-designed a new set of Commons-specific APIs that will help the Foundation really lean into supporting Flickr Commons members from now on.

We are going to build: 1) a new registration form, 2) improved onboarding resources/workflow, 3) the new discovery layer you can now see at commons.flickr.org, and 4) better admin tools for the team to watch over the health of the program, and the happiness of our members. This will all be rolling out in the first half of this year. I don’t have a date for our first new tranche of members, but rest assured, we’ll let you know!

Later in the year, we want to find out a lot more about Flickr members interact with Flickr Commons and see if we can support them to more easily keep track of their input and progress. If you fit into this group, we’d like to know you!

Data Lifeboat

Last year, we applied to the National Endowment for the Humanities (NEH) to develop a first set of prototyping for our Data Lifeboat concept. That’s the idea that we should actually plan for a possible end of flickr.com, developing “lifeboats” that can carry Flickr photos to other places if the big ship goes down. It was gratifying that the NEH decided to support this first block of work.

Our framing for the grant is to create two identical lifeboats containing Flickr pictures, “objective metadata” like EXIF, and a first crack at “social metadata”—the stuff that is only created on Flickr—because we think that’s essential for longer term contextual, archival framing of the existence of a Flickr photo. After all, on Flickr (and off) a photo is a social object, that is discussed, arranged, annotated, pointed at, and displayed, and EXIF data (the data that is created when a digital camera takes a photograph) falls short.

We’re planning to post NEH-grant-specific updates the blog at the end of each month, so stay tuned for that. (I’d better write that next!)

The 100-year plan

I don’t have a structure or plan written yet. But, I’ve really enjoyed all the discussions I’ve had about the idea, and especially the various workshops we’ve run in different groups about the idea. Basically, the workshop is called How to write a 100-year plan and my opening gambit is “I don’t know, what do you think” and conversation ensues.

We do hope to be able to at least get that workshop into a form where you might be able to run it without us. We’d let you know about that too.

Pipeline

We’re just over one year old, launching officially in November 2022. We’ve had an amazing start, thanks to support from SmugMug and our first cornerstone funder, Filecoin Foundation for the Decentralized Web. Since then, we’ve figured out how to accept donations of cash online via Stripe, and even stock donations! We’ve sketched out the grants we’re planning to apply for too.

People

Ewa Spohn, who also helped write the NEH grant for Data Lifeboat, has joined the crew to manage the project. With a background in mechanical engineering, program management, and people-arranging, we’re lucky to have her! Welcome, Ewa!

We’ve brought on a new part-time team member to help wrangle our Pipeline work, Susan Mernit. (Check out her sledgehammer!!) A veteran of the tech industry, Susan changed gears to lead two non-profits in California, to great success. She’s now working with nonprofits to help shore up their development plans and strategy, and we’re very glad she’s come on board to support us.

And, in case you missed it, we’re hiring: Our first job ad for this year is Archivist. It’s live now, closing January 31st.