Last week we wrote an introductory post about our new Commons Explorer; today we’re diving into some of the technical details. How does it work under the hood?

When we were designing the Commons Explorer, we knew we wanted to look across the Commons collection – we love seeing a mix of photos from different members, not just one account at a time. We wanted to build more views that emphasize the breadth of the collection, and help people find more photos from more members.

We knew we’d need the Flickr API, but it wasn’t immediately obvious how to use it for this task. The API exposes a lot of data, but it can only query the data in certain ways.

For example, we wanted the homepage to show a list of recent uploads from every Flickr Commons member. You can make an API call to get the recent uploads for a single user, but there’s no way to get all the uploads for multiple users in a single API call. We could make an API call for every member, but with over 100 members we’d be making a lot of API calls just to render one component of one page!

It would be impractical to fetch data from the API every time we render a page – but we don’t need to. We know that there isn’t that much activity in Flickr Commons – it isn’t a social media network with thousands of updates a second – so rather than get data from the API every time somebody loads a page, we decided it’s good enough to get it once a day. We trade off a bit of “freshness” for a much faster and more reliable website.

We’ve built a Commons crawler that runs every night, and makes thousands of Flickr API calls (within the API’s limits) to populate a SQLite database with all the data we need to power the Commons Explorer. SQLite is a great fit for this sort of data – it’s easy to run, it gives us lots of flexibility in how we query the data, and it’s wicked fast with the size of our collection.

There are three main tables in the database:

• The members
• The photos uploaded by all the members
• The comments on all those photos

We’re using a couple of different APIs to get this information:

• The flickr.commons.getInstitutions API gives us a list of all the current Commons members. We combine this with the flickr.people.getInfo API to get more detailed information about each member (like their profile page description).
• The flickr.people.getPhotos API gives us a list of all the photos in each member’s photostream. This takes quite a while to run – it returns up to 500 photos per call, but there are over 1.8 million photos in Flickr Commons.
• The flickr.photos.comments.getList API gives us a list of all the comments on a single photo. To save us calling this 1.8 million times, we have some logic to check if there are any (new) comments since the last crawl – we don’t need to call this API if nothing has changed.

We can then write SQL queries to query this data in interesting ways, including searching photos and comments from every member at once.

We have a lightweight Flask web app that queries the SQLite database and renders them as nice HTML pages. This is what you see when you browse the website at https://commons.flickr.org/.

We have a couple of pages where we call the Flickr API to get the most up-to-date data (on individual member pages and the cross-Commons search), but most of the site is coming from the SQLite database. After fine-tuning the database with a couple of indexes, it’s now plenty fast, and gives us a bunch of exciting new ways to explore the Commons.

Having all the data in our own database also allows us to learn new stuff about the Flickr Commons collection that we can’t see on Flickr itself – like the fact that it has 1.8 million photos, or that together Flickr Commons as a whole has had 4.4 billion views.

This crawling code has been an interesting test bed for another project – we’ll be doing something very similar to populate a Data Lifeboat, but we’ll talk more about that in a separate post.

At the Flickr Foundation, one of the goals we set early on when we took over responsibility for running the Flickr Commons program was to build an improved ‘discovery layer’ for the Commons collection.

We’re pleased to share with you a first look at our new Commons Explorer, available at commons.flickr.org.

We’ve built the explorer using the standard Flickr API, and have created a secondary database which is updated pretty regularly. (This is a way of us saying not all the data is live live.) And, please note that photos on display link back to flickr.com.

It’s a work in development, but we wanted to show you our progress in this early version. We’ve prioritized being able to look across the Commons in an interface that’s richer than search results. We’re surfacing activity levels across the collection too, to show that there’s a ton of chatting happening, and new uploads all the time.

The views we’ve built so far:

We hope you enjoy exploring, and, please let us know if you have ideas for how we can improve upon what’s there so far!

In other Flickr Commons news

We are working with the Flickr company to develop a new set of API methods the Foundation will be able to use to build the member management tools we need to really lean into rejuvenating the Commons and especially growing the new membership. If we can introduce 5-10 new members to the program this year, we’ll be stoked! More, we’ll be even stoked-er.

This will involve a new home for registrations of interest, and a smoother onboarding experience for new members as they come on board. Generally, we’re looking forward to new insights into the overall health of the program in the form of better views on activity (the beginnings of which you can see in commons.flickr.org).

If you are either from an existing member institution, or you’re curious about joining in and sharing your historical photography collections, please let us know.

In our early research back in 2021, we noted we wanted to get to know more of the volunteer community too, and see if we can learn about their needs for participating with research and commentary, and I’m pleased to report we’ve begun that, with our first interview with a prominent community member last week. (I was so excited I could barely talk, but Jessamyn wisely recorded the conversation and will be reporting on it soon.)

During a recent episode of Abbott Elementary, the satisfying conclusion of the episode discussed the “specialness” of the fictional school in Philadelphia where the show takes place.

The last scene shows teachers and students assembled to hang a framed photograph showing the first Black teachers who worked at the school. The photo, which they found in the “school archives” was highlighted as a point of pride, something they could all feel good about. The photograph looked familiar to me and I wondered if it was one from Flickr Commons.

Using some image editing skills and reverse image search tools, I found that the photograph shown above was from one of the Flickr Commons members, the Library of Congress. Going to the original source of this image showed that it was from a Black photographer Thomas E. Askew who worked at the turn of the last century.

Another photograph from the same series as the one in Abbott Elementary is in the Commons.

Not much is known about Askew. He was a formerly-enslaved man, born in 1847. He lived in Atlanta, Georgia where these photographs were taken. Business directories show him working at the CW Motes Studio and he had a personal photography studio in his home. From a blog post from the Historic Oakland Foundation (where he is buried):

Askew’s personal, intimate portraits showed a broader range of the Black experience that stood in stark contrast to the stereotypes present in the media of the time. His subjects ranged in age, skin-tone, attire, and vocation and reframed the visual aesthetics and culture developed by Black Americans in the decades following emancipation. This imagery challenged the perception that the American middle class was an exclusively white experience.

He became better-known after his photographs were included in an album titled Types of American Negroes that was compiled by W. E. B. Du Bois for the Exposition Universelle of 1900 in Paris. Du Bois won a gold medal for his role as “collaborator” and “compiler” of materials for the exhibit.

The Library of Congress has determined that Du Bois specifically commissioned Askew to take photographs of Black middle-class people for this exhibition. Askew’s wife Mary was a seamstress and the role of both clothing and accessorizing was an important part of these photos. As the LoC States in their 2003 book A small nation of people : W.E.B. Du Bois and African American portraits of progress,

When we look at the photographs Askew took of these people, we can see a tension in the well-dressed students and residents of Georgia. The style of dress worn by the subjects reveals the status of the sitters, either real or hoped for; we see them today as class-conscious Blacks.

Of the albums Du Bois curated with Thomas J. Calloway, four were photographic, and included this image (photographer unknown) of a baseball team from Morris Brown College which had been founded less than two decades previously by the African Methodist Episcopal Church.

Askew had nine children. Five of them (plus one neighbor) are in this photograph, which was also sent to the Paris Exposition.

From a five-second peek at a photograph in a television show, we can look closer and learn more about the history of the United States, and even of photography itself.

 

State Archives of North Carolina – Charlotte Hawkins Brown

Charlotte Hawkins Brown was an educator and civil rights activist who opened the Palmer Institute for Black students in Sedalia North Carolina in 1902.

The Palmer Institute was the only accredited rural high school (for African American or white students) in Guilford County NC. It graduated generations of Black educators; Brown worked there herself until she retired in 1952.

The State Archives also have a set of sixty archival images of North Carolinian women from the 1800s through the 1950s.

Other notable collections include this set of photographs of Black soldiers from North Carolina who fought in World War I and a collection of Raleigh’s lost African American architectural landmarks (as well as some that are still around).

 

San Diego Air and Space Museum Archives – African Americans in Aviation

From the Tuskeegee Airmen to Mae Jemison, the San Diego Air and Space Museum Archives collects photographs and other ephemera, some of it from personal scrapbooks, documenting Black people working in aviation and aerospace.

Benjamin Davis, specifically had a long military career, retiring in 1998 as a four-star general.

Leroy Criss, another of the Tuskegee Airmen, kept a scrapbook where many of these images are from.

Willa Brown was the first Black woman to earn a pilot’s license in the United States.

 

While we’re on the subject of space, NASA also has created a collection of Black astronauts and other people who worked in aerospace.

 

National Library of Medicine – African American Medical Practitioners

The NLM has curated a collection of Black workers, mostly women, in the Public Health Service for their History of Medicine division.

Mennonite Church USA – Camp Ebenezer Photographs, 1947-1950

Tillie Yoder Nauraine founded an early “fresh air” camp in Ohio for poor Black  children from Chicago. This was part of the Mennonite movement towards “building an interracial church in a segregated society.” Yoder opened the camp out of her conviction that “all people are equal in God’s eyes.”

 

Kheel Center for Labor-Management Documentation Cornell University – Civil Rights

The International Ladies Garment Workers Union actively worked for the rights of Black workers in including picketing Woolworths and making a New York to Washington DC Prayer pilgrimage to mark the anniversary of the Supreme Court decision that segregated schools are unconstitutional.

The Kheel Center also has documentation of the Southern Tenants Farmers Union, an integrated union which held meetings in Parkin Arkansas in 1937.

If you’d like to see more archival photography (or other material) about Black history and culture, the Schomburg Center for Research in Black Culture, Photographs and Prints Division at New York Public Library owns over 300,000 images, thousands of which are online and over a thousand of which are in the public domain.

Or if you’re interested in modern Black photographers read this GQ article where twenty-five Black photographers discuss what drives their work or this Guardian article showcasing the best photography by Black female photographers or this blog post at Flickr.com spotlighting the work of photographer Ayesha Kazim.

 

State Archives of North Carolina – Charlotte Hawkins Brown

Charlotte Hawkins Brown was an educator and civil rights activist who opened the Palmer Institute for Black students in Sedalia North Carolina in 1902.

The Palmer Institute was the only accredited rural high school (for African American or white students) in Guilford County NC. It graduated generations of Black educators; Brown worked there herself until she retired in 1952.

The State Archives also have a set of sixty archival images of North Carolinian women from the 1800s through the 1950s.

Other notable collections include this set of photographs of Black soldiers from North Carolina who fought in World War I and a collection of Raleigh’s lost African American architectural landmarks (as well as some that are still around).

 

San Diego Air and Space Museum Archives – African Americans in Aviation

From the Tuskeegee Airmen to Mae Jemison, the San Diego Air and Space Museum Archives collects photographs and other ephemera, some of it from personal scrapbooks, documenting Black people working in aviation and aerospace.

Benjamin Davis, specifically had a long military career, retiring in 1998 as a four-star general.

Leroy Criss, another of the Tuskegee Airmen, kept a scrapbook where many of these images are from.

Willa Brown was the first Black woman to earn a pilot’s license in the United States.

 

While we’re on the subject of space, NASA also has created a collection of Black astronauts and other people who worked in aerospace.

 

National Library of Medicine – African American Medical Practitioners

The NLM has curated a collection of Black workers, mostly women, in the Public Health Service for their History of Medicine division.

Mennonite Church USA – Camp Ebenezer Photographs, 1947-1950

Tillie Yoder Nauraine founded an early “fresh air” camp in Ohio for poor Black  children from Chicago. This was part of the Mennonite movement towards “building an interracial church in a segregated society.” Yoder opened the camp out of her conviction that “all people are equal in God’s eyes.”

 

Kheel Center for Labor-Management Documentation Cornell University – Civil Rights

The International Ladies Garment Workers Union actively worked for the rights of Black workers in including picketing Woolworths and making a New York to Washington DC Prayer pilgrimage to mark the anniversary of the Supreme Court decision that segregated schools are unconstitutional.

The Kheel Center also has documentation of the Southern Tenants Farmers Union, an integrated union which held meetings in Parkin Arkansas in 1937.

If you’d like to see more archival photography (or other material) about Black history and culture, the Schomburg Center for Research in Black Culture, Photographs and Prints Division at New York Public Library owns over 300,000 images, thousands of which are online and over a thousand of which are in the public domain.

Or if you’re interested in modern Black photographers read this GQ article where twenty-five Black photographers discuss what drives their work or this Guardian article showcasing the best photography by Black female photographers or this blog post at Flickr.com spotlighting the work of photographer Ayesha Kazim.

 

 

When I do planning, I usually carve it up along three axes: Projects, Pipeline, and People. I want to keep our project list very short in 2024. That allows us to focus more deeply, I think, and spend time thinking and waxing and wandering a bit as we map the new terrain of our mission, to keep Flickr images visible for 100 years.

Projects

There are three main flows of project work for the team:

1. Flickr Commons nurturing and growing
2. Start Data Lifeboat
3. Continue 100-year plan ideation and workshopping

Flickr Commons

Flickr Commons turned 16 years old last week. To celebrate, we launched the first instantiation of a new front door which lives at commons.flickr.org. The intent is to help Commons fans explore the different members’ collections more easily, and get a sense of recent activity across the aggregate. We hope to do another handful of releases over this year and beyond.

The other good news is that we’re nearly, finally, ready to welcome new members into the program. The software that supported new registrations and members had decayed a bit over the last decade, so, working with the company team—thanks Ruppel et al—we’ve co-designed a new set of Commons-specific APIs that will help the Foundation really lean into supporting Flickr Commons members from now on.

We are going to build: 1) a new registration form, 2) improved onboarding resources/workflow, 3) the new discovery layer you can now see at commons.flickr.org, and 4) better admin tools for the team to watch over the health of the program, and the happiness of our members. This will all be rolling out in the first half of this year. I don’t have a date for our first new tranche of members, but rest assured, we’ll let you know!

Later in the year, we want to find out a lot more about Flickr members interact with Flickr Commons and see if we can support them to more easily keep track of their input and progress. If you fit into this group, we’d like to know you!

Data Lifeboat

Last year, we applied to the National Endowment for the Humanities (NEH) to develop a first set of prototyping for our Data Lifeboat concept. That’s the idea that we should actually plan for a possible end of flickr.com, developing “lifeboats” that can carry Flickr photos to other places if the big ship goes down. It was gratifying that the NEH decided to support this first block of work.

Our framing for the grant is to create two identical lifeboats containing Flickr pictures, “objective metadata” like EXIF, and a first crack at “social metadata”—the stuff that is only created on Flickr—because we think that’s essential for longer term contextual, archival framing of the existence of a Flickr photo. After all, on Flickr (and off) a photo is a social object, that is discussed, arranged, annotated, pointed at, and displayed, and EXIF data (the data that is created when a digital camera takes a photograph) falls short.

We’re planning to post NEH-grant-specific updates the blog at the end of each month, so stay tuned for that. (I’d better write that next!)

The 100-year plan

I don’t have a structure or plan written yet. But, I’ve really enjoyed all the discussions I’ve had about the idea, and especially the various workshops we’ve run in different groups about the idea. Basically, the workshop is called How to write a 100-year plan and my opening gambit is “I don’t know, what do you think” and conversation ensues.

We do hope to be able to at least get that workshop into a form where you might be able to run it without us. We’d let you know about that too.

Pipeline

We’re just over one year old, launching officially in November 2022. We’ve had an amazing start, thanks to support from SmugMug and our first cornerstone funder, Filecoin Foundation for the Decentralized Web. Since then, we’ve figured out how to accept donations of cash online via Stripe, and even stock donations! We’ve sketched out the grants we’re planning to apply for too.

People

Ewa Spohn, who also helped write the NEH grant for Data Lifeboat, has joined the crew to manage the project. With a background in mechanical engineering, program management, and people-arranging, we’re lucky to have her! Welcome, Ewa!

We’ve brought on a new part-time team member to help wrangle our Pipeline work, Susan Mernit. (Check out her sledgehammer!!) A veteran of the tech industry, Susan changed gears to lead two non-profits in California, to great success. She’s now working with nonprofits to help shore up their development plans and strategy, and we’re very glad she’s come on board to support us.

And, in case you missed it, we’re hiring: Our first job ad for this year is Archivist. It’s live now, closing January 31st.

Since January 2008, the Flickr Commons has been a place for cultural heritage organizations to share their unique historical photography collections with a global community of people interested in visual culture, and see how Flickr volunteer researchers can add knowledge and new descriptions to them.

Flickr Commons has grown to include over 100 members in 24 countries, amassing roughly 1.5 million pictures on flickr.com over the past 16 years. And from today you can now delve into all of the organizations who’ve contributed to this vast historical resource in our new, dedicated Flickr Commons Explorer! Not only can you see which organizations take part in the Commons, but also the scale and scope of their collections. Have a look around and let us know how it can be more useful to you!

We’re celebrating in other ways too:

🎈For one day only, we’re taking over Flickr’s Explore page with photos from the Commons (with thanks to Josie and Crystal in the .com team)

🖼️ We’re featuring the stories of sixteen less well-known gems from the Flickr Commons collection

And no birthday is complete without looking back at the year that’s passed and the year ahead.

U.S. Marine wife Marge Brown blowing out the candles on her birthday cake, from State Archives of North Carolina.

2023: Spotlight on Flickr Commons

We’ve been working hard to breathe new life into the Flickr Commons program after a quiet period.

• We welcomed Jessamyn West to the Flickr Foundation to look after the Flickr Commons as our Community Manager.
• Alex Chan joined us as our Tech Lead. They’re building all the new tools to help Flickr Commons partners see inside their Commons collections and measure their impact.
• We opened a ‘new front door’ for Flickr Commons, a way of exploring all of the Commons members in one place – have a look around!

(This is a screenshot showing you our four newest members, from the Flickr Commons Explorer.)

2024: Growing the Commons community

This year we’re continuing to reconnect with existing Commons members as well as opening the doors to new ones. We’re chuffed to share that we have our first new members coming online very soon: Community Archives of Belleville and Hastings County (Ontario, Canada); State Archives of North Carolina (USA); and Port Morien Digital Archive (Nova Scotia, Canada).

You can meet our three newest members on the blog, and, if you’re a cultural institution with a photography collection, do consider joining in. We’d love to hear from you, and we’ve written about how Flickr Commons works and what it means to join.

Generally, we are working to make it easier for cultural organizations to use flickr.com to easily reach a global audience of millions, especially smaller institutions. We’re planning to build tools that support joining the Flickr Commons. That includes resources and workflows for onboarding, member management, and engaging the community of volunteer researchers who use, comment and tag Commons photos. Expect these to be rolled out throughout this birthday year and beyond!

Data Lifeboat begins

We’re also starting a major project this year, called Data Lifeboat, and we’ve enlisted three staff from our Commons member institutions to help with user research: Dr Mia Ridge (Digital Curator, British Library), Alan Renga (Digital Archivist, San Diego Air & Space Museum), and Mary Grace Kosta (Congregational Archivist, Sisters of St. Joseph in Canada). Trevor Owens, who heads up Digital Preservation at Library of Congress, is also on the advisory board so that’s a nice virtuous circle right there since LC was the first Flickr Commons partner back in 2008.

We’re hiring!

We’re excited to announce that hiring our first-ever Archivist: https://boards.greenhouse.io/flickrorg/jobs/5614267

It’s a fundamental role at the Flickr Foundation, designed to help us archive ourselves for future team members, and, of course, help think through the challenge of keeping Flickr visible for 100 years.

If you or someone you know has experience in digital archives and/or visual archives, has a creative streak and a desire to work for a young technologically-oriented nonprofit, we’d love you to apply.

Celebrate with us!

Follow us on your social media platform of choice for more birthday festivities:

• Twitter
• Mastodon
• Bluesky

One of our goals when we started revitalizing Flickr Commons was to bring in new members. We’re so excited for you to meet them. We’re starting small but mighty.

Bringing in new people gives us a chance to spiff up our existing procedures and documentation, test out our onboarding documents, and make new friends. It also places even more precious memories and cultural heritage into a space that has long term plans, with no known copyright restrictions.

Best of all, all of these folks are existing Flickr users so we can share some of what’s special about them with you before our official “relaunch.”

Without further ado, here is our new cohort. Welcome!

Community Archives of Belleville and Hastings County

Flickr: https://www.flickr.com/photos/cabhc/

Website: https://www.cabhc.ca/

These community archives, located in Belleville Ontario are comprised of “textual records, photographs, maps, newspapers, and other materials that provide information about the people, places and development of Belleville and Hastings County, Ontario.”

This covers the community of about 55,000 people, possibly nicknamed Bellevegas if Wikipedia is to be believed, on the Eastern end of Lake Ontario.

They’ve been posting content to their blog since 2015 including a story of archival survival, a low-tech crowdsourced assessment rolls project, and a tale of reassembling a scrapbook’s pages based on archival material held in three separate archives.

State Archives of North Carolina

Flickr: https://flickr.com/photos/north-carolina-state-archives/
Website: https://archives.ncdcr.gov

The North Carolina State Archives are located in Raleigh, North Carolina. They use their Flickr account to highlight some of the unique and interesting items in their collection. They also interact with the Flickr Community to try to get better information for their unidentified and poorly identified photographs.

Viewing their pictures gives you a great look at both the rural and urban parts of the state. One of my favorite albums is the Carolina Power and Light Photograph Collection, holding images from the photographic library of Carolina Power and Light. The pictures cover 1900 through 1975 give glimpses into random slices of North Carolina.

I have a personal soft spot for the Sidney E. Rochelle Photograph Collection a much smaller collection showing motorcycles in and around Durham in the early 1900s. This is where I found the single photograph of Della Crewe on her Harley Davidson motorcycle, with a sign saying “Around the World on a Harley-Davidson.” Intriguing! That image, thanks to its open license, now illustrates her Wikipedia page.

The Archives have had a blog since 2007, starting from when the State Library and Archives Building was undergoing renovations. They now have several more. There are a lot of fun stories in there though I am always partial to the odd ones.

Port Morien Digital Archive

Flickr: https://www.flickr.com/photos/portmorienarchive/

Port Morien, formerly called Cow Bay, is an historic village located on the rugged east coast of Cape Breton Island in Nova Scotia, Canada. It is now primarily a fishing village, but it is steeped in coal mining history. It is the location of the first commercial coal mine in North America in 1720, as well as the site of the first Boy Scout troop in North America in 1908.

Local history has been well preserved in the community over the years. Historic plaques have been installed, seniors have been interviewed, and there have been a number of audio visual presentations about the community. In addition, numerous books have been written about various aspects of Port Morien history.

Keeping with the rich tradition in preserving heritage, a small group of community volunteers have collected approximately 2900 photos of our community of Port Morien. It started as a project that was developed in conjunction with the community homecoming in 2015 called Morien Memories. Photos and short videos include people and places from the past as well as the present. Our mission is to provide a digital visual record of our community for future generations to enjoy.