Making of my data essay for The Pudding

Pitching

In 2023 with the news of my partner’s pregnancy I decided to double down on my efforts to transition into journalism. This meant taking risks financially, abandoning my secure consulting position and using every available hour to apply myself to becoming a visual journalist. About 2 months before my son was born, I felt like my portfolio might be solid enough to start pitching stories to the Pudding.

The first two didn’t make it through, but each rejection came with detailed feedback on how to improve and what was missing - the pitching process alone was a great bootcamp in data storytelling. Armed with additional knowledge I pitched a third story about baby sleep training, you can see the pitch along with Matt’s feedback verbatim here.

This was the original logline for the story: A meta-analysis on medical parenting advice regarding infant sleep training, from the 1800s to today. A data-driven historical lens on parenting trends to give readers perspective.

A few days later I got an email from Matt confirming the team had agreed to move forward with the project. For context, I’ve dreamt of working for or with The Pudding for almost 7 years, so this was a huge breakthrough and definitely an emotional victory - like many of the journalists I admire most, I was going to publish an essay with the Pudding.

Data storytelling

We started by going through a few iterations on possible narrative outlines, the project took a lot of turns, specifically on how to frame our findings so that it would add something new to the existing discourse. Here’s an early version of the narrative structure.

Once we'd reached a solid structure I started collecting data to verify or correct our assumptions, informed by various books and articles from early research. This was before studying at Lede so my scraping skills were sub-par and I relied on Apify to collect the data from Google and Reddit. After collecting the raw CSVs I prompted Claude for python scripts that helped me clean up and reformat the data, after which there was a tremendous amount of manual work and reading to categorize each row according to the user or author’s opinion.

The final data sheet is accessible here.

In need of expert support and a quote or two, I contacted Dr. Jodi Mindell (who’s work I’d been analyzing for weeks) who promptly agreed to meet and confirmed my findings, but also contributed broader context and gave directions to continue looking into. This is one of my favorite moments in any story, when the collaboration with a subject expert begins and you start learning about an entirely new field.

Script

Matt and I then spent many, many, many hours on Google Docs editing copy for the script. This was by far the most difficult part of the process for me, but also the most instructive. I regularly opened the document to see 60 or more editing suggestions and comments, the task was daunting. Matt was diligent in challenging assumptions and pushing me to fact-check any claims, but also in correcting copy to make the script more impactful for readers.

The Pudding team then gave a round of feedback that brought important changes to the narrative structure, with another round later in production once we had a working prototype. The feedback was excellent, the article changed from a lengthy and chaotic analysis to a buttoned down essay. It’s hard to make those decisions when you’ve been working on a story for so long, and I definitely had to cut some darlings.

Design & Code

The design and coding process was much more straightforward, even though I was completely new to Svelte. I had ideas for the visualization early on and we didn’t steer too far away from those initial prototypes.

My mistake was to start prototyping in Figma too early, which was slow and time-consuming given the constant editorial changes. Matt redirected the process to a Google Sheet with two columns, script and visuals.

Once the structure was working I worked on a design overhaul to match the theme of the article (sleep training takes place at night) which was inspired by ProPublica’s art direction.

Matt’s participation in the creative process was wonderful, and the current editorial design is the result of both our work.

I was able to get through the first layer of code, integrating ArchieML and structuring the story with components for each scrolly section with d3-force, d3-geo and a simple linear scale for the timeline. But the standards for the final polish were beyond my level of experience and Matt had to refactor a lot of my code.

Pair programming with him was undoubtedly my favourite part of the project, his passion for debugging is palpable and I constantly discovered different approaches to building the logic we needed.

The project repository is open-sourced here.

We ended with a round of accessibility and performance improvements far beyond anything I'd worked on before, which gave me a new level of respect for the Pudding's ethos: their dedication to good journalism and unwavering passion for making their stories as accessible and impactful as possible.

You can read the story here!