Analyzing 600k Reddit comments with AI

On a recent personal project, I explored what I thought was a neglected application of LLMs: data analysis. It seems like there are a million examples of people sharing the games, apps, websites, CLIs, etc. they made with coding agents, but personally, I have not seen as many examples of people using LLMs to analyze large datasets and turn them into cool visualizations.

I decided to look at one of my favorite subreddits, r/FoodNYC, which I often use for finding new restaurants in the city. It’s a useful sub, but there are some problems with exploring the data itself:

• Reddit’s search feature is notoriously inaccurate, and searching for types of food, cuisines, or restaurants often gives search results that have little to do with my query.
• It’s hard to get a bird’s-eye view of the best, highest-rated restaurants. The data is divided between hundreds of threads and comment chains.
• It doesn’t have the best interface for exploring comments and replies to them.
• There are a bunch of general questions about NYC, which are irrelevant to me.

I thought I could use AI to address this. There’s a tool called Arctic Shift that lets you download past comments and posts for almost any subreddit. I could download the data for FoodNYC, use LLMs to analyze it and extract the useful data, and visualize the results in a way that makes it easier to explore.

Overall, it was a fun project, and you can view it here. I also wrote a detailed explanation of how I built the data analysis pipeline using LLMs.

What I found

I uncovered some pretty cool insights about the subreddit:

• There are a few “dead zones,” which are areas of NYC with almost no restaurant mentions. Those include the South Bronx, southeast Brooklyn (Crown Heights to East New York), and eastern Queens. The densest zones include most of Manhattan south of 116th Street, and western Brooklyn. This probably says something about the demographics of the subreddit. The East Village alone accounts for over 52,000 mentions, which is more than all of Queens and the Bronx combined!
• Classic restaurants actually score lower than the site median. An NYC classic is a beloved institution with a long history in the city. The lowest-rated ones are The Halal Guys, Di Fara Pizza, and the Russian Tea Room.
• The single most-discussed dish at one restaurant is L’Industrie’s pizza with 606 mentions, followed by Los Tacos No. 1’s tacos, Katz’s pastrami, and the Minetta burger.

What I learned

Some of the key things I learned from this project include:

• It’s important to benchmark LLMs against a human-picked dataset.
• It’s ideal to do multiple test runs so the pipeline can be adjusted and then run end to end with as little human supervision as possible. Human supervision isn’t possible for hundreds of thousands of comments.
• An efficient way of sniffing out edge cases is using LLMs to check each other’s work (and funneling ambiguous cases up to smarter models).

What I liked about this project was how the same process can be done for a million different topics like music, art, video games, fashion, content creation, etc. And with tools like Jev and more intelligent, smaller models, the cost of this kind of experiment is going way down as well.