How Messari’s CTO Revealed the Blueprint for Scaling Crypto Intelligence

In an era where blockchain data is growing at an explosive rate, building tools that can help developers derive actionable insights is more critical than ever. At a recent Solana developer gathering, Messari’s CTO, Diran Li, shared a behind-the-scenes look at how his team has been pioneering tools to scale crypto intelligence. His session titled “Scale or Die” wasn’t just a catchy headline, it was a candid acknowledgement of the rapidly evolving demands of the Web3 ecosystem.

Li’s presentation focused on how Messari has tackled challenges in ingesting, cleaning, and serving blockchain data at scale. He broke down the tools, strategies, and hard lessons learned while building infrastructure capable of powering next-generation data-driven applications on Solana. From fragmented data to noisy signals and infrastructure bottlenecks, the session was filled with insights for any developer navigating the modern crypto landscape.

The Challenge of Scaling Insight in a Noisy Ecosystem

Diran opened his talk by laying out the problem: the crypto ecosystem is vast, decentralized, and chaotic. Developers today are inundated with raw data spread across multiple chains, protocols, and formats. Whether it’s on-chain transactions, social media sentiment, or funding news, parsing meaningful signals from this noise is a major hurdle.

According to Li, one of the key challenges in scaling crypto intelligence is the fragmentation of data. In a typical workday, even the most experienced analysts have a dozen tabs open just to track developments. Making sense of what’s happening on Solana, or any other chain, requires synthesizing a huge variety of data sources, each with its own quirks and inconsistencies.

Lessons Learned from Building at Scale

One of the most valuable parts of Li’s talk was his breakdown of how Messari’s architecture evolved. Initially, back in 2018, the team used basic ETL (Extract, Transform, Load) jobs running on Go, funneling data into a PostgreSQL database. As demand grew, the team added more services, databases, and pipelines, which quickly made the system more complex and harder to manage. The real turning point came around 2022 with the rise of large language models (LLMs) and AI-based data analysis. 

Messari began exploring AI-powered data curation, which, ironically, added to their data fragmentation issues. The insight that helped them simplify everything was a switch from ETL to ELT, Extract, Load, then Transform. This shift allowed them to store all raw data first and then apply transformations with full visibility into every change made. This decision was key in building a lineage-aware system that could trace every data output back to its source, making data engineering more transparent and resilient.

Observability and the Power of Data Lineage

The second key takeaway was the importance of data observability. Messari developed clear dashboards to monitor all backend jobs, how they run, what they process, and how frequently they operate. This ability to inspect and debug pipelines in real-time has been a game-changer, especially when trying to maintain reliable, trustworthy datasets at scale. By embracing modern observability frameworks and open-source tools, Messari ensured they could scale without compromising on accuracy or trust. Developers in the audience were encouraged to implement similar practices to manage data transformation pipelines at volume.

Crypto Intelligence Needs AI, But Good AI Needs Clean Data

Li emphasized a crucial insight that resonated with many in the audience: doing AI well means doing data engineering well. AI models, especially those used for embeddings or natural language insights, require high-quality input. If your data isn’t clean or traceable, your AI outputs won’t be reliable. Messari’s LLM pipelines are not only used for summarization or sentiment, they also support source-grounded responses with proper citations. This approach ensures that any insights generated by their systems can be verified and traced back to the original datasets.

Developer Tools Built for the Solana Ecosystem

Toward the end of his session, Li introduced two free tools now available for developers working on Solana.

1. SIG (Signal) Dataset: This is the curated output of Messari’s data pipelines, providing AI-enhanced insights into trending tokens, community sentiment, and major events. It’s designed to help developers understand what the ecosystem is talking about, and why it matters.
   
2. AI Toolkit: This assistant pulls from over 170 terabytes of curated crypto data, offering real-time answers complete with citations, tables, and charts. It integrates seamlessly into products like the Coinbase AI Agent Kit or Eliza OS, and has been widely adopted by Solana devs aiming to integrate smarter, data-informed features.
   

The Future of Crypto Intelligence Is Already Here

Diran Li’s talk was more than just a technical overview, it was a glimpse into the future of how blockchain data will be accessed, understood, and used at scale. With the explosion of on-chain activity and developer innovation, tools like those being built at Messari will become essential infrastructure for the Web3 era. For developers, the message was clear: don’t just build fast, build smart. Use structured, scalable pipelines. Prioritize crypto intelligence that is source-grounded and actionable. And most importantly, start treating your data pipelines with the same care you treat your codebase. The tools are available, the playbook is open, and the next wave of intelligent crypto applications is already in motion.