Following our previous article about why we're building Codatum, let's dive deeper into how we're rethinking the relationship between SQL, data analysis, and the thought process itself.
The Art of "Playing with Data"
Understanding vast amounts of data, such as terabytes or petabytes, requires more than just powerful tools—it requires a process that feels as natural as thinking itself. When facing raw data, you need to process it, slice it along different axes, and view it from various angles. But this isn't a straightforward, linear process.
The reality is that you don't know exactly where to focus or how to process the data "before you begin." Instead, you develop hypotheses while exploring the data, drawing from your domain knowledge, and testing different approaches. This iterative process of trial and error gradually deepens your understanding and interpretation of the data.
The Power of Physical Notebooks
To understand what makes a tool truly effective for data analysis, let's look at something we all know well: the physical notebook.
A notebook doesn't interrupt your thinking; instead, it becomes an extension of your thought process. When you write something down, it naturally leads to the next idea. One diagram sparks another. By continuously outputting your thoughts and adding interpretations, you're essentially expanding your “working memory” in your brain.
While the exact mechanism of how our brain coordinates with our hand to express thoughts on paper remains a mystery, we all recognize that feeling when a tool becomes a natural extension of our thinking—when it feels "comfortable in hand". This comfort comes from having enough freedom of physical notebooks to express our thoughts without friction.
The Current State of Data Analysis Tools
Throughout my career—from academic research to business applications—I've worked with various data processing and BI tools. However, most of these tools, particularly those with GUIs, often interrupt rather than enhance the thought process. They require taking hands off the keyboard and offer limited ways to express complex ideas.
SQL stands out as a powerful tool for data analysis, allowing for complex data processing while keeping your hands on the keyboard. However, it comes with significant challenges:
Environment setup and query execution can be cumbersome
Dividing code into manageable parts (subqueries) and managing results is difficult
Saving, sharing, and managing SQL work across teams isn't straightforward
Unlike regular programming, SQL often requires extensive comments to explain logic and interpret results
There's no built-in way to document thought processes and trial-and-error alongside queries
These issues often lead to fragmented workflows. For instance, many engineers bypass BI tools, writing directly into BigQuery console to extract data temporarily, then pasting the SQL and its results into GitHub, internal wikis, or spreadsheets for storage and sharing.
The Evolution of Development Environments
Modern application development environments, particularly for languages like Node.js or Golang, offer a seamless experience where thoughts can flow freely through your fingertips. Similarly, Markdown has revolutionized documentation, allowing writers to focus on content rather than formatting.
Block editors like Notion have taken this further, combining Markdown's simplicity with structured content capabilities. The '/' command enables smooth transitions between different types of content, creating a fluid writing experience.
Codatum: Creating the Natural Data Analysis Environment
To bridge this gap, we're developing our own Block Editor that incorporates SQL capabilities. While we considered using existing rich text editor libraries or forking cell-based editors like Jupyter Notebook, we believed the core experience needed to be built from the ground up.
We're building our Notebook functionality from scratch based on Prosemirror and Tiptap, creating an environment that:
Allows writing, executing, and dividing SQL queries naturally
Provides immediate visualization and manipulation of results
Enables documentation of thought processes alongside analysis
Supports iterative exploration and refinement
Maintains the flow of thinking throughout the entire process
Looking Forward
While there's much more to share about Codatum's capabilities, our core mission remains consistent: creating a tool that feels as natural for data analysis as a notebook feels for writing. We want to provide an environment where you can focus on understanding your data rather than wrestling with tools.
We're still a small team of less than 10 people, learning and growing as we develop this next-generation data analysis tool. If you're interested in joining us on this journey to revolutionize how people interact with data, we'd love to hear from you.
