Trends in SQL & Data Tools
The landscape of SQL tools is far from static. These forces are shaping the SQL tools of the future:
- Next-gen exploratory tools threaten rigid legacy tools
- More roles are writing SQL than ever before
- Collaboration saves time and reduces cost
- The cloud is a given (but it's not cheap)
- Isolation puts your team at risk
- Remote is the new normal
Next-gen exploratory tools threaten rigid legacy tools
Gartner explores ten titanic shifts in data tools in their 2020 report on data trends.1 Their #2 trend is that rigid, centralized dashboards are on the decline. Data tools can no longer be buffet-style (i.e. have as much data as you like, as long it's on this dashboard).
Instead, Gartner predicts a "shift to in-context data stories [where] the most relevant insights will stream to each user based on their context, role, or use". Put simply, different users have different data needs at different times. Tools can't be one-size-fits-all, powered by "visual, point-and-click authoring".
While Gartner points to AI and ML as the main drivers of this future personalization, people are clearly taking matters into their own hands.
More roles are writing SQL than ever before
The stereotype has long been that only data analysts and data scientists write SQL. While they are indeed power users, that stereotype ignores the diversity of roles writing SQL. To prove it, we looked at the roles of our own active users:
This trend is much broader than just PopSQL. Non-traditional roles are rushing towards SQL because of time pressure, an urgency to explore new business lines, and a hunger for personal growth.
Collaboration saves time and reduces cost
Gartner again reports that "through 2023, data scientists and analysts will lose 60% to 70% of their productive time to activities like finding, preparing, integrating and sharing datasets." 🤯
Your most valuable data experts waste 70% of their time slogging through data prep and hunting for reliable data sources.
Tools that minimize this slog quickly pay for themselves. On the data prep side, collaborative tools like dbt have exploded in popularity. On the exploration side, tools like PopSQL let teammates to share connections, queries, and more.
The cloud is a given (but it's not cheap)
"By 2022, public cloud services will be essential for 90% of data and analytics innovation." (again, Gartner)3. However, "data and analytics leaders need to focus on cost optimization when moving to cloud."
Cloud data warehouses like Snowflake and BigQuery charge for data processed (i.e. queries ran). The simplest way to reduce cloud costs is to reduce queries that are wastefully re-run. Collaborative SQL editors like PopSQL allow users to run a query once and then share those results. Everyone gets access, but without the redundant runs.
Counterpoint: even with this massive shift to the cloud, teams still have real-time production databases (MySQL and PostgreSQL, being the two most common)4. Collaborative SQL editors like PopSQL support all major databases and data warehouses, keeping users in just one tool as opposed to siloed in fragmented tools.
Isolation puts your team at risk
By a landslide, the most popular Towards Data Science post of the last two years was titled: "Why so many data scientists are leaving their jobs".5 52,000 members of the data community clapped their hearts out at this post, an order of magnitude over other top posts. Clearly the sentiment resonated.
The author, a data science leader at Deliveroo, warns that "data scientists quit their jobs because they feel isolated". They are pigeon-holed into ever-smaller portions of the data science workflow (e.g. stuck "executing ML algorithms") and expected to possess a breadth of skills that only a "true data science unicorn" could know.
He acknowledges that "organizing isolated teams to work on collaborative projects in large enterprises is not easy". There is no simple solution to address this problem. Tools that bring teammates together around their data, though, certainly reduce that feeling of isolation.
Remote is the new normal
No need for sources on this final trend. Remote collaboration has become a fact of life. How should you respond, though? The most comprehensive guide to remote work on the internet6 stresses the importance of choosing the right tools:
The data tools of the future trend towards exploratory, as more roles have niche data needs and can't wait. With more individuals pulling their own data, there's ample opportunity for collaboration (and time savings if equipped with collaborative tools). Remain with the status quo? You risk increasing isolation in your team, now easier to do than ever in these remote times.
Which type of SQL tool has evolved to meet all these trends? We make the case that collaborative SQL editors meet the challenge and then some.