

## We upgraded our Changelog 💪🏼

We have revamped our changelog in partnership with [makelog](https://makelog.com) and
migrated it to a new [location](https://changelog.popsql.com/ledger). A goal of this revamp is to publish
more often and add some new search/filter tools around the posts. For a while now, we have been releasing updates
to the PopSQL client on a continuous rolling basis and then summing up what has shipped over a two week period in
one changelog post. To better align with how we release new features and bug fixes,
we've moved to a rolling post cadence for our changelog. You will now see posts
going out at the same time as update, and as part of this, we are now categorizing
our posts with tags related to type, scope, plan, etc. so that you can filter the posts to just what you care about.




## We launched a new version of our documentation pages
We overhauled our [documentation](https://docs.popsql.com/docs) to better meet your needs! The directory is now easier
to navigate, and you can search through it to quickly find the topic you are looking for.
On each page, you will also see more details, including when the page
was last updated. At the bottom, you can tell us if the page was helpful or not.
You can also suggest changes with the "Suggest Edits" button on the top right corner of every page.
We have also integrated it with our chat, so you can search through our documentation without leaving the chat window.
Hopefully, this should help you on your journey to become a PopSQL power user!

## For BigQuery, you can now see how much a query will cost before running it
This new feature has been enabled for all customers on business and enterprise plans. Next
to the Run button, you will see an estimate for how many bytes of data the query will process.
Then, on top of that, we have added a cost estimate using a standard conversion $5.00 per TB. Check out our
BigQuery [documentation](https://docs.popsql.com/docs/connecting-to-bigquery#bigquery-cost-estimator) for more details.

## Other Improvements
* For dashboard users, when you right click on a dashboard, you will now have the option to duplicate it
* On the query tree, you can now select multiple queries at the same time and bulk delete
* Any team member given edit privileges to a query will now be able to edit the presentation link options
* For the scheduled query approval flow, when a user edits a query and resubmits a previously approved
schedule for review, admins will now see the following note when they are reviewing the query for approval:
"Previously approved schedule that is being resubmitted for approval because the query was updated."
* For big number charts, we have added the option to trim insignificant trailing zeroes for non-raw values
* You can now use the search modal to search for nested columns in BigQuery

## Bugs Squashed
* When there is an error in the SQL statement, if the database provides a line position for the error,
we will show the dot and red squiggle underneath the line. If there is no position, we will now only show
the red dot at the beginning of the statement.
* We have updated the banner for a query locked by a schedule approval to be easier to read in dark mode
* For data catalog, tables with the same name in different schemas will be treated as separate objects
* On the desktop app, Ctrl/Cmd + enter will kick off running a query 100% of the time
* For Databricks, all your schemas will show up in the sidebar
* On big number charts, when you add a currency prefix to a negative number, it won't create a new line.
We have also improved the default settings around precision for big number charts.
* For mac users, the last row of data won't be cut off anymore or hidden by the scroll bar
* For the chart editor, we updated the column meta labels to be readable in dark mode
* For functions and procedures in the schema tab, we now show the expected SQL definition
* We have improved our SQL syntax highlighting throughout the app



## New customization options for big number charts

We added a lot more ways to customize your big number charts. You can now pick the type of format --
number, rounded number, percentage, and raw number. On top of that, you can add currency
either as a prefix or suffix, specify the number of decimal digits for number or percentage format, specify the number of significant digits for rounded format, and pick your preferred combination of separators (e.g. commas and point).
We also improved our trend indicator, adding a new icon and the ability to swap the both trend direction and color.

## Added a locked state to queries when the scheduled query approval flow is turned on

Now, if a user wants to edit a query that already has a running schedule, they will be asked to confirm that they would like to edit
the query. If they say yes, we turn off all the schedules on that query. Once they are done editing the query, they can accept changes,
and their schedules will automatically be resubmitted for admin review.


## Other Fixes & Improvements
* Schedules that are waiting admin approval are now sorted by the date they were requested
* When a user creates a schedule, we added a confirmation window at the end to summarize all the details
* On the Schedules page under preferences, you can now filter scheduled runs by a dashboard name
* When a user clicks on the schedule query or dashboard icon, it will automatically open a new schedule modal if there aren't any schedules
* When a schema fetch is triggered, we've updated the static yellow bar to be a moving icon, so you know that your refresh is still running
* For BigQuery, nested columns have been added to both the schema tree and table details page
* For BigQuery, we've added a `repeated` tag for columns that have the `REPEATED` mode set
* Fixed a bug with the copy results link when results are on the right sidebar
* Fixed a bug with populating the project selector for BigQuery in the schema sidebar for direct connections
* Fixed a bug with the `esc` keyboard shortcut not working as expected to escape modals in the desktop app
* Fixed a bug that was triggering multiple schema fetches at the same time
* Fixed a bug on the chart editor window where the last color selector in the group by section was getting cut off
* Fixed syncing issues in GitLab that caused files to disappear and reappear in both Gitlab and PopSQL



## Added more details around refreshing your schema data

We have now updated the schema sidebar to show more helpful details when it comes to fetching your schema data!
When you hover over the refresh icon, you will be able to see when the schema was last updated and how long it
took to refresh. In addition, we have replaced the yellow "Schema updating" bar that would appear above the tree,
with a rotating schema refresh icon. When you hover over the rotating icon, you will be shown even more details on how
long the schema refresh has been running for and who was the team member who triggered it. Lastly, once the schema
refresh is done, you will see a toast notification on the bottom right of the screen. That way, you can move to another sidebar
and still know once your schema refresh has succesfully finished running!


## Other Fixes & Improvements

* Added support for Clickhouse as a cloud connection!
* Added support to fetch schemas for different databases for cloud connections
* Added support to show BigQuery functions and procedures in the schema tab
* Added the ability to attach a CSV to scheduled slack messages
* Added a toggle to the editor settings to enable/disable <enter> as an autocomplete selector key
* Upgraded WorkOS integration to the latest version
* Delayed making any updates to the dashboard until the user has left the input box for a query variable
* Fixed bug where legend with many series shown above / below chart would not display properly for bar charts
* Improved the way we store data in the app to improve app performance
* Reduced the width of the ascii table in our Slack integration to better support when a user embeds a notification
* Fixed deep links, so you end up at the right screen after authentication
* Fixed open tabs to be dependent on the active organization, so when you switch between organizations, you will only see opened tabs from the current org
* Fixed sorting for line charts when using a group by
* Fixed chart issues around handling x-axis tick label overflow
* Fixed issue with autocomplete results where "undefined" was prefixing a table or view when not using the data catalog



## Show sample row results directly in Slack over screenshot

When scheduled queries are configured to post in Slack, we would previously either show a "No results" message or a screenshot of the query results and any chart that had been additionally configured for it. While the screenshot makes sense for queries with charts as there's no other way to display the chart in slack, for queries that only returned data, this added a clickthrough to get to the data, which was especially annoying to users on mobile devices. Now, for these data only queries, we will show a sample of the results directly in slack, while still including the links to presentation. To ensure the table displays equally well without line overflows for both desktop and mobile users, the table is limited to only show 5 columns that are also limited to in max total width that can be shown. Per a Slack limitation of only allowing 3000 characters per message, we also limit the table to 10 columns.

## Other Fixes & Improvements

* Add backend flag to filter projects fetched for bigquery cloud connections
* Limit updating user credentials to 2 per minute to prevent spam emails on rapidly changing emails
* Fix crash on adding query with variables to dashboard that did not previously have variables
* Fix bug where large BigQuery schemas (512mb+) would not load in Chrome or desktop app
* Fix crash on trying to open table, view, or column from search results
* Fix Athena only showing first 100 schemas in database
* Fix bug where user would need to re-authenticate with BigQuery OAuth after logging in via Google SSO
* Make the billing system more lenient dealing with cases where an invoice is overdue
* Show descriptions rendered in markdown in the right sidebar trees
* Fix bug where existing number and combo charts would not fallback appropriately when applied to a query with different columns
* Do not show a past due invoice to users when their trial ends
* Improve text contrast on pie charts when using dark colors for the slices
* Change the query description input field to be a textarea to make it easier to have multiple lines
* Fix formatting of x-axis in tooltip for combo charts
* Fix schema refresh button getting cut-off when viewing a connection with a long name
* Add preview button to the titlebar when viewing tables or views
* Fix case where query result column would be detected as "nulls" when majority of values were null, but not all. In these cases, we
  fallback to the second most  prominent detected type. For example, the column with `null, null, 1` will be detected as numeric now
  instead of null, and will have the numeric aggregation types when charting instead of only `count` aggregation.



## Updated the Header for our Schema Sidebar

We've refreshed the design for the header on our schema sidebar. Now, all of the fields show
an arrow to highlight that all of the fields (connection, database, and schemas) are all dropdowns.
Additionally, we've modified the design of the schema selector, losing the outline and adding a rad
new icon so that it looks and feels similar to the other selectors.

## Status Page

We now have a [Status Page](https://popsqlstatus.com/) to give better visibility into when
something is causing PopSQL to not work for customers. While we strive to have 100% uptime,
sometimes things happen that brings some component of PopSQL offline, be it some internal issue or
related to an upstream provider, like [AWS going down](https://aws.amazon.com/message/12721/) back
in December. In addition to the static page, you can easily subscribe to updates via email, SMS,
Slack, MS Teams, Webhook, or RSS so that you can always be in the know if something is bringing down
PopSQL services. We will be adding a status icon to our messaging service to make sure all customers
have full visibility into it and to help avoid customer confusion and writing in when things go down.

## Other Fixes & Improvements

* Fix crash on filtering users when adding to a team
* Improve performance on showing tables with cells containing lots of text
* Fix bigquery schema fetch failure if 1+ projects failed for some reason (e.g. BigQuery API was not
  enabled on project), and other projects succeeded in fetching
* Update number of icons to have more consistent internal design
* Fix rare cases where old connection form could still be opened
* Fix bug where query variables only used inside Liquid control flow tags' bodies and secondary
  branches were not being detected (for example, if you have `{% if foo %} {{ bar }} {% endif %}`,
  then bar will now be properly detected)
* Improve cloud query performance, most notably 1-2s for queries against Snowflake
* Fix 500 error on viewing billing page if Stripe collection method is set to send invoice
* Fix not using showing saved avatar picture under user details popup
* Line charts now default to using straight line instead of natural curves
* Allow using SSL for cloud connections without uploading any certificates or keys
* Fix "Download full results" for cloud queries returning >100,000 results not working in Chrome or
  in the desktop app
* Improve cancelling cloud query logic for Redshift, where we now check both `STV_RECENTS` and
  `STV_INFLIGHT` for the running queries
* Fix usernames in some dropdowns being hard to read under dark mode
* Fix line charts only showing y-axis info when x-axis was using timestamps (when x-axis was
  anything other than timestamps, then x-axis info would show in the tooltip as expected)



## Snowflake Role and Warehouse Choosers

When running queries against Snowflake, it can be useful to change the warehouse you
are executing against, or the role you are using. Previously, you could set these values at the
connection level, where this meant you would have to make a new connection for each warehouse or
role you wanted. We have released a new update where these options can now be changed on the fly via
dropdowns under the "Run" button in the query editor, similar to the existing database option. On changing
your role, we will automatically refresh the available warehouse list to help prevent trying to use a
warehouse that a given role does not have access to. The fetched schema for the connection will continue
to use the originally set warehouse and role, but we are looking into making this also adjust to using
the adhoc selected role as well in the future. This is immediately available to all customers for both
direct and cloud connections, and the chosen role, warehouse, and database are all set on a per query
basis.

## Other Fixes & Improvements

* Fix running queries ending with semi-colons in Presto
* Performance improvements for admins of organizations with several thousand queries
* Fix "empty url" error on trying to connect with Azure cloud connections
* Fix z-index style bug for some dropdowns of popovers
* Creating a new dashboard in shared folder will have it be properly shared from the onset
* Add carets to dropdowns beneath the Run button to better indicate they are changeable
* Open number chart color filter dropdown above instead of below field
* Show full number unless precision is non-zero for big number charts. If precision is non-zero, then the number will be rounded and show K, M, etc. for large numbers
* Allow using CMD/CTRL+# to move between tabs for 0-8, and CMD/CTRL+9 will always open the last tab
* Fix bug where SHIFT+CMD/CTRL+Z would open dbt panel instead of trigger redo in editor. Keyboard shortcut for opening dbt panel is now SHIFT+CMD/CTRL+U
* Fix case where we were not properly analyzing queries run against direct connections for use in data catalog and popularity statistics



## New Schedule Drawer and Modal

We have shipped a refreshed design for our schedule modal for queries and dashboards. Now, when initially clicking the schedule button, instead of a screen covering modal, we show the list of schedules in a drawer that opens on the right side of the screen, similar to details, comments, and other UI elements. From that, we've then simplified the schedule modal to better utilize the same colors and design as the rest of the application. Finally, you can now give your schedules names to make it easier to differentiate between them.

## Schedule Approval Workflow

We are currently rolling out to enterprise customers the ability to turn on a new schedule approval workflow. For organizations who have opted into this workflow, any new schedule that has been created by a user will need to be approved by an admin. Additionally, admins will be able to get a full holistic view of all schedules that have been created for their organization, as well as see the prior run results for those schedules. These views are available under "Preferences" under a new "Schedules" tab on the left sidebar.

## Toggle X-Axis Sorting in Charts

We have rolled out a new feature on charts to allow end users to control how the X-Axis value will be sorted. Previously, we would always apply an ascending sort to the x-axis values in our charts. While we will still continue applying that sort by default, you can easily change it to descending or turning it off altogether by clicking the new icon to the right of the "X-Axis" header under chart settings.

## Other Fixes & Improvements

* Remove the unneeded database input field for Preto connections
* Moved our search modal to run in a web worker to improve performance
* Reduced the data initially loaded when starting the app to improve performance
* Added ability to optionally set initial role for Snowflake connections. If not set, will use whatever default is set within snowflake
* Marked warehouse input for snowflake as optional. If omitted, will use whatever default is set within snowflake
* For snowflake connection, marked database, warehouse, and role fields as default to better indicate they're only used for initially connecting, but can be changed after connecting
* Fixed bug regarding changing time for scheduled queries
* Added buttons to navigate from the Schedules overview page to your related queries and dashboards
* Fixed bug allowing scheduled variables to override query variables
* Fixed bug on fetching "BigInt" fields for direct connections
* Fixed bug around printing interval-based schedules in scheduled emails
* Added a [download page](https://popsql.com/download)
* Fixed a bug around circular folder references in Firebase
* Allow dbt_project.yml to be located in a subfolder for the dbt integration (multiple dbt_project.yml files is not yet supported)
* Upgraded our dbt instance to 0.20.0
* Fixed bug with user names not showing up with their avatars
* Hide all axis settings in the chart editor when the axis has been disabled




## Adding Limit Text to Run Button

A common complaint that we have received from customers was not knowing why their query was being limited to 100 results. While we have a product tour in place that tells you about the feature the first time you run a query, it is easy to click through it, and forget about it months later. To help here, we have updated our "Run" button so that it will say "(Limit 100)" when the setting is active. Enabling and disabling the feature remains exactly the same as before in that you just need to hit the down arrow to the right of the button, and deselect the "Limit 100" checkbox.

## Add Warning for using localhost with Cloud Connections

A common thing we get from first time users, especially within the webapp, is asking why their connection does not work, and it is usually because they are trying to connect to their localhost database with a cloud connection. Now, whenever a user inputs "localhost" or "127.0.0.1", they will see a red error text immediately below the hostname field letting them know that localhost is not supported on cloud connections. We are still evaluating this behavior, and may change location of the connection type toggle as well as other UI tweaks to help cut down on the support tickets we receive about this.

## Improved Editing for Cloud Connection Username and Password Fields

Previously, we would only fetch the username and password details to be shown in the edit connection UI for cloud connections on the desktop app. This led to a less than desirous user experience on the web app where on opening the edit connection form, the username and password would be blank, and users would edit some other detail and save the connection without realizing that that would overwrite their previously set values with empty values. Users would then end up confused on why their previously working connection would no longer work. To improve things, we now show these values in a disabled input initially, and only on hitting the "Edit" button will we save those details, giving a more consistent editing experience on both desktop and web. We still however will only ever fetch username and password to be pre-filled into those inputs when editing on the desktop application.

## Other Fixes & Improvements

* Fixed issue where generating PDFs for scheduled queries with empty results would not work
* Limiting lengths of inputs on sign up page to prevent potential attack vector for a DDOS attack
* Added support for filtering scheduled runs by name on Schedules page
* Fixed bug for x-axis not rotating for line chart with time-series data
* Fixed bug where clicking edges of export button did not trigger an action
* Add support for using gravatar for a user's avatar
* Fixed bug where you were not able to change a connection for a scheduled query
* Added support for querying binary data in PopSQL
* Fixed bug with charts where numerical values were treated as strings for sorting



## Schedule Approval Flow

We have begun internally testing a new schedule query and dashboard approval flow. With this, any schedule that a regular user creates will be set to "pending" and await admin approval under a new page in the "Preferences" view within the app. We plan to roll this out in the next month, and will be available to enterprise customers.

## Query Groups for Redshift

For Redshift connections, we now allow setting a [query group](https://docs.aws.amazon.com/redshift/latest/dg/r_query_group.html) for "User Runs" and "Scheduled Runs" for any query using that connection. The former applies to when users manually hit "Run" on a given query, and the former for [schedules](/docs/getting-more-out-of-popsql/scheduled-queries). These new settings apply to both direct and cloud connections.

## Other Fixes & Improvements

* Fix files stored in S3 (e.g. schemas, run results) not having appropriate content-type
* Fix clicking table/view name opening "Table Details" tab instead of expanding the object in the schema tree
* Update Clickhouse adapter to support newer versions of Clickhouse
* Fix missing data catalog entries in some rare cases
* Fix bug on loading BigQuery schemas where we find datasets we cannot access. These datasets are now skipped.
* Update dropdown for dbt connection to only show cloud connections
* Fix bug where notebook connection chooser would revert value on focusing


Changelog

Ready for a modern SQL editor?