Over the last year, we’ve done a lot of work to improve key elements of the Terra infrastructure in order to provide better resource management features for researchers and administrators. As a result of these upgrades, we are now able to start rolling out more granular spend reporting capabilities as well as improvements to the scale and performance of workspace management services. 

Today, we’re announcing a new spend report feature that will provide more transparency into cloud computing costs incurred in Terra per billing project.

To access the spend report for a given billing project, go to the Billing dashboard in Terra, select your billing project in the menu panel on the left and click on the “Spend report” tab in the panel on the right. As shown in the screenshot below, you will see a total spend summary for the selected period, as well as a breakdown of the expenses incurred for compute versus storage. The spend report also includes a chart that breaks down expenses per workspace for the top ten spending workspaces on that billing project.

In order to access the spend report, you’ll first need to configure spend reporting for your Terra Billing Project (requires ownership of the billing project). This documentation article will walk you through how to enable cost reporting through BigQuery for your Google Billing Account and connect it to Terra. Please note if you are using a billing account owned by the Broad Institute, this step is not necessary. 

As you are trying out this new feature, we welcome your feedback and suggestions on how we can make your experience more convenient and productive. If you have any feedback or issues, please reach out to us at support@terra.bio or through the Terra community forum.

Reference documentation

Displaying Cloud costs in Terra
How much did my workflow cost?

The post New in Terra: Spend report per billing project appeared first on Terra.

If you own any workspaces created before that date, please read the full blog to avoid any surprises when the changes are implemented. If all your workspaces were created after that date, feel free to stop reading and get back to your science.

Prior to September 2021, any workspaces attached to a given billing project in Terra shared the same underlying Google Project. This had some side effects that limited our ability to build resource management features, so we introduced a “one workspace per project” model to remove those limitations. Since then, every new workspace created in Terra has been attached to a dedicated Google Project uniquely created for that workspace. 

Yet, a number of workspaces created before the switch to the new model are still linked to Google Projects that support multiple workspaces. In order to move forward with upcoming infrastructure changes that will enable more granular spend reporting as well as improve the scale and performance of workspace management services, we need to migrate those older workspaces to dedicated per-workspace Google Projects.

An automated migration process to minimize impact

We have developed a process to automate this migration, so you will not have to do anything on your end to migrate your workspaces. The new Google Projects will appear in your Google Cloud console as described in this overview article. Data and metadata stored in your workspaces will not be affected in any way; the Google Bucket ID in your workspace and any associated references will not change. 

However, your workspaces will become temporarily unavailable for use while the migration process is taking place. We estimate that most workspace migrations will take less than 5 minutes to complete, although workspaces with a significant amount of data may take longer. (For reference, a migration test on a bucket with 1PB of data and 400,000 files took about 15 minutes in total.)

In addition, any persistent disks currently attached to an older workspace will become unattached during the migration process. You will find them listed on your Cloud Environments page, colored in red, with an action button that you can use to reattach them to a workspace. Note that we always recommend saving your persistent disk data to your workspace’s storage when you are not actively/regularly using those resources.

Rolling migrations to take place starting August 8, 2022

There are over 33,000 workspaces that need to be migrated, so we will be going through them on a rolling basis, starting on August 8, 2022. We will attempt to migrate all workspaces that belong to the same billing project at the same time to minimize disruption. We expect most migrations to be completed by August 26, 2022 (within 3 weeks). 

Given the scale of this undertaking, we cannot give you a specific window of time when your workspaces will be migrated. According to our usage records, most older workspaces are accessed very infrequently, so we hope this migration will happen invisibly for most of you. However, if you are actively using a workspace created before September 27, 2021 and you are concerned that this migration timeline might interrupt time-sensitive work, please reach out to the Terra helpdesk and we can work with you to schedule the migration of your workspaces.

Thank you for your patience as we work to improve our services and provide you with new capabilities to support your research. 

The post Upcoming changes to workspaces created before September 2021 appeared first on Terra.

Sep 2022 Update:

We’ve negotiated a pricing extension with Google to keep the current storage pricing through March 15th 2024 for Broad billing accounts.  Non-Broad billing account managers should reach out to their Google representatives to determine if they have received the pricing extension or are eligible to receive the pricing extension and discuss options.

Given this pricing extension, we have decided to postpone a solution for migrating multi-region buckets until summer 2023.  More details will be announced in a follow-up communication around that time.

Google Cloud recently announced upcoming changes (starting Oct 1, 2022) to the pricing structure of a number of services, including multi-regional storage buckets. Crucially, they will start charging egress fees — which they normally charge when data stored in one region needs to be transferred to another region for computing — on data stored in multi-regional buckets.

Multi-regional buckets provide “geo-redundancy“, meaning that the data they contain is copied in data centers located in multiple regions. This offers a degree of protection against data unavailability or loss in case of large-scale failure (such as a disaster affecting an entire region), which can be an attractive feature for some use cases. 

Until this announced pricing change, multi-regional buckets also offered the opportunity to use the data in multiple compute regions without incurring egress fees. That is why, until now, all Terra workspaces were created with a multi-regional bucket (unless you specified a region at time of creation).

Under the new pricing structure, however, it will no longer be advantageous to use multi-regional buckets for Terra workspaces by default. Using a multi-regional storage bucket would lead to egress fees every time you run any analysis on data stored in the bucket. 

Going forward: single-region buckets for all new Terra workspaces

To address this emerging issue, we are switching the default bucket setting for new workspaces from “US multi-regional” to “us-central1”, which is the same region Terra uses by default for compute. This will protect you from incurring egress fees related to the region of your workspace buckets that would otherwise result from Google Cloud’s pricing change. In fact, you may notice a slight drop in your storage costs, since single-region buckets are a little cheaper than multi-regional buckets.

You will of course still be able to select a different storage region when you create a workspace, and the “US multi-regional” option will remain available, with a notice of caution regarding the pricing policy that will be displayed upon selection of that option. There’s also a documentation article that details the trade-offs involved.

Migrating existing Terra workspaces to single-region storage

There are already over 30,000 Terra workspaces with multi-regional buckets in existence, all of which will be subject to Google Cloud’s new pricing structure starting in October 2022. For those, we are currently developing a migration protocol that will allow us to convert existing workspaces to using single-region storage buckets without breaking any links. 

In order to minimize burden on individual users, we envision doing this on an opt-out basis: we will migrate all workspaces UNLESS the workspace owner requests their workspace(s) be excluded from the migration. We have not yet determined the exact timeframe for this; however, we are committed to providing ample notice, and will send out a separate communication announcing the specifics of the migration plan in due time. 

As always, we are undertaking these changes with the goal of serving the best interests of the research community, and we are open to your suggestions, questions, and concerns. Don’t hesitate to reach out to our support team in the Terra community forum or privately through the Helpdesk as you prefer. 

The post Moving away from multi-regional storage buckets appeared first on Terra.