Having transparent, controllable cloud costs is a top priority for Terra. In response to feedback from our community, we have been working on three key initiatives to improve cost management on Terra:<\/p>\n\n\n\n
When you are working with data in Google Cloud, you are paying to store and analyze your data. Running a workflow or a Jupyter notebook all incur charges through your Google Cloud billing account. More details can be found on Terra\u2019s support site around Managing Cloud Costs<\/a>.<\/p>\n\n\n\n We encourage all billing project owners to set up spend reporting<\/a> for any new billing account. By completing the setup, you ensure that your billing data is captured and accessible in Terra, giving you greater control over your cloud expenses. Once set up, you can access reports<\/a> for financial oversight and optimization of your cloud costs and resources. <\/p>\n\n\n\n Workspace owners now have a consolidated spend report to see costs for all of their workspaces across billing projects to identify what workspaces are costing them the most money. Billing project owners can also dive into the details of the workspaces in a billing project and see costs over time, aggregated daily to observe how costs are trending. For more details, see our roadmap article on reported costs<\/a>.<\/p>\n\n\n\n To reduce a workflow\u2019s cost, it\u2019s essential to know which tasks require the most resources, and which resources cost the most. You can now gather this information by selecting the \u201cresource monitoring\u201d option when configuring a workflow submission. Read Monitoring GCP cloud resources used in a workflow<\/a> for more information and step-by-step instructions.<\/p>\n\n\n\n Once you\u2019ve run a workflow with resource monitoring enabled, you can break down the resources and costs consumed by the workflow. Several tutorial notebooks<\/a> demonstrate how to generate plots to visualize the time spent on each workflow task: CPU, memory, and disk usage, and the cost of each resource for each task. This information will help you optimize your workflow for cost and estimate the costs of similar workflows in the future.<\/p>\n\n\n\n To generate these plots for your own data, clone the Workflow Resource and Cost-Monitoring Guide workspace<\/a> and follow the instructions on the dashboard.<\/p>\n\n\n\n This feature gives you the ability to set a cost threshold for a workflow to stop runaway costs from occurring. We recommend you use this feature more as a safety net to prevent runaway costs than a strict budgeting tool. Although our findings show high accuracy and reliability in our cost estimations, workflows may not terminate immediately upon hitting the defined threshold. <\/p>\n\n\n\n This feature is currently in preview. For more details, see our roadmap article<\/a>.<\/p>\n\n\n\n For new workflows submitted after March 3, 2025, Terra shows estimated costs of workflows using the same estimates used to calculate workflow cost thresholds. These estimated workflow costs are displayed on the submission details page. If you have spend reporting<\/a> set up, the estimates will convert to the actual costs from Google within 24 hours. <\/p>\n\n\n\n To estimate the cost of one of Terra\u2019s featured workflows, refer to costs of selected features workflows<\/a>. Note that your cost may differ from the estimates listed in this article due to differences in your data\u2019s size and runtime settings like disk size.<\/p>\n\n\n\n Example workspaces for these workflows offer tools and workflows with benchmarked cost and time estimates in the workspace dashboards. For example: CellBender<\/a>, one of our most commonly used single cell workflows, GWAS (using REGENIE)<\/a>, and GATK-SV single sample<\/a> analysis workflow are among the many featured and public workspaces which contain cost estimates.<\/p>\n\n\n\n By default, all Terra workspaces and Terra Data Repository<\/a> datasets now leverage Autoclass<\/a> storage to automatically move infrequently-accessed data to a less expensive, \u201ccolder\u201d storage tier (see Google’s storage pricing tables<\/a> for details). Terra defaults to move files all the way to the Archive storage class for maximum savings. We estimate this will save users an average of 30% to 50% on annual storage costs.<\/p>\n\n\n\n Workflows can generate many intermediate files that users do not wish to keep long-term. Workflows now include an option to delete intermediate files<\/a>. This reduces the storage costs incurred by workflows configured to run with this setting. <\/p>\n\n\n\n For situations where you need to clean up intermediate files from workflows that were not run with this setting or after a period of time, Terra has introduced a new way to clean up intermediate files that you no longer need using Google lifecycle rules<\/a>. Lifecycle rules will both save money on storage costs and let you better manage files that are produced as a part of your analysis. For more details, see our roadmap article<\/a>. <\/p>\n\n\n\n While users with read-only access to your workspaces cannot run compute, they can incur costs to your account by downloading or copying the data. To avoid paying when other users download or copy data from your workspace, you can enable requester pays on your bucket. Read using Requester Pays workspaces<\/a> for more information and step-by-step instructions. This option is particularly helpful when hosting large datasets on Terra.<\/p>\n\n\n\n Working in the cloud opens up many opportunities to analyze data at scale. We appreciate the feedback we have received from our community about the importance of transparent costs and the ability to control them, and are dedicated to making improvements to better meet your needs. With these new tools in hand, it will be easier to track and control costs.<\/p>\n\n\n\n We\u2019d like to continue to hear your feedback on these features and suggestions for the future! Check out our public Terra roadmap<\/a> to see features that are planned for the near term or submit a new feature request<\/a> on our public community forums to suggest improvements.<\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" Having transparent, controllable cloud costs is a top priority for Terra. In response to feedback from our community, we have been working on three key initiatives to improve cost management on Terra: Overview of Cloud Costs in Terra When you are working with data in Google Cloud, you are paying to store and analyze your […]<\/p>\n","protected":false},"author":64,"featured_media":1027,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[55],"tags":[213,217,215,216],"class_list":["post-1026","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cost-control","tag-cloud-costs","tag-cost-optimization","tag-cost-reporting","tag-spend-reporting"],"acf":[],"yoast_head":"\nInitiative 1. Cost Reporting<\/h2>\n\n\n\n
Reminder: Set up spend reporting<\/h4>\n\n\n\n
Improved spend reports<\/h4>\n\n\n\n
Monitor and analyze workflow costs as they run<\/h4>\n\n\n\n
Initiative 2. Cost Controls and Estimates<\/h2>\n\n\n\n
Workflow cost thresholds<\/h4>\n\n\n\n
Estimate costs of previous and in-progress workflows<\/h4>\n\n\n\n
Estimate costs for commonly-used workflows<\/h4>\n\n\n\n
Initiative 3: Cost Optimizations<\/h2>\n\n\n\n
Autoclass storage<\/h4>\n\n\n\n
Cleaning up intermediate workflow files<\/h4>\n\n\n\n
Requester pays for data egress<\/h4>\n\n\n\n
We want to hear from you!<\/h2>\n\n\n\n