{"id":283,"date":"2020-11-17T15:03:05","date_gmt":"2020-11-17T15:03:05","guid":{"rendered":"https:\/\/terrabioappdev.wpenginepowered.com\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/"},"modified":"2023-12-27T04:54:31","modified_gmt":"2023-12-27T04:54:31","slug":"update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos","status":"publish","type":"post","link":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/","title":{"rendered":"Update your Terra Notebooks Utilities for continued access to data via DRS\/DOS"},"content":{"rendered":"<p><strong><em>TL;DR:\u00a0<\/em><em>If you have been using the Terra Notebook Utilities (TNU) to access data through DRS\/DOS URIs, you need to update your version of TNU before December 1, 2020 as described further below.\u00a0<\/em><\/strong><\/p>\n<p>Some of the many data repositories that are accessible through Terra use systems called Data Repository Service (DRS, pronounced &#8220;duhrs&#8221;) and Data Object Service (DOS) to manage file locations in a way that allows you to get access to data without having to know exactly where it is stored. In other words, you can run an analysis on the data without actually knowing the exact path to where it lives. Without going into the details of how this sorcery works, the basic idea is that you give the system a unique identifier, and it gives whatever tool you&#8217;re using access to the data.<\/p>\n<p><em>You can learn more about the DRS\/DOS system and how to use it in Terra in\u00a0<\/em><a href=\"https:\/\/support.terra.bio\/hc\/en-us\/articles\/360039330211\" target=\"_blank\" rel=\"noopener\">this documentation article<\/a><em>.\u00a0<\/em><\/p>\n<p>If you&#8217;ve already been using a dataset that&#8217;s accessed through DRS\/DOS, you&#8217;ve probably had to use identifiers to work with some or all of the data.<\/p>\n<p>For workflows this is pretty transparent; you just point to the identifiers listed in the data table as your inputs, and the Cromwell workflow manager will work with Terra&#8217;s DRS\/DOS processing system, which is called Martha, to get the files localized at runtime. All the relevant components are managed for you behind the scenes so you don&#8217;t need to do anything to stay up to date.<\/p>\n<p>For notebooks there&#8217;s an extra layer involved; you have to use a Python package called Terra Notebook Utilities (TNU) to connect to Martha and access the files. TNU is a package you install yourself in your notebook environment, so you may occasionally need to update your version of the package to keep up with system updates.<\/p>\n<p><strong>Which is where today&#8217;s ACTION ITEM comes in.\u00a0<\/strong>Our engineering team has recently updated the Martha service to provide new functionality, including better error messages! This is valuable progress, but it involves some functional changes that require updating the Terra Notebook Utilities to use the new version.\u00a0<strong>Importantly, the old version will stop working on December 1, 2020, so you must update your installed version of the TNU package (to version 0.5.0 or later) if you want to continue accessing DRS\/DOS-mediated datasets.\u00a0<\/strong><\/p>\n<p>The good news is that the update process is fairly straightforward; you just need to use the command corresponding to the environment you&#8217;re working in:<\/p>\n<p><strong>From any Jupyter notebook in Terra:<\/strong>\u00a0(be sure to include the leading &#8220;%&#8221;)<\/p>\n<pre><em><span style=\"color: #993366;\">%pip install --upgrade --no-cache-dir terra-notebook-utils<\/span><\/em><\/pre>\n<p><strong>From the CLI on standard Terra-provided Notebook environments:<\/strong><\/p>\n<pre><em><span style=\"color: #993366;\">\/usr\/local\/bin\/pip install --upgrade --no-cache-dir terra-notebook-utils<\/span><\/em><\/pre>\n<p>Note that all standard Notebook environments on Terra are based on\u00a0<a href=\"https:\/\/github.com\/databiosphere\/terra-docker#terra-base-images\" target=\"_blank\" rel=\"noopener\">this Docker image<\/a>.<\/p>\n<p><strong>For other environments:<\/strong>\u00a0it should be enough to do the following:<\/p>\n<pre><em><span style=\"color: #993366;\">pip install --upgrade --no-cache-dir terra-notebook-utils<\/span><\/em><\/pre>\n<p>If you run into any trouble, please reach out to the Terra support team through the Helpdesk form or the\u00a0<a href=\"https:\/\/support.terra.bio\/hc\/en-us\/community\/topics\/360000500432-General-Discussion\" target=\"_blank\" rel=\"noopener\">community forum<\/a>. For more information on how to use the Terra Notebook Utilities to access data through DRS\/DOS, see\u00a0<a href=\"https:\/\/support.terra.bio\/hc\/en-us\/articles\/360039330211\" target=\"_blank\" rel=\"noopener\">this documentation article<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you have been using the Terra Notebook Utilities (TNU) to access data through DRS\/DOS URIs, you need to update your version of TNU before December 1, 2020, as described further in this article. Read on to learn more.<\/p>\n","protected":false},"author":7,"featured_media":284,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[18,44,45],"tags":[],"class_list":["post-283","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data","category-interactive-analysis","category-notebooks"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.0 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Update your Terra Notebooks Utilities for continued access to data via DRS\/DOS - Terra<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Update your Terra Notebooks Utilities for continued access to data via DRS\/DOS - Terra\" \/>\n<meta property=\"og:description\" content=\"If you have been using the Terra Notebook Utilities (TNU) to access data through DRS\/DOS URIs, you need to update your version of TNU before December 1, 2020, as described further in this article. Read on to learn more.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/\" \/>\n<meta property=\"og:site_name\" content=\"Terra\" \/>\n<meta property=\"article:published_time\" content=\"2020-11-17T15:03:05+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-12-27T04:54:31+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Networks_in_a_Field.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"627\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Kyle Vernest\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Kyle Vernest\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/\"},\"author\":{\"name\":\"Kyle Vernest\",\"@id\":\"https:\/\/terra.bio\/#\/schema\/person\/f9107f1ce8c3d544c61f1bbd0a22081c\"},\"headline\":\"Update your Terra Notebooks Utilities for continued access to data via DRS\/DOS\",\"datePublished\":\"2020-11-17T15:03:05+00:00\",\"dateModified\":\"2023-12-27T04:54:31+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/\"},\"wordCount\":521,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/terra.bio\/#organization\"},\"image\":{\"@id\":\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Networks_in_a_Field.png\",\"articleSection\":[\"Data\",\"Interactive Analysis\",\"Notebooks\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/\",\"url\":\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/\",\"name\":\"Update your Terra Notebooks Utilities for continued access to data via DRS\/DOS - Terra\",\"isPartOf\":{\"@id\":\"https:\/\/terra.bio\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Networks_in_a_Field.png\",\"datePublished\":\"2020-11-17T15:03:05+00:00\",\"dateModified\":\"2023-12-27T04:54:31+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#primaryimage\",\"url\":\"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Networks_in_a_Field.png\",\"contentUrl\":\"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Networks_in_a_Field.png\",\"width\":1200,\"height\":627,\"caption\":\"Networks_in_a_Field\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/terra.bio\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Update your Terra Notebooks Utilities for continued access to data via DRS\/DOS\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/terra.bio\/#website\",\"url\":\"https:\/\/terra.bio\/\",\"name\":\"Terra\",\"description\":\"Science at Scale\",\"publisher\":{\"@id\":\"https:\/\/terra.bio\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/terra.bio\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/terra.bio\/#organization\",\"name\":\"Terra\",\"url\":\"https:\/\/terra.bio\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/terra.bio\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Terra-Bio-App@2x.webp\",\"contentUrl\":\"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Terra-Bio-App@2x.webp\",\"width\":287,\"height\":318,\"caption\":\"Terra\"},\"image\":{\"@id\":\"https:\/\/terra.bio\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/terra.bio\/#\/schema\/person\/f9107f1ce8c3d544c61f1bbd0a22081c\",\"name\":\"Kyle Vernest\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/terra.bio\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/6f02e1252581e9e225f2c1bfe6232b192af997438cfc67a13339f2179ed11eef?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/6f02e1252581e9e225f2c1bfe6232b192af997438cfc67a13339f2179ed11eef?s=96&d=mm&r=g\",\"caption\":\"Kyle Vernest\"},\"url\":\"https:\/\/terra.bio\/author\/kvernest\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Update your Terra Notebooks Utilities for continued access to data via DRS\/DOS - Terra","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/","og_locale":"en_US","og_type":"article","og_title":"Update your Terra Notebooks Utilities for continued access to data via DRS\/DOS - Terra","og_description":"If you have been using the Terra Notebook Utilities (TNU) to access data through DRS\/DOS URIs, you need to update your version of TNU before December 1, 2020, as described further in this article. Read on to learn more.","og_url":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/","og_site_name":"Terra","article_published_time":"2020-11-17T15:03:05+00:00","article_modified_time":"2023-12-27T04:54:31+00:00","og_image":[{"width":1200,"height":627,"url":"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Networks_in_a_Field.png","type":"image\/png"}],"author":"Kyle Vernest","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Kyle Vernest","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#article","isPartOf":{"@id":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/"},"author":{"name":"Kyle Vernest","@id":"https:\/\/terra.bio\/#\/schema\/person\/f9107f1ce8c3d544c61f1bbd0a22081c"},"headline":"Update your Terra Notebooks Utilities for continued access to data via DRS\/DOS","datePublished":"2020-11-17T15:03:05+00:00","dateModified":"2023-12-27T04:54:31+00:00","mainEntityOfPage":{"@id":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/"},"wordCount":521,"commentCount":0,"publisher":{"@id":"https:\/\/terra.bio\/#organization"},"image":{"@id":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#primaryimage"},"thumbnailUrl":"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Networks_in_a_Field.png","articleSection":["Data","Interactive Analysis","Notebooks"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/","url":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/","name":"Update your Terra Notebooks Utilities for continued access to data via DRS\/DOS - Terra","isPartOf":{"@id":"https:\/\/terra.bio\/#website"},"primaryImageOfPage":{"@id":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#primaryimage"},"image":{"@id":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#primaryimage"},"thumbnailUrl":"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Networks_in_a_Field.png","datePublished":"2020-11-17T15:03:05+00:00","dateModified":"2023-12-27T04:54:31+00:00","breadcrumb":{"@id":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#primaryimage","url":"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Networks_in_a_Field.png","contentUrl":"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Networks_in_a_Field.png","width":1200,"height":627,"caption":"Networks_in_a_Field"},{"@type":"BreadcrumbList","@id":"https:\/\/terra.bio\/update-your-terra-notebooks-utilities-for-continued-access-to-data-via-drs-dos\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/terra.bio\/"},{"@type":"ListItem","position":2,"name":"Update your Terra Notebooks Utilities for continued access to data via DRS\/DOS"}]},{"@type":"WebSite","@id":"https:\/\/terra.bio\/#website","url":"https:\/\/terra.bio\/","name":"Terra","description":"Science at Scale","publisher":{"@id":"https:\/\/terra.bio\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/terra.bio\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/terra.bio\/#organization","name":"Terra","url":"https:\/\/terra.bio\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/terra.bio\/#\/schema\/logo\/image\/","url":"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Terra-Bio-App@2x.webp","contentUrl":"https:\/\/terra.bio\/wp-content\/uploads\/2023\/12\/Terra-Bio-App@2x.webp","width":287,"height":318,"caption":"Terra"},"image":{"@id":"https:\/\/terra.bio\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/terra.bio\/#\/schema\/person\/f9107f1ce8c3d544c61f1bbd0a22081c","name":"Kyle Vernest","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/terra.bio\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/6f02e1252581e9e225f2c1bfe6232b192af997438cfc67a13339f2179ed11eef?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/6f02e1252581e9e225f2c1bfe6232b192af997438cfc67a13339f2179ed11eef?s=96&d=mm&r=g","caption":"Kyle Vernest"},"url":"https:\/\/terra.bio\/author\/kvernest\/"}]}},"_links":{"self":[{"href":"https:\/\/terra.bio\/wp-json\/wp\/v2\/posts\/283","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/terra.bio\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/terra.bio\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/terra.bio\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/terra.bio\/wp-json\/wp\/v2\/comments?post=283"}],"version-history":[{"count":0,"href":"https:\/\/terra.bio\/wp-json\/wp\/v2\/posts\/283\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/terra.bio\/wp-json\/wp\/v2\/media\/284"}],"wp:attachment":[{"href":"https:\/\/terra.bio\/wp-json\/wp\/v2\/media?parent=283"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/terra.bio\/wp-json\/wp\/v2\/categories?post=283"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/terra.bio\/wp-json\/wp\/v2\/tags?post=283"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}