{"id":87791,"date":"2023-02-09T17:12:47","date_gmt":"2023-02-09T17:12:47","guid":{"rendered":"https:\/\/www.techrepublic.com\/?p=4034559"},"modified":"2023-02-09T17:12:47","modified_gmt":"2023-02-09T17:12:47","slug":"for-enterprise-apis-is-zero-copy-integration-the-david-to-big-datas-goliath","status":"publish","type":"post","link":"https:\/\/cloudnewshub.com\/?p=87791","title":{"rendered":"For enterprise APIs, is Zero-Copy Integration the David to big data\u2019s Goliath?"},"content":{"rendered":"<figure id=\"attachment_4034560\" aria-describedby=\"caption-attachment-4034560\" class=\"wp-caption alignnone\"><img loading=\"lazy\" decoding=\"async\" class=\"size-article wp-image-4034560\" src=\"http:\/\/cloudnewshub.com\/wp-content\/uploads\/2023\/02\/for-enterprise-apis-is-zero-copy-integration-the-david-to-big-datas-goliath.jpg\" alt=\"blue digital binary data on computer screen\" width=\"770\" height=\"433\"><figcaption id=\"caption-attachment-4034560\" class=\"wp-caption-text\">Image: gonin\/Adobe Stock<\/figcaption><\/figure>\n<p>In Rodgers and Hammerstein\u2019s \u201cThe King and I,\u201d the King explains to \u201cI\u201d that the bee always flies from flower to flower, the flower never flies from bee to bee. That justification for philandering didn\u2019t fly with Mrs. Anna, but it does make sense when applied to the relationship between applications and data: Should data fly from application to application, or should the data stay put like a flower and let applications approach it on its terms?<\/p>\n<p>A new framework, formulated as an open standard that has just received the imprimatur of the Canadian government, is keeping data firmly rooted.<\/p>\n<p>Jump to:<\/p>\n<h2 id=\"integration\">What is Zero-Copy Integration?<\/h2>\n<p>Zero-Copy Integration is an initiative championed by the Canadian collaborative data company Cinchy. It aims to overturn the enterprise software <a href=\"https:\/\/technologyadvice.com\/blog\/information-technology\/how-to-use-an-api\/\" target=\"_blank\" rel=\"noopener noreferrer\">API integration<\/a> paradigm with a totally new model \u2014 the company calls it dataware \u2014 that keeps data effectively rooted while removing complexity and data redundancy from the enterprise software integration process.<\/p>\n<h2 id=\"benefits\">Benefits of Zero-Data Integration<\/h2>\n<p>Proponents of zero-copy integration and dataware say the framework will lower data storage costs, improve performance of IT teams, improve privacy and security of data, and drive innovation in systems for public health, social research, open banking and sustainability through innovations in:<\/p>\n<ul>\n<li>Application development and enrichment.<\/li>\n<li>Predictive analytics.<\/li>\n<li>Digital twins.<\/li>\n<li>Customer 360 technology.<\/li>\n<li>Artificial intelligence and machine learning.<\/li>\n<li>Workflow automation.<\/li>\n<li>Legacy system modernization.<\/li>\n<\/ul>\n<p><strong>SEE: <a href=\"https:\/\/www.techrepublic.com\/article\/big-data-productive-cloud\/\" target=\"_blank\" rel=\"noopener noreferrer\">Big data vs the right data: Becoming more productive in the cloud<\/a> (TechRepublic)<\/strong><\/p>\n<p>On Tuesday, Canada\u2019s Digital Governance Council and the not-for-profit Data Collaboration Alliance, created by Cinchy, announced CAN\/CIOSC 100-9, Data governance \u2013 Part 9: Zero-Copy Integration, a national standard approved by the Standards Council of Canada, to be published as an open standard.<\/p>\n<p>Read more about the announcement and Canada\u2019s Digital Governance Council in <a href=\"https:\/\/www.techrepublic.com\/article\/zero-copy-integration-framework-released\/\" target=\"_blank\" rel=\"noopener noreferrer\">this TechRepublic article<\/a>.<\/p>\n<h2 id=\"silos\">Zero-Copy Integration seeks to eliminate API-driven data silos<\/h2>\n<p>The basic idea, according to Dan DeMers, Cinchy\u2019s CEO, is that the framework aims to remove application data silos by using access-based data collaboration versus standard API-base data integration that involves copying data and branding it with complex app-specific coding. This would be done by access controls set in the data layer. It would also involve:<\/p>\n<ul>\n<li>Data governance via data products and <a href=\"https:\/\/www.techrepublic.com\/article\/data-stewardship-vs-data-governance\/\" target=\"_blank\" rel=\"noopener noreferrer\">federated stewardship<\/a>, not centralized teams.<\/li>\n<li>Prioritization of \u201cdata-centricity\u201d and active metadata over complex code.<\/li>\n<li>Prioritization of solution modularity over monolithic design.<\/li>\n<\/ul>\n<aside class=\"pinbox right\">\n<h3 class=\"heading\">Must-read big data coverage<\/h3>\n<\/aside>\n<p>The initiative said viable projects for Zero-Copy Integration include the development of new applications, predictive analytics, <a href=\"https:\/\/www.techrepublic.com\/article\/digital-twins-are-moving-into-the-mainstream\/\" target=\"_blank\" rel=\"noopener noreferrer\">digital twins<\/a>, customer 360 views, AI\/ML operationalization and workflow automations as well as legacy system modernization and SaaS application enrichment.<\/p>\n<p>DeMers, who is also technical committee member for the standard, promises a revolution in data.<\/p>\n<p>\u201cAt some point in a world of increasing complexity, you fall off a cliff, so we believe we\u2019re at the beginning of the simplification revolution,\u201d he said. \u201cThe fact is that data is becoming increasingly central, and the way that we share it is with APIs and <a href=\"https:\/\/www.techrepublic.com\/article\/what-is-etl\/\" target=\"_blank\" rel=\"noopener noreferrer\">ETLs<\/a>, which involves creating copies and vastly increases complexity and cost. It amounts to half the IT capacity of every complex organization on the planet, and every year it gets more expensive.\u201d<\/p>\n<p>He said even more concerning is that every time a copy is generated, a degree of control is lost.<\/p>\n<p>\u201cIf I run a bank, and I have a thousand applications, and they all need to interact with some representation of my customer, and by doing that are copying that representation, I now have a thousand copies of that customer,\u201d DeMers said. \u201cHow do I protect that?\u201d<\/p>\n<p><strong>SEE:<a href=\"https:\/\/www.techrepublic.com\/resource-library\/downloads\/data-governance-checklist\/\" target=\"_blank\" rel=\"noopener noreferrer\"> Data governance checklist for your organization<\/a> (TechRepublic Premium)<\/strong><\/p>\n<h2 id=\"security\">Security through Zero-Copy frameworks<\/h2>\n<p>Laws describing ownership of data limit how organizations or governments can use that data \u2014 but they are laws, not systematic controls, noted DeMers. A key point of the Zero-Data Integration argument, and Canada\u2019s adoption of a framework in principle, is that it makes data security easier by limiting access and control.<\/p>\n<p>\u201cZero Copy is a paradigm shift because it allows you to embed controls in the data itself,\u201d DeMers said. \u201cBecause it\u2019s access based, not copy based, access can be granted and it can be revoked, whereas copies are forever and you can quickly lose control over who has them, and any attempt to limit what organizations do when they obtain a copy is hard. \u201c<\/p>\n<p>Cinchy is aiming for a \u201cdata fabric architecture\u201d to transform data warehouses, lakes and\/or <a href=\"https:\/\/www.techrepublic.com\/article\/top-5-things-to-know-about-data-lakehouses\/\" target=\"_blank\" rel=\"noopener noreferrer\">lake houses<\/a> into repositories that can actualize both analytics and operational software. This is so apps can come to it, not carry copies of data back to the application walled garden.<\/p>\n<p>DeMers argued that the creation and storage of copies costs money, both because of storage and data pipelines and the time IT has to spend managing the iterations of data generated by hundreds or thousands of apps an enterprise may host.<\/p>\n<p>\u201cCopies of data require storage; the creation of the copy and synchronizing it not only uses storage, but also uses computation,\u201d he said. \u201cIf you imagine most of the processes running on servers in the bank right now, they\u2019re moving and reconciling copies of data, which constitutes energy use.\u201d<\/p>\n<p>He added that copying and moving data creates opportunities to introduce errors. If two systems connected by a data pipeline desync, then data can be lost or corrupted, reducing data quality. With one copy of the data used collectively by all systems, there\u2019s no chance of records appearing differently in different contexts.<\/p>\n<h2 id=\"dream\">Is Zero-Copy Integration an L.A. subway dream?<\/h2>\n<p>Matt McLarty, chief technology officer of Salesforce\u2019s MuleSoft, agrees that data replication is a perennial issue.<\/p>\n<p>\u201cNot even data replication, but the existence of semantically equivalent data in different places,\u201d he said.<\/p>\n<p>He sees it as a bit like Los Angeles and subways: A great idea in principle, but nobody is going to tear Los Angeles down and rebuild it around mass transit.<\/p>\n<p>\u201cIt\u2019s both a huge issue but also an unavoidable reality,\u201d he said. \u201cFrom a problem statement, yes, but I would say there are multiple categories of software in the space, including Salesforce Genie, all about how you harness all of the customer data widely dispersed across the ecosystem.\u201d<\/p>\n<p><strong>SEE: <a href=\"https:\/\/www.techrepublic.com\/article\/mulesoft-study-companies-applications-disconnect\/\" target=\"_blank\" rel=\"noopener noreferrer\">Study: Companies have upwards of 1,000 apps but only a third are integrated<\/a> (TechRepublic)<\/strong><\/p>\n<h2 id=\"lake\">Operational elephants and analytical zebras drinking from the same data lake<\/h2>\n<p>Most enterprises, explained McLarty, have two massive areas of data that, while not at cross purposes, need to live separately: operational data and <a href=\"https:\/\/www.techrepublic.com\/article\/data-analytics-growth-in-down-market\/\" target=\"_blank\" rel=\"noopener noreferrer\">analytical data<\/a>. Operational data is employed by such user-facing applications as mobile banking; analytical data takes data out of the flow of operational activities and uses it for business analytics and intelligence.<\/p>\n<p>\u201cThey have historically lived separately because of the processing differences,\u201d he said. \u201cOperationally, there\u2019s high speed, high-scale processing and analytically, small internal groups crunching big numbers.\u201d<\/p>\n<p>DeMers explained that what dataware does, among other things, is to incorporate \u201coperational data fabric.\u201d This, he said, makes \u201clast time\u201d integration from external data sources to an architecture based on a \u201cnetwork of datasets\u201d that\u2019s capable of powering unlimited business models.<\/p>\n<p>\u201cOnce created, these models can be readily operationalized as metadata-based experiences or exposed as APIs to power low code and pro code UX designs,\u201d he said, adding that it eliminates the need to stand up new databases, perform point-to-point data integration or set app-specific data protections.<\/p>\n<p>\u201cAnother core concept associated with dataware technology is \u2018collaborative intelligence,\u2019 which is created as a result of users and connected systems, simultaneously enriching the information within the dataset network,\u201d he said.<\/p>\n<p>DeMers said users granted access to a dataset by its owners get an interface called a \u201cdata browser\u201d offering a \u201cself-serve experience.\u201d<\/p>\n<p>\u201cIn principle, this works a bit like Google Docs, where multiple colleagues collaborate on a white paper or business proposal while the software automatically offers grammatical suggestions and manages roles, permissions, versioning and backup,\u201d he said.<\/p>\n<p>DeMers added that the end result is super-enriched and auto-protected data that can be instantly queried by teams to power unlimited dashboards, 360 views and other analytics projects.<\/p>\n<h2 id=\"chaos\">Will companies simplify or \u201cembrace the chaos?\u201d<\/h2>\n<p>By some estimates, companies are taking the \u201cembrace the chaos\u201d route to find new approaches that concede that the enterprise data frameworks will remain complex and L.A.-like. These include <a href=\"https:\/\/www.eweek.com\/enterprise-apps\/data-mesh\/\" target=\"_blank\" rel=\"noopener noreferrer\">data mesh<\/a> frameworks and automation and machine learning systems creating models that integrate different kinds of data.<\/p>\n<p>\u201cI think the biggest shift right now in the world of data is that the two worlds \u2014 analytical and operational \u2014 are colliding,\u201d McLarty said. \u201cWhat\u2019s happening now, because of the big data movement and machine learning, is data-derived coding \u2014 writing code with data, ingesting data and producing machine learning models based on the data that I can put into my applications.\u201d<\/p>\n<p>DeMers said that the dataware paradigm enables data mesh concepts.<\/p>\n<p>\u201cRequiring a single team to manage every dataset in the organization is a sure path to failed <a href=\"https:\/\/www.techrepublic.com\/article\/data-governance-framework\/\" target=\"_blank\" rel=\"noopener noreferrer\">data governance<\/a>,\u201d he said.<\/p>\n<p>He also argued that in a data-centric organization, data stewards should reflect the granularity of your organization chart.<\/p>\n<p>\u201cThis approach to federated data governance organized around data domains and data products is the data mesh, and it\u2019s a big part of establishing a more agile enterprise,\u201d DeMers said.<\/p>\n<p>Data silos make this difficult because of the unrestricted point-to-point data integration that it involves.<\/p>\n<h2 id=\"liberating\">Liberating data from the application<\/h2>\n<p>Sylvie Veilleux, former chief information officer of Dropbox, said data silos are a fundamental part of the <a href=\"https:\/\/www.techrepublic.com\/article\/software-as-a-service-saas-a-cheat-sheet\/\" target=\"_blank\" rel=\"noopener noreferrer\">Software as a Service<\/a> ecosystem, but that is a problem dataware can solve.<\/p>\n<p>\u201cEvery app solves a specific and unique purpose, and they are tending toward more and more specialization, she said. \u201cThe more SaaS adoption continues, which is very healthy in terms of how the business gets access to tools, the more it\u2019s continuously creating a hundred, thousand or more data silos in larger corporations. This number will continue to grow without us taking a whole new approach to how we think about data applications.\u201d<\/p>\n<p>She said dataware and Zero-Data Integration allows enterprises to eliminate extra data integrations by having the app connect to a network data source.<\/p>\n<p>\u201cIt changes how we work by pivoting the process from data being the captive of an application to keeping it on a network, thereby letting users collaborate, and giving businesses real-time access to it,\u201d Veilleux said.<\/p>\n<p>With data repositories moving to the cloud, a boon to collaboration, companies have more flexibility and reduced costs, but at what cost to security and threats? <a href=\"https:\/\/www.techrepublic.com\/resource-library\/whitepapers\/cloud-data-storage-policy\/\" target=\"_blank\" rel=\"noopener noreferrer\">Download this TechRepublic Premium policy<\/a>, which includes guidelines that will help you achieve secure cloud data management for integrity and privacy of company-owned information.<\/p>\n<p> <!-- default newsletter at the end --> <\/p>\n","protected":false},"excerpt":{"rendered":"<p>Image: gonin\/Adobe Stock In Rodgers and Hammerstein\u2019s \u201cThe King and I,\u201d the King explains to \u201cI\u201d that the bee always flies from flower to flower, the flower never flies from bee to bee. That justification for philandering didn\u2019t fly with Mrs. Anna, but it does make sense when applied to the relationship between applications and [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":87792,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[39,40,783],"tags":[],"class_list":["post-87791","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-big-data","category-cloud","category-cloudsync"],"_links":{"self":[{"href":"https:\/\/cloudnewshub.com\/index.php?rest_route=\/wp\/v2\/posts\/87791","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cloudnewshub.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cloudnewshub.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cloudnewshub.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/cloudnewshub.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=87791"}],"version-history":[{"count":0,"href":"https:\/\/cloudnewshub.com\/index.php?rest_route=\/wp\/v2\/posts\/87791\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cloudnewshub.com\/index.php?rest_route=\/wp\/v2\/media\/87792"}],"wp:attachment":[{"href":"https:\/\/cloudnewshub.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=87791"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cloudnewshub.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=87791"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cloudnewshub.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=87791"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}