The Emergence of Post-Relational Databasesa Shift to the Third Computing Paradigm by April 1997 ForewordThis paper was written by Strategic Marketing. It is designed for application developers, and especially those professionals who set strategic direction for application development organizations. The paper was prepared to help these individuals assess alternative database technologies as they prepare to build new systems. Most legacy applications will be rebuilt in the coming few years. These applications will be rebuilt for a wide range of reasons, including Year 2000 considerations, the need to incorporate new technologies (such as GUI and Internet capabilities), and the need to accommodate new and changing business requirements. Technology platform choice is often a key consideration when rebuilding applications. Database is among the most critical platform choices to be made. Unfortunately for those who must make DBMS decisions, database technology is now in a state of considerable flux. Just a few years ago the obvious answer to any DBMS question was "relational" and the real question became one of choice of vendor. Today however, there are serious alternatives to relational database that must be considered by any developer contemplating an application platform technology decision. This paper is designed to provide insight into the changing technologies in the database market. It is intended to help the professional developer make a choice that will work well for the seven-to-fifteen-year life that is common to enterprise-class applications. Strategic Marketing is a Park City, Utah-based management consulting firm that specializes in strategies for new technologies in the software industry. Strategic Marketing clients include IBM, Intel, Price Waterhouse, SCO, Amdahl, Tandem, SSA, PeopleSoft, Manugistics, VIASOFT, JYACC and Software AG. This report represents the opinions of its authors only. The report is
based on our own independent research and knowledge of the industry.
Information in this report was believed to be accurate at the time of its
publication but can become out-of-date quickly in the dynamics of the
information technology industry, and the reader is cautioned to seek current
information. © Copyright 1997 Strategic Marketing Table of Contents
The Emergence of Post-Relational Database TechnologyThe relational database wave has crested. Post-relational databases will soon be moving to the forefront. For many of you, it may seem inconceivable that such a venerable and well-entrenched IT standard as relational database (RDBMS) could be fading. If it is fading, could the SQL and RDBMS standard be eclipsed by something as ill-defined as "post-relational database" (PRDBMS)? Yet its life span today approaches that of previously "heavily entrenched" database technologies such as IDMS and IMS - technologies that at the end of their heyday seemed so entrenched as to be unassailable by competitors. RDBMS is fading simply because it does not suit some of the most significant
challenges currently facing developers. A shift in the fundamental computing
paradigm that drives the requirements for information technology is already
underway. This shift in paradigms makes RDBMS far less attractive and
post-relational far more so.
Return to the table of contents. Will RDBMS Simply Go Away?Of course we must put any "fading" of RDBMS in perspective. The vast inventory of legacy RDBMS-based applications will not be converted overnight; RDBMS sales will not wither immediately; nor will RDBMS vendors quickly fade away. A relational view of data will persist into the foreseeable future, irrespective of the future of relational data models -- at the very least, most PRDBMS systems will continue to provide SQL access. Pure relational database will even continue to grow in the short term, despite new factors that increasingly favor post-relational technology. It will be years before any significant decline will become obvious in the pure RDBMS technology sales. After all, the proprietary DBMS systems that preceded RDBMS (such as IMS and IDMS) were supposedly replaced more than a decade ago by RDBMS. Yet proprietary DBMS systems are still very much in production use in some of the most demanding and important "mission critical" applications. While relational database systems may fade over time, they will leave a legacy of two key technology benefits: standards and portability. Clearly, these key benefits will continue to be in the forefront of the post-relational wave (and those nascent PRDBMS systems that seem to be struggling in the market today are often those that omit one or both of these key benefits). Computer end-users highly value portability and standards. End-users view standards and portability as primary weapons to protect their system investment from the vagaries of vendors and even from the shifting sands within their own organization, such as mergers and divestiture of business units and the corresponding changes in computing platforms. VAR developers place even more value on portability and standards.
Portability and standards serve not only to not protect precious technology
investments but also to expand markets while lowering cost and risk.
Return to the table of contents. Has RDBMS delivered?The claims of RDBMS have often been delivered more in brochureware than in filed-released software. Let's look at some of the foremost tenets of the relational database movement: 1. End-user Queries and Reports. To a great extent the rise of RDBMS was fueled by the technology's promise of improved responsiveness to reporting and query needs, but has it delivered? While originally promoted as an end-user reporting and query language, today no one seriously considers SQL to be end-user technology. Indeed, many organizations are concerned about widespread use of SQL by programmers. For example, a key benefit promoted by multi-tier tool vendors and OO proponents is the ability to isolate SQL from application programmers. But even more troubling than the limitations of SQL, are the limitations of the relational data model. The inherent simplicity of the relational data model results in simplicity of data access for the most simple of implementations. When more complicated data structures are implemented in the relational model, scores or even hundreds of tables become common. In these complex environments, the relational model presents both a programming and performance nightmare. Where the relational model presents the greatest challenges is in complex transactional systems. With its report-writing heritage, the relational model suffers serious performance limitations in all but the most simplistic data model environments. While RDBMS vendors provide a variety of tricks and techniques for the database administrator to tweak performance behind the scenes (such as data clustering), achieving high performance in large scale, complex real world applications remains an elusive goal for most ardent RDBMS shops. The transactional system performance problems are even more challenging for VARs since an application developer does not have control of the DBA process at customer locations. 2. Standard and Portable. Does RDBMS constitute a "portable standard?" Generally far less than popularly believed. Performance for large complex systems dictates extensive coding of stored procedures and the use of other highly proprietary techniques. Complexity of system management and tuning is another barrier to true portability. And, the use of proprietary application development tools has further stymied portability. The greatest success for portability has been with low performance ODBC
connectivity, typically used for reporting and queries, most commonly by
so-called third party (non-DBMS-vendor) tool vendors. Hence, VARs and end-users
seeking database vendor independence have usually discovered, upon completion of
their application development, that portability is elusive.
Return to the table of contents. The Trends Driving Post-RelationalTo understand the underlying forces that are driving a slow but steady shift away from pure-relational to post-relational database technology, we must first understand the computing paradigms that fundamentally drive database requirements and markets. Relational wasn't just a better idea for DBMS. Rather, RDBMS was a response to a major change in the computing paradigm. As the computing paradigm continues to change, the value of RDBMS erodes. There have been three fundamental computing paradigms that have driven database technology: technology-centric computing, user-centric computing and network-centric computing. We are now moving away from the second paradigm: user-centric computing, towards the third paradigm of network-centric computing. Scott McNealy's prescient comment of some years ago - "the network is
the system" - describes today's major computing trend, which clearly favors
network-centric computing. The explosion of attention in client/server,
Internet, World Wide Web and Intranets are reflective of this shift in paradigm,
from user-centric to network-centric computing.
Return to the table of contents. The driving forces behind changing paradigmsWhat were the forces behind these three computing paradigms? The computing paradigm shifts are the result of fundamental changes in technology economics and business requirements. Let's look at the influences of changing technology economics and business requirements further. In the technology-centric paradigm, hardware was dear and people comparatively cheap. Accordingly, technology-centric computing emphasized technology productivity. For example, assembly language was prevalent in the technology- centric paradigm as it enabled the maximum productive use of computing hardware. In technology-centric computing, programmer productivity was clearly secondary to hardware performance optimization. The second paradigm of user-centric computing began as technology prices plummeted. Advances in integrated circuits and the emergence of a computer-on-a-chip (the microprocessor), slashed core computing costs. Costs dropped precipitously in data storage thanks to advances in chip-based memory and magnetic and optical physical-media-based data storage peripherals. Conversely, people became relatively dear in the second computing paradigm. Hence, people productivity was favored. Hardware performance optimization is clearly secondary in user-centric computing - indeed, bloated programs and operating systems are becoming nearly synonymous with user-centric computing. Originally, computers had an "awesome" and "intimidating" aura. But, as users became less afraid of computing and started demanding more service, particularly reporting and queries, Information Systems departments were quickly overrun by end-user report and query requests. The relational model and SQL emerged as technologies designed to simplify end-user report writing and query requirements. The simplification of data structures into rows and columns aided the development of a simple non-procedural query language (SQL) which in turn aided the development of more powerful and general report writing - putting query power into the hands of the user for the first time. RDBMS technology, however, was not driven by a need for high performance transaction processing which in many respects is at odds with its simplistic data structure. Primary examples of software technologies in the second computing paradigm include GUI/Windows, 4GLs for programmers and spreadsheets for end-users. These technologies are touted for their people-productivity benefit breakthroughs. In the third paradigm, which is rapidly building momentum today,
enterprise productivity is now emphasized. Integration is key to
leveraging enterprise productivity. In network-centric computing, enterprise
productivity is leveraged via integrating systems throughout the enterprise and
even beyond the enterprise. For example, in the manufacturing sector today, a
red hot business is "supply chain management," in which integration of
information from suppliers through manufacturers on to distributors, retailers
and customers, yields significant improvements in competitiveness, cost
reduction and customer satisfaction.
Return to the table of contents. Database technology and changing computing paradigmsThe rise and fall of various DBMS technologies parallel changes in computing
paradigms. Specifically, DBMS technologies are driven primarily by the
requirements for online information access. In the earliest days of computing,
there was no or minimal online access. Hence, there were no database management
systems. In primarily batch-oriented systems, data is processed at the
convenience of the computer. Sequential access, indexed sequential access, and
simple file systems, as opposed to database management systems, are adequate.
Return to the table of contents. Network/hierarchical DBMS for technology-centric computingThe emergence of practical online data access gave rise to the original need for DBMS technology. The first online systems emerged in the times of the technology-centric paradigm. Early DBMS systems focused on providing useful information in the context of the need to minimize costly hardware technology. Maximizing computer processing efficiency means data models that closely mirror usage. For example, basic order entry, customer service and MRP applications typically process information hierarchically or in a network fashion. Hence, hierarchical and network models predominated for these applications during the technology-centric era. In the technology-centric paradigm, proprietary DBMS systems are preferable. When processing efficiency is paramount, the proprietary DBMS, which is optimized specifically for proprietary computer hardware, will outperform a generalized DBMS. DBMS systems such as IDMS and IMS would become leaders during this time. While there may be a tendency to look back at both "hierarchical" and "proprietary" with equal disdain, we do not agree. Clearly, we have come to realize the enormous economic benefits that accrue from the use of standards, most notably the benefits of commoditization of hardware and the resultant dramatic decline of computing costs. What is less clear are the disadvantages of the hierarchical data model.
Quite obviously much of the world's information seems to naturally occur
hierarchically, and much of the performance problems in the relational model are
directly attributed to attempting to optimize hierarchical information
processing with normalized data structures.
Return to the table of contents. Relational DBMS for user-centric computingThe technology-centric paradigm ended with plummeting hardware costs. Breakthroughs in computing costs enabled the rise of the second computing paradigm: user-centric computing. The rise of the PC epitomized the user-centric paradigm. The responsive immediacy of PC computing showcases the benefits of boosting user productivity. User-centric computing also grew rapidly in the deployment of work group and departmental configurations. As the information technology era progressed, users became less afraid of computers and their information needs became more demanding (especially for information needs unanticipated by application programmers). Programmers, and programming productivity, became more important, and offloading report writing to sophisticated power users became important to protecting precious resources. At the center of user-centric computing is the rise in importance of standards, portability and flexibility. UNIX and DOS became the popular operating environments as they promised, and to varying extents delivered portability across a variety of vendors' hardware. Standards and portability lead to commoditized hardware. When hardware becomes commoditized the resultant competition increase serves to further lower costs. The relational database rose to prominence by showcasing the benefits of standards, portability and user empowerment for report writing. Unfortunately, the benefits of standards, portability and a report-writing-optimized data model are offset by RDBMS's clear disadvantages of slow performance and high resource consumption. Initial relational database usage centered on departmental and larger computers where performance and resource requirements were often a problem. RDBMS did not become popular on PCs until very powerful processors and copious memory became common. The relational database epitomizes resource-bloat much as GUI operating systems and applications. The increasingly escalating RDBMS benchmark performance claims primarily reflect phenomenal increases in hardware computing power and RDBMS vendor mastery of the simplistic transaction models used in standardized benchmark tests. These performance claims clearly reflect massive programmer attention to optimizing only a couple of benchmarks rather than the multitude of real world applications. In fact, the whole concept of standardized RDBMS benchmarks reflects, in many ways, the needs of inefficient RDBMS vendors to focus performance on a couple of things on which they can concentrate massive resources. Currently, most database benchmarks are based upon benchmarks developed specifically for the RDBMS industry, and the RDBMS vendors increasingly excel at meeting their own tests. The rise of relational database was also propelled by significantly changing business requirements. These changing business requirements fueled the drive to the user-centric computing paradigm. Increasing global competition required productivity breakthroughs. Leveraged buyouts and corporate raiders provided a carrot and stick to increased productivity. The required productivity boosts necessitated "changing the way business is done"- changes that meant more and more information-enabled decision-makers. More productive use of information emerged as a competitive advantage, empowering users. Empowered users require great flexibility in information systems. Information rigidity roadblocks users' decision support abilities and the ability to readily "change the way business is done." The relational database model promises a breakthrough in flexibility. Data,
now stored in third normal form, can be flexibly retrieved to meet all sorts of
unanticipated, ad hoc requirements. The relational data model is ideal for
supporting information empowered individual users working autonomously. It is
also ideal for small groups of autonomous users.
Return to the table of contents. Post-relational DBMS for network-centric computingThe business requirements for ever-increasing productivity continue to grow unabated. Increased competition and business restructuring are not one-time phenomena, but rather a way of life for the modern business. The productivity increase requirements exceed what can be obtained from collectively increasing individual productivity - what is required are enterprise and even extra-enterprise productivity boosters. Competitive business success is increasingly dimensioned by the complexity and scope of the information systems. The more users that are connected to more application functions, the more competitive the business potential. The explosion of user demand has outstripped the capacity of RDBMS to meet the shared access requirements of large quantities of data. More and more data is being captured and retained and used in operational activities. Today's critical IT focus is less often "can I ask questions and generate reports", but rather "how can I handle massive volumes of transactions." Increasingly we are realizing that the relational database, born of the needs of user-centric computing, is not up to the tasks required for network-centric computing and enterprise productivity requirements. Business success is less and less defined by collections of sophisticated individual end-users performing autonomous decision-making and asking unanticipated questions - the ideal computing environment for the highly flexible relational data model. Empowered users analyzing problems to optimize decisions does not constitute the bulk of activities performed in the typical enterprise environment. For example, in the large medical center, a few staff personnel and managers may be focused on information that can lead to reduced costs and improved outcomes. Hundreds or even thousands of staff and managers, however, perform the routine functions of the institution - admitting and discharging patients, performing lab and radiology tests, ordering procedures and dispensing medication. Increased automation of their tasks, therefore, is crucial to reducing costs. More and more businesses find success dimensioned by the numbers of users served. For example, transportation companies, who are often a generation ahead of other industries in computing usage, are well on their way to leveraging network-centric computing. Airlines, which were very early with online systems, have moved their online systems from hundreds of reservations clerks to the tens of thousands of employees enterprise wide covering every aspect of the business from crew scheduling to flight planning to maintenance - nearly every airline employee is connected online in the modern airline. Now, airlines compete for business through the automation of their distribution channel - the hundreds of thousands of travel agents worldwide. The airlines are on the forefront of competitive use of technology by connecting with their millions of customers directly via the World Wide Web. This environment continues to grow from hundreds, to thousands, to hundreds of thousands, to millions of users, and from central computers, to distributed computers to networks of computers within and outside the enterprise. While the airlines may be early to new information technologies, every other industry cannot be far behind. For example, only a few years ago many large manufacturers initiated enterprise-wide integrated systems from vendors such as SAP and Baan. Now, they are implementing Web-based order processing and customer services systems and integrated supplier-to-distributor automated supply chain management systems. Healthcare is in the throes of community health care networks. Federal Express and UPS promote shipping, scheduling and tracking via the Internet. Though computers have become less intimidating, they still retain an almost
mystical aura for most people. The introduction of RDBMS with its success in
providing query and ad-hoc reporting resulted in an image of highly advanced
technology that engendered rapid acceptability. Return to the table of contents. The Post-Relational DatabaseSpecial Challenges for PRDBMSWhy are experts increasingly noting that new database technology is required for effective network-centric computing? Simply stated, it is because the problems in network centric-computing are different than those in user-centric computing. Problems of "scale" are at the forefront in network-centric computing. There are three major dimensions to the scalar problem faced in network-centric:
On all three of these dimensions, scalability of one, two or more orders of magnitude are not uncommon. It should not be surprising that computing technology changes are required to accommodate the significant changes in scale found in network computing. Despite the enormous increase in transaction and data sizes, user demands for nearly instantaneous response are, if anything, increasing. Many of the users are now cross-enterprise or even extra-enterprise. Therefore, poor system performance may now represent "unacceptable customer service" rather than merely the "extra cost" (such as adding more clerks) that typified poor performance in traditional systems. The relational database was never conceived to handle the types of problems increasingly found in network-centric computing. It is ill suited for scaling up to massive sized databases with massive numbers of users and transactions. The relational benefits of reporting flexibility pale by comparison to the requirements for superior performance and extraordinary scalability. The post-relational technologies are emerging to take over where the
relational technologies leave off - with superior performance with massive
scalability. Yet the large RDBMS vendors are at a distinct disadvantage at
making the transition to providing these technologies despite their resources
due to: a) compatibility demands from a large client base, b) the constant
pressure for short term technology evolution, and c) the large bureaucratic
inertia-laden nature of these vendors which is not conducive to breakthrough
technology development. (The R&D environment of these firms is often more
conducive to implementing academic models designed for mathematical elegance by
computer scientists without practical application experience.)
Return to the table of contents. Special technologies for PRDBMSPost-relational database is not a single technology, but rather a collection of technologies that provides solutions for the most pressing problems of network-centric computing. These problems are those associated primarily with ensuring superior performance in large-scale environments for complex requirements. There is not a single "model" for post-relational that drives what technology must be included in PRDBMS products. Even with a defined relational model, and Dr. Codd's zealous push for strict adherence to pure relational standards, the various RDBMS systems evolved into a diversity of technologies. For example, much of what we associate today with relational database systems, such as "two phase commit" and "stored procedures," were not part of the original relational model. These and other database capabilities were developed over time by RDBMS vendors as the vendors sought to gain competitive advantage and to respond to evolving market requirements. In the end, it will be market acceptance that shapes the definition of post-relational database technology much as market acceptance shaped RDBMS. We believe that the successful PRDBMS products will continue to feature three of the prominent RDBMS capabilities, namely standards, portability and flexibility. Two of the more prominent distinctly post-relational components are Object Oriented (OO) and Multi-Dimensional Database (MDD) technologies. These technologies serve these key roles in a PRDBMS:
Return to the table of contents. Categories of PRDBMS SystemsHow will post-relational technology evolve? What will the product winners look like? Who will be the winning vendors? Answering these and similar questions involves predicting the future, certainly an imprecise and risky endeavor. We can, with some measure of reliability, comment on those key issues that will shape the post-relational market. Below are presented some of these key issues and our comments and predictions. Many industry analysts today classify a variety of DBMS systems as "post-relational." Object-Oriented Database systems (OODBMS) are nearly always included. Over time, OO will emerge as a key enabling technology for all post-relational databases. Furthermore, OO will be retrofitted into many relational systems as well. We question whether OO alone will emerge as the defining characteristic for post-relational DBMS. We do not see a long-term major market developing for Object Oriented database systems. Rather, we see the successful post-relational DBMS systems dividing into categories based on usage and problem focus, rather than technologies employed, such as OO. Technologies such as OO will then become part of the products deployed in the categories. The two key usage categories we see are as follows:
We believe that different MDD-based PRDBMS products will be used for OLAP and OLTP implementations. The OLAP and OLTP environments are markedly different. MDD for OLAP will commonly operate on large centralized data stores, such as a data warehouse, where a relatively few number of users (when contrasted to the number of enterprise employees) perform autonomously with different informational requirements. Conversely, T/MDD for OLTP will more commonly operate in a distributed database and distributed application environment where many of the enterprise employees access and update common information with a high number of similar transactions. For example, key technologies for OLAP-based products will include
integration with analytical, reporting and query tools. Key technologies for
OLTP-based products will include integrated networking capabilities and
information maintenance (add, change and delete) as the most common usage.
Return to the table of contents. Finding winning post-relational products and vendorsHow do you pick the winning vendors and products in the emerging post-relational marketplace? Clearly the existing relational database vendors will be participants in the post-relational market, but how successful will they be? If history provides any guidance, new vendors are more likely sources of effective post-relational database. Let's look at the transition from technology-centric to user-centric computing, from proprietary network/hierarchical to open relational DBMS. Only IBM would successfully transition from a leader in the proprietary DBMS arena (with IMS and DL/1) to a leader in relational DBMS (with DB/2). Many of the major proprietary network/hierarchical vendors have disappeared - two of the largest, ADR (Datacom) and Cullinane (IDMS), through acquisition by Computer Associates. Datacom, IDMS and others offered relational-like capabilities. The relational-market leaders, however, were new companies that specialized in relational database, such as Oracle, Sybase and Informix and Ingres (an RDBMS pioneer and former giant no longer viewed as a top tier RDBMS.). Unencumbered by the baggage of legacy technology and customers these vendors could build and deliver market-winning relational products. Of course, the software industry is different today than it was at the time of transition to relational from proprietary DBMS. The relational database vendors are larger companies and, thanks in great measure to a buoyant stock market, have much greater resources. Some of the vendors, notably Oracle, are better diversified in areas such as applications and services. These vendors may well be able to build new post-relational technologies or at least acquire them, although this would represent a significant shift from their commitment to the SQL engine. While hybrids of proprietary and relational did not fare well in the market, hybrid relational/post-relational may well do so. Certainly we would expect to see OO appear widely in database and tool technologies. The obvious concern of many developers is the potential risk in dealing with a "new vendor" to acquire leading post-relational technologies. However, buying the next generation post-relational technologies does not necessitate buying from a brand new start-up vendor. Because technology waves take so long to mature, many post-relational vendors have been in business for years and have revenue in the comfortable tens-of-millions-of-dollars range. This should come as no surprise since relational vendors were around for years growing slowly until the time came for the relational market to explode. Of course for these richer data models to succeed in the marketplace, they must also support traditional interoperability standards such as SQL. Increasingly, SQL as an interoperable query language will begin to be differentiated from an RDBMS as SQL is extended to incorporate some object features that don't fit well with the RDBMS model. While RDBMS vendors will be incorporating aspects of objects and other technologies into their offerings, the most likely scenario is that they will fundamentally retain their strong RDBMS nature with its older data model as the low level basis for their products. This strategy may pay off in marketing as it allows them to talk about hybrid models, but it fails to reap many of the technology rewards as they try to shoehorn richer technologies into a highly limited data model. Much stronger products will be those that incorporate a much richer and more sophisticated data model into the core of the product. It is much easier for them to then provide an access path for older technology interoperability without stunting the opportunities inherent in a more sophisticated and advanced technology. As with the car industry of the 1970s and 1980s, it will be interesting to
see what happens as entrenched large companies become challenged by more
technologically advanced and cost effective alternatives.
Return to the table of contents. How to proceedWe believe that the factors favoring an explosion of post-relational technology usage are very strong. These factors are based in fundamental changes that are occurring in technology, economics and business conditions. Therefore, any developer building a data warehouse or transactional application today should look hard at post-relational technologies when making an application platform decision. What are the key factors to look at when examining post-relational database technology? Standards, portability and flexibility remain hallmarks of database technology regardless of relational/post-relational technology choice. And, vendor issues are still important; for example, is a solution from a start-up worth the risk? In addition to these issues, any evaluation must also include those areas of special importance to your own development. Post-relational database evaluation criteria will also vary based on your primary usage: OLAP or OLTP. Below we have overviewed some suggested key criteria for evaluating alternative post-relational OLTP database technology. 1. PerformanceProblem/Needs
Benefits Sought
2. ProductivityProblems/Needs
Benefits Sought
3. Cost EffectiveProblem/Needs
Features
Benefits
Problem/Needs
Features
Benefits
Problem/Needs
Features
Benefits
Problem/Needs
Features
Benefits
7. ReliableProblem/Needs
Features
Benefits
Return to the table of contents. |