The Development Of Data Warehouse - amazonia.fiocruz.br

The Development Of Data Warehouse Video

With the rise of the concept of data Lake in recent years, the comparison between data warehouse and data Lake in the industry has been constantly debated. Some people say that data lake is the next generation of big data platform, and major cloud manufacturers Dtaa also putting forward their own data Lake solutions. Some cloud data warehouse products have also added the feature of linkage with data lake.

But what is the difference between data warehouse and data lake? Is it a dispute of technical route?

What is the data lake

Is it a dispute over data management? Are the two incompatible or can they coexist harmoniously and even complement each other? The field of big data has developed for 20 years from the beginning of this century to the present. From the macro level, the development law can be highly summarized as follows:.

DATA WAREHOUSE SERVICES

Figure 1. First of all, the concept of data warehouse appeared much earlier than data lake, which can be traced back to the s when database was king. Therefore, it is necessary for us to sort out the time, the reason and the more important reasons behind the emergence of these nouns from the historical context.

Generally speaking, the development of data processing technology in the field of computer science can be divided into Datx stages. In the s, the concept of data warehouse was born. At this time, the concept of data warehouse is more about the methodology of how to manage multiple database instances in an enterprise. However, limited by the processing capacity of single database and the high price of multi machine database sub database and sub table for a long time, the data warehouse is still far away from ordinary enterprises and users.

People are even debating The Development Of Data Warehouse data warehouse centralized management or data mart centralized management by department and domain is more feasible.

Traditional database solutions can no longer provide computing power at an acceptable cost. The huge demand for data processing begins to find a breakthrough, and the era of big data begins to sprout. Inandthree classic papers GFS, MapReduce and BigTable of Google laid the basic technical framework of this Warrehouse data era, Developemnt distributed storage, distributed scheduling and distributed computing model. Then, almost at the same time, the excellent distributed technology system represented by Google, Microsoft cosmos and open-source Hadoop was born. These computing engines are optimized The Development Of Data Warehouse different scenarios, but they all use SQL language with very low threshold, which article source reduces the use cost of big data technology.

During this period, the technical route began to be subdivided. Integrated systems such as AWS redshift, Google bigquery, snowflake, including maxcompute, which are mainly promoted by cloud manufacturers, are called data warehouses in the era of big data.]