ACID refers to the properties of atomicity, consistency, isolation, and durability, in database terminologies. These properties ensure that the database transactions (Add, Update, Select, Delete) are processed reliably. It is focused on recovery upon failure, should a transaction go wrong. Traditionally, these have been limited to relational/SQL databases, which are extensively used in relational data warehouses.
However, with the rise of NoSQL systems, the handling of non-relational/unstructured data has become a problem. Therefore, Data Lakes are quickly becoming the defacto standard in Online Analytical Processing systems and replacing data warehouses. With these modern systems in-place, ACID has been sidelined, which has resulted in a certain level of unreliability in transactions to surface.
Fortunately, Databricks has released Delta Lake, which brings back ACID to modern data warehousing and management and serves as a fast and efficient tool to manage and standardize the implementation of data lakes.
Discussion Topics: