Data engineering is a software engineering approach to the building of data systems, to enable the collection and usage of data. This data is usually used to enable subsequent analysis and data science, which often involves machine learning. Making the data usable usually involves substantial computing and storage, as well as data processing.

History

Around the 1970s/1980s the term information engineering methodology (IEM) was created to describe database design and the use of software for data analysis and processing. These techniques were intended to be used by database administrators (DBAs) and by systems analysts based upon an understanding of the operational processing needs of organizations for the 1980s. In particular, these techniques were meant to help bridge the gap between strategic business planning and information systems. A key early contributor (often called the "father" of information engineering methodology) was the Australian Clive Finkelstein, who wrote several articles about it between 1976 and 1980, and also co-authored an influential Savant Institute report on it with James Martin. Over the next few years, Finkelstein continued work in a more business-driven direction, which was intended to address a rapidly changing business environment; Martin continued work in a more data processing-driven direction. From 1983 to 1987, Charles M. Richter, guided by Clive Finkelstein, played a significant role in revamping IEM as well as helping to design the IEM software product (user data), which helped automate IEM.

In the early 2000s, the data and data tooling was generally held by the information technology (IT) teams in most companies. Other teams then used data for their work (e.g. reporting), and there was usually little overlap in data skillset between these parts of the business.

In the early 2010s, with the rise of the internet, the massive increase in data volumes, velocity, and variety led to the term big data to describe the data itself, and data-driven tech companies like Facebook and Airbnb started using the phrase data engineer. Popular implementations include Apache Spark, and the deep learning specific TensorFlow. More recent implementations, such as Differential/Timely Dataflow, have used incremental computing for much more efficient data processing.

Storage

Data is stored in a variety of ways, one of the key deciding factors is in how the data will be used.

Data engineers optimize data storage and processing systems to reduce costs. They use data compression, partitioning, and archiving.

Databases

If the data is structured and some form of online transaction processing is required, then databases are generally used. Originally mostly relational databases were used, with strong ACID transaction correctness guarantees; most relational databases use SQL for their queries. However, with the growth of data in the 2010s, NoSQL databases have also become popular since they horizontally scaled more easily than relational databases by giving up the ACID transaction guarantees, as well as reducing the object-relational impedance mismatch. More recently, NewSQL databases — which attempt to allow horizontal scaling while retaining ACID guarantees — have become popular.

Data warehouses

If the data is structured and online analytical processing is required (but not online transaction processing), then data warehouses are a main choice. They enable data analysis, mining, and artificial intelligence on a much larger scale than databases can allow, Business analysts, data engineers, and data scientists can access data warehouses using tools such as SQL or business intelligence software.

  • Block storage splits data into regularly sized chunks;

Management

The number and variety of different data processes and storage locations can become overwhelming for users. This inspired the usage of a workflow management system (e.g. Airflow) to allow the data tasks to be specified, created, and monitored. The tasks are often specified as a directed acyclic graph (DAG).

Data modeling

Data modeling is the process of producing a data model, an abstract model to describe the data and relationships between different parts of the data.

Roles

Data engineer

A data engineer is a type of software engineer who creates big data ETL pipelines to manage the flow of data through the organization. This makes it possible to take huge amounts of data and translate it into insights. They are focused on the production readiness of data and things like formats, resilience, scaling, and security. Data engineers usually hail from a software engineering background and are proficient in programming languages like Java, Python, Scala, and Rust.

See also

  • Big data
  • Computer science
  • Information technology
  • List of data science software
  • Software engineering

References

Further reading

  • Ian Macdonald (1986). "Information engineering". in: Information Systems Design Methodologies. T.W. Olle et al. (ed.). North-Holland.
  • Ian Macdonald (1988). "Automating the Information engineering methodology with the Information Engineering Facility". In: Computerized Assistance during the Information Systems Life Cycle. T.W. Olle et al. (ed.). North-Holland.
  • James Martin and Clive Finkelstein. (1981). Information engineering. Technical Report (2 volumes), Savant Institute, Carnforth, Lancs, UK.
  • James Martin (1989). Information engineering. (3 volumes), Prentice-Hall Inc.
  • The Complex Method IEM
  • Rapid Application Development
  • Enterprise Engineering and Rapid Delivery of Enterprise Architecture
  • Data Engineering Vault, a connected knowledge hub covering data engineering concepts, tools, and patterns