Why DBMS instead of Traditional file system?
I am a data engineer who is responsible for designing, building, maintaining, and testing the infrastructure and systems that are used to store, process, and analyze data. I work closely with data scientists and analysts to ensure that the data pipelines and systems are able to support the data needs of an organization.
I have a strong background in computer science and software engineering, and skilled in programming languages such as Python, Java, and SQL also familiar with database systems and big data technologies like Hadoop, Spark, and NoSQL databases.
Some of my key responsibilities as a data engineer:
Designing and building data pipelines to extract, transform, and load data from various sources Setting up and maintaining data storage and processing systems, including data warehouses and data lakes Collaborating with data scientists and analysts to understand their data needs and ensure that the data infrastructure can support their requirements Performing data quality checks and troubleshooting any issues that arise Implementing security and privacy measures to protect sensitive data
Why DBMS instead of File system?
File system will not be efficient for long run and frequent changes. File System useful only when need of bulk dump or temporary storage.
Advantages of DBMS over file system
1. Querying: DMBS uses indexing concept so we can query the data with help of predefined algorithms like Binary tree search and Hashing.
2. Data redundancy: No need of storing same kind of data like customer address can be stored in single table and we can use it further with help of table join.
3. Consistency: If any customer changes the address we just update the customer table alone and it will be automatically picked up while joining.
4. Data Independence: Hides the low level information like where it is stored, how it is stored and when it is loaded. Engineers not need to worry about this details.
5. Security: Critical Data can be locked for particular users and some data available only to few users. And this can be done in File systems too but it need high level knowledge on OS and other computer knowledge.
