Hive SerDe
I am a data engineer who is responsible for designing, building, maintaining, and testing the infrastructure and systems that are used to store, process, and analyze data. I work closely with data scientists and analysts to ensure that the data pipelines and systems are able to support the data needs of an organization.
I have a strong background in computer science and software engineering, and skilled in programming languages such as Python, Java, and SQL also familiar with database systems and big data technologies like Hadoop, Spark, and NoSQL databases.
Some of my key responsibilities as a data engineer:
Designing and building data pipelines to extract, transform, and load data from various sources Setting up and maintaining data storage and processing systems, including data warehouses and data lakes Collaborating with data scientists and analysts to understand their data needs and ensure that the data infrastructure can support their requirements Performing data quality checks and troubleshooting any issues that arise Implementing security and privacy measures to protect sensitive data
Hive SerDe
Hive uses SerDe and File format for IO operations on HDFS.
Serialization -> Converting Java Objects into bytes(String or Binary data)
De-serialization ->Converting bytes into Java Objects.
Hive uses inbuilt library for this kind of operation, In this library we have so many methods for different different file formats. Built-in SerDes are Avro, ORC, RegEx, Parquet, CSV, JsonSerDe, etc.
We can also create our own Serde and assign some properties over there.
lets take an example, I want to store csv file and in this file I have column which may have following kind of data Hi, Siva. How are you?. Do you know "SerDe". In this text we have comma which is used as separator in csv file and double quote which is used as quoted string in csv files. So our raw text should be like below, "Hi, Siva. How are you?. Do you know \"SerDe\"".
To do this we have some properties in CSV SerDe library. Those properties need to be applied while creating a table.
