I will share my thoughts and notes about my studies
What is Apache Livy? Apache Livy is an open-source REST service for managing long-running Apache Spark jobs. It allows you to submit, manage, and interact with Spark jobs over a RESTful interface, making it easier to integrate Spark with other applic...
In Apache Kafka, a cluster is a group of one or more Kafka brokers working together to serve a set of topics and provide a distributed messaging service. The Kafka cluster is responsible for storing and replicating data across multiple brokers, allow...
In Apache Kafka, a consumer is a client application that reads messages from one or more topics in a Kafka cluster. The consumer is responsible for subscribing to a topic and receiving messages from Kafka brokers. Consumers in Kafka can be implement...
In Apache Kafka, a broker is a single node in a Kafka cluster. It is a server that is responsible for handling read and writes requests from producers and consumers, and for storing and replicating data across partitions. A broker in Kafka can be th...
In Apache Kafka, a partition is a unit of parallelism and scaling for topics. Each topic in Kafka can be split into one or more partitions, and each partition can be located on a different broker in the Kafka cluster. A partition is an ordered and im...
In Apache Kafka, a topic is a category or feed name to which messages are published. A topic in Kafka is similar to a table in a database or a folder in a file system. It's a logical container that holds a stream of records, where each record is a ke...
Producer In Apache Kafka, Producer is a client application that sends data to a Kafka cluster. The producer is responsible for publishing records on one or more Kafka topics. Each record is a Key-value pair stored in a topic partition within the clus...
Here we are going to learn Spark Memory Management Before starting we need to understand the below points clearly, One core will process one partition of data at a time Spark partition is equivalent to HDFS blocks and repartition is possible One t...
Spark Submit From Now on we are going to use spark Submit frequently So that we are going to learn the Syntax for Spark Submit first, Once the Spark application build is completed, we use to execute that application via the spark-submit command. spar...
