What is spark Executor?

Executor is a distributed agent that is responsible for executing tasks. Executors are managed by executor backend (ExecutorBackend is a pluggable interface that TaskRunners use to send task status updates to a scheduler).
Executors reports heartbeat to HeartbeatReceiver RPC Endpoint on the driver. Executors provide in-memory storage for RDDs using via Block Manager.

BlockManager is a key-value store of blocks of data (block storage). BlockManager acts as a local cache that runs on every node in a Spark cluster. BlockManager provides interface for uploading and fetching blocks both locally and remotely using various stores.

Read my previous articles on Apache Spark

Apache Spark Architecture

Deep Understanding of SparkContext & Application’s Driver Process

Leave a Comment

Your email address will not be published. Required fields are marked *