The following are some of the demerits of using Apache Spark:
1. Since Spark utilizes more storage space compared to Hadoop and MapReduce, there may arise certain problems.
2. Developers need to be careful while running their applications in Spark.
3. Instead of running everything on a single node, the work must be distributed over multiple clusters.
4. Spark’s “in-memory†capability can become a bottleneck when it comes to cost-efficient processing of big data.
5. Spark consumes a huge amount of data when compared to Hadoop.
Posted Date:- 2021-09-25 06:23:40
Worker node refers to any node that can run the application code in a cluster. The driver program must listen for and accept incoming connections from its executors and must be network addressable from the worker nodes.
Worker node is basically the slave node. Master node assigns work and worker node actually performs the assigned tasks. Worker nodes process the data stored on the node and report the resources to the master. Based on the resource availability, the master schedule tasks.
Posted Date:- 2021-09-25 06:22:14
The following three file systems are supported by Spark:
1. Hadoop Distributed File System (HDFS).
2. Local File system.
3. Amazon S3.
Posted Date:- 2021-09-25 06:21:43
Parquet is a columnar format file supported by many other data processing systems. Spark SQL performs both read and write operations with Parquet file and consider it be one of the best big data analytics formats so far.
Parquet is a columnar format, supported by many data processing systems. The advantages of having a columnar storage are as follows:
1. Columnar storage limits IO operations.
2. It can fetch specific columns that you need to access.
3. Columnar storage consumes less space.
4. It gives better-summarized data and follows type-specific encoding.
Posted Date:- 2021-09-25 06:19:50
In such spark interview questions, try giving an explanation too (not just the name of the operators).
Property Operator: Property operators modify the vertex or edge properties using a user-defined map function and produce a new graph.
Structural Operator: Structure operators operate on the structure of an input graph and produce a new graph.
Join Operator: Join operators add data to graphs and generate new graphs.
Posted Date:- 2021-09-25 06:18:50
Catalyst optimizer leverages advanced programming language features (such as Scala’s pattern matching and quasi quotes) in a novel way to build an extensible query optimizer.
Posted Date:- 2021-09-25 06:18:21
To connect Hive to Spark SQL, place the hive-site.xml file in the conf directory of Spark.
Using the Spark Session object, you can construct a DataFrame.
result=spark.sql(“select * from <hive_table>â€)
Posted Date:- 2021-09-25 06:17:54
MLlib is scalable machine learning library provided by Spark. It aims at making machine learning easy and scalable with common learning algorithms and use cases like clustering, regression filtering, dimensional reduction, and alike.
Posted Date:- 2021-09-25 06:16:44
A Sparse vector is a type of local vector which is represented by an index array and a value array.
public class SparseVector
extends Object
implements Vector
Example: sparse1 = SparseVector(4, [1, 3], [3.0, 4.0])
where:
4 is the size of the vector
[1,3] are the ordered indices of the vector
[3,4] are the value
Do you have a better example for this spark interview question? If yes, let us know.
Posted Date:- 2021-09-25 06:13:41
DISK_ONLY - Stores the RDD partitions only on the disk
MEMORY_ONLY_SER - Stores the RDD as serialized Java objects with a one-byte array per partition
MEMORY_ONLY - Stores the RDD as deserialized Java objects in the JVM. If the RDD is not able to fit in the memory available, some partitions won’t be cached
OFF_HEAP - Works like MEMORY_ONLY_SER but stores the data in off-heap memory
MEMORY_AND_DISK - Stores RDD as deserialized Java objects in the JVM. In case the RDD is not able to fit in the memory, additional partitions are stored on the disk
MEMORY_AND_DISK_SER - Identical to MEMORY_ONLY_SER with the exception of storing partitions not able to fit in the memory to the disk
Posted Date:- 2021-09-25 06:12:43
Controlling the transmission of data packets between multiple computer networks is done by the sliding window. Spark Streaming library provides windowed computations where the transformations on RDDs are applied over a sliding window of data.
Posted Date:- 2021-09-25 06:11:38
1. map(func)
2. transform(func)
3. filter(func)
4. count()
The correct answer is c) filter(func).
Posted Date:- 2021-09-25 06:07:37
Caching also known as Persistence is an optimization technique for Spark computations. Similar to RDDs, DStreams also allow developers to persist the stream’s data in memory. That is, using the persist() method on a DStream will automatically persist every RDD of that DStream in memory. It helps to save interim partial results so they can be reused in subsequent stages.
The default persistence level is set to replicate the data to two nodes for fault-tolerance, and for input streams that receive data over the network.
Posted Date:- 2021-09-25 06:07:01
Discretized Streams is the basic abstraction provided by Spark Streaming.
It represents a continuous stream of data that is either in the form of an input source or processed data stream generated by transforming the input stream.
Posted Date:- 2021-09-25 06:06:11
Databricks connect is the way to connect the databricks cluster to local IDE on your local machine. You need to install the dataricks-connect client and then needed the configuration details like ADB url, token etc. Using all these you can configure the local IDE to run and debug the code on the cluster.
pip install -U "databricks-connect==7.3.*" # or X.Y.* to match your cluster version.
databricks-connect configure
Posted Date:- 2021-09-25 06:05:36
For performing the data analytics in databricks where the data source is the azure storage, in that scenario we need the way to connect the azure storage to the databricks. Once this connection is done we can load the file in data frame like a normal operation and can continue writing our code.
To connect the azure blob storage in the databricks, you need to mount the azure stoarge container in the databricks. This needed to be done once only. Once the mounting is done, we can starting access the files from azure blob storage using the mount directory name. For creating the mount you need to provide the SAS token, storage account name and container name.
dbutils.fs.mount(
source = "wasbs://<container-name>@<storage-account-name>.blob.core.windows.net",
mount_point = "/mnt/<mount-name>",
extra_configs = {"<conf-key>":dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>")})
Posted Date:- 2021-09-25 06:04:29
Sometimes while writing your code, you may needed the various third party dependencies. If you are using the databricks in the SCALA language then you may need the external jars, otherwise you are using the databricks in python then you may need to import the external module.
You need to understand that the importing of the dependency not happened at the notebook level however it happened over the cluster level. You have to add the external dependencies (jar/module) to the cluster runtime environment.
Posted Date:- 2021-09-25 06:03:33
This is another frequently asked spark interview question. A Lineage Graph is a dependencies graph between the existing RDD and the new RDD. It means that all the dependencies between the RDD will be recorded in a graph, rather than the original data.
The need for an RDD lineage graph happens when we want to compute a new RDD or if we want to recover the lost data from the lost persisted RDD. Spark does not support data replication in memory. So, if any data is lost, it can be rebuilt using RDD lineage. It is also called an RDD operator graph or RDD dependency graph.
Posted Date:- 2021-09-25 06:02:36
Spark Streaming is used for processing real-time streaming data. Thus it is a useful addition to the core Spark API. It enables high-throughput and fault-tolerant stream processing of live data streams. The fundamental stream unit is DStream which is basically a series of RDDs (Resilient Distributed Datasets) to process the real-time data. The data from different sources like Flume, HDFS is streamed and finally processed to file systems, live dashboards and databases. It is similar to batch processing as the input data is divided into streams like batches.
Posted Date:- 2021-09-25 06:01:30
1. Spark Core: Base engine for large-scale parallel and distributed data processing
2. Spark Streaming: Used for processing real-time streaming data
3. Spark SQL: Integrates relational processing with Spark’s functional programming API
4. GraphX: Graphs and graph-parallel computation
5. MLlib: Performs machine learning in Apache Spark
Posted Date:- 2021-09-25 06:01:12
Spark Core is the base engine for large-scale parallel and distributed data processing. The core is the distributed execution engine and the Java, Scala, and Python APIs offer a platform for distributed ETL application development. SparkCore performs various important functions like memory management, monitoring jobs, fault-tolerance, job scheduling and interaction with storage systems. Further, additional libraries, built atop the core allow diverse workloads for streaming, SQL, and machine learning. It is responsible for:
1. Memory management and fault recovery
2. Scheduling, distributing and monitoring jobs on a cluster
3. Interacting with storage systems
Posted Date:- 2021-09-25 06:00:14
An action helps in bringing back the data from RDD to the local machine. An action’s execution is the result of all previously created transformations. Actions triggers execution using lineage graph to load the data into original RDD, carry out all intermediate transformations and return final results to Driver program or write it out to file system.
reduce() is an action that implements the function passed again and again until one value if left. take() action takes all the values from RDD to a local node.
moviesData.saveAsTextFile(“MoviesData.txtâ€)
Posted Date:- 2021-09-25 05:59:43
Transformations are functions applied on RDD, resulting into another RDD. It does not execute until an action occurs. map() and filter() are examples of transformations, where the former applies the function passed to it on each element of RDD and results into another RDD. The filter() creates a new RDD by selecting elements from current RDD that pass function argument.
val rawData=sc.textFile("path to/movies.txt")
val moviesData=rawData.map(x=>x.split(" "))
As we can see here, rawData RDD is transformed into moviesData RDD. Transformations are lazily evaluated.
Posted Date:- 2021-09-25 05:59:22
As the name suggests, partition is a smaller and logical division of data similar to ‘split’ in MapReduce. It is a logical chunk of a large distributed data set. Partitioning is the process to derive logical units of data to speed up the processing process. Spark manages data using partitions that help parallelize distributed data processing with minimal network traffic for sending data between executors. By default, Spark tries to read data into an RDD from the nodes that are close to it. Since Spark usually accesses distributed partitioned data, to optimize transformation operations it creates partitions to hold the data chunks. Everything in Spark is a partitioned RDD.
Posted Date:- 2021-09-25 05:59:00
Every spark application has same fixed heap size and fixed number of cores for a spark executor. The heap size is what referred to as the Spark executor memory which is controlled with the spark.executor.memory property of the –executor-memory flag. Every spark application will have one executor on each worker node. The executor memory is basically a measure on how much memory of the worker node will the application utilize.
Posted Date:- 2021-09-25 05:58:39
Spark provides two methods to create RDD:
1. By parallelizing a collection in your Driver program.
2. This makes use of SparkContext’s ‘parallelize’
method val DataArray = Array(2,4,6,8,10)
val DataRDD = sc.parallelize(DataArray)
3. By loading an external dataset from external storage like HDFS, HBase, shared file system.
Posted Date:- 2021-09-25 05:58:26
Yes, MapReduce is a paradigm used by many big data tools including Spark as well. It is extremely relevant to use MapReduce when the data grows bigger and bigger. Most tools like Pig and Hive convert their queries into MapReduce phases to optimize them better.
Posted Date:- 2021-09-25 05:58:00
No, because Spark runs on top of YARN. Spark runs independently from its installation. Spark has some options to use YARN when dispatching jobs to the cluster, rather than its own built-in manager, or Mesos. Further, there are some configurations to run YARN. They include master, deploy-mode, driver-memory, executor-memory, executor-cores, and queue.
Posted Date:- 2021-09-25 05:57:44
Spark Core is the engine for parallel and distributed processing of large data sets. The various functionalities supported by Spark Core include:
* Scheduling and monitoring jobs
* Memory management
* Fault recovery
* Task dispatching
Posted Date:- 2021-09-25 05:57:27
There are a total of 4 steps that can help you connect Spark to Apache Mesos.
* Configure the Spark Driver program to connect with Apache Mesos.
* Put the Spark binary package in a location accessible by Mesos.
* Install Spark in the same location as that of the Apache Mesos.
* Configure the spark.mesos.executor.home property for pointing to the location where Spark is installed.
Posted Date:- 2021-09-25 05:56:30
Apache Spark stores data in-memory for faster processing and building machine learning models. Machine Learning algorithms require multiple iterations and different conceptual steps to create an optimal model. Graph algorithms traverse through all the nodes and edges to generate a graph. These low latency workloads that need multiple iterations can lead to increased performance.
Posted Date:- 2021-09-25 05:45:30
When Spark operates on any dataset, it remembers the instructions. When a transformation such as a map() is called on an RDD, the operation is not performed instantly. Transformations in Spark are not evaluated until you perform an action, which aids in optimizing the overall data processing workflow, known as lazy evaluation.
Posted Date:- 2021-09-25 05:44:55