Databricks Interview Questions

1. What is Databricks?

Databricks is a Cloud-based industry-leading data engineering platform designed to process & transform huge volumes of data. Databricks is the latest big data tool that was recently added to Azure.

2. What is DBU?

Databricks Unified platform is a Databricks unit used to process the power, and it is also used to measure the pricing purposes.

3. Is Azure Databricks different from databricks?

Azure Databricks is an Artificial intelligence service developed by Microsoft and Databricks jointly to introduce innovation in data analytics, machine learning, and data engineering.

4. What is caching?

A cache is a temporary storage. The process of storing the data in this temporary storage is called caching.

Whenever you return to a recently used page, the browser will retrieve the data from the cache instead of recovering it from the server, which saves time and reduces the burden on the server.

5. Is it ok to clear the cache?

Yes, as cache stores all the irrelevant information(that is, kind of files that are not helpful to the operation of any application), so there is no problem in deleting or clearing the cache.

6. List different types of caching?

There are four types of caching they are:

1. Data caching
2. Web caching
3. Distributed caching
4. Output or Application caching

7. Do we need to store the results of one action in other variables?

No, there is no need to store the results of one action in other variables.

8. Which SQL version is used by databricks?

Spark implements ANSI 2003

Syntax: https://spark.apache.org/releases/spark-release-2-0-0.html

9. What are the different types of pricing tiers available in Databricks?

There are two types of pricing tiers available in Databricks they are:

1. Premium Tier
2. Standard Tier

10. What is the use of Kafka?

Whenever Azure Databricks want to collect or stream the data, it connects to Event hubs and sources like Kafka.

11. What is the use of the databricks file system?

Databricks file system is a distributed file system used to ensure data reliability even after eliminating the cluster in Azure databricks.

12. What are the different ETL operations done on data in Azure Databricks?

The different ETL operations performed on data in Azure Databricks are:

1. The data is transformed from the databricks to the data warehouse.
2. Bold storage is used to load the data.
3. Bold storage acts as temporary storage of the data.

13. How to generate a personal access token in databricks?

We can generate a personal access token in seven steps they are:

1. In the upper right corner of Databricks workspace, click the icon named: “user profile.”
2. In the second step, you have to choose “User setting.”
3. navigate to the tab called “Access Tokens.”
4. Then you can find a “Generate New Token” button. Click it.

14. How to revoke a personal access token?

We have to follow five steps to revoke a personal access token they are:

1. In the upper right corner of Databricks workspace, click the icon named: “user profile.”
2. In the second step, you have to choose “User setting.”
3. navigate to the tab called “Access Tokens.”
4. In this step, you have to click x for the token you need to revoke.

5. Finally, click the button “Revoke Token” on the Revoke Token dialog.

15. What is the purpose of databricks runtime?

Databricks runtime is used to run the set of components on the databricks platform.

16. What is an azure data lake?

An Azure data lake is a public cloud that enables all Microsoft users, scientists, business professionals, and developers to gain perspicacity from vast and complicated data sets.

17. What does Azure data lake do?

Azure data lake works amidst IT investments for managing, securing, and identifying data governance and management. It also allows us to extend the data applications by combining data warehouses and operational stores.

18. Which storage generation of Data lake is used by Azure synapse?

Azure Data lake storage generation2(Gen2) is used by Azure synapse.

19. Why should one maintain backup Azure blob storage?

Even though blob storage supports data replication, it may not handle the application errors that can crash the entire data. For this reason, we need to maintain backup Azure blob storage.

20. What is a Recovery Services Vault?

Recovery Services Vault is where the azure backups are stored. We can easily configure the data using RSV(Recovery Services Vault).

21. How to reuse the code in the azure notebook?

If we want to reuse the code in the azure notebook, then we must import that code into our notebook. We can import it in two ways–> 1) if the code is in a different workspace, we have to create a module/jar of the code and then import it into a module or jar. 2) if the code is in the same workspace, we can directly import and reuse it.

22. Write the syntax to connect the Azure storage account and databricks?

dbutils.fs.mount( source = “wasbs://@.blob.core.windows.net”, mount_point = “/mnt/”, extra_configs = {“”:dbutils.secrets.get(scope = “”, key = “”)})

23. What is the use of ‘%sql’?

‘%sql’ is used to switch the scala/python notebook into a mere SQL notebook.

24. What is a databricks cluster?

A databricks cluster is a group of configurations and computation resources on which we can run data science, data analytics workloads, data engineering, like production ETL ad-hoc analytics, pipelines, machine learning, and streaming analytics.

25. What can we do using API or command-line interface?

By using databricks API or command-line interface, we can:

1. Schedule the jobs.
2. Create/Delete or View jobs.
3. We can immediately run the jobs.
4. We can make it dynamic by passing the parameters at runtime.

26. List the different types of cluster modes in the azure databricks?

There are three different types of cluster modes in the azure databricks they are:

1. Single-node Cluster.
2. Standard Cluster.
3. High Concurrency Cluster.

27. What is the use of Continuous Integration?

Continuous Integration allows various developers to combine the code modification into the central repository. Every combination triggers an automated build that compiles and executes the unit tests.

28. What is a CD(Continuous Delivery)?

Continuous delivery (CD) elaborates on CI by expediting code modifications to various environments like QA and staging after the build is completed. Moreover it also used to test new changes for stability, performance, and security.

29. What are the critical challenges for CI/CD while building a data pipeline?

The five critical challenges for CI/CD while building a data pipeline are:

1. Pushing the data pipeline to the environment of production.
2. Exploration of data.
3. Pushing the data pipeline to the staging environment.
4. Developing the unit tests iteratively.
5. Constant build and Integration.

30. How to set up a dev environment in databricks?

The five steps to set up a dev environment in databricks are:

1. Generate a branch and checkout that code to the PC.
2. By using CLI, copy the local directory notebooks to the databricks.
3. Using DBFS CLI, copy local directory libraries to DBFS.
4. By using UI or API, generate a cluster.
5. Finally, using API libraries, connect the libraries within DBFS.

31. What is the use of %run?

The %run command is used to parameterize a databricks notebook. %run is also used to modularize the code.

32. What is the use of widgets in databricks?

Widgets enable us to add parameters to our dashboards and notebooks. The API widget consists of calls to generate multiple input widgets, get the bound values and remove them.

33. What is a secret in databricks?

A secret is a key-value pair that stocks up the secret material; it consists of a unique key name inside a secret scope. The limit of each scope is up to 1000 secrets. The maximum size of the secret value is 128 KB.

34. Write a syntax to list secrets in a specific scope?

The syntax to list secrets in a specific scope is:

databricks secrets list –scope

35. What is the use of Secrets utility?

Secrets utility is used to read the secrets in the job or notebooks.

36. How to delete a Secret?

We can use Azure Portal UI or Azure SetSecret Rest API to delete a Secret from any scope that is backed by an Azure key vault.

37. What are the two types of secret scopes?

There are two types of secret scopes they are:

1. Databricks-backed scopes.
2. Azure key Vault-backed scopes.

38. What are the rules to name a secret scope?

There are three main rules to name a secret scope they are:

1.  A secret scope name must contain underscores, periods, alphanumeric characters, and dashes.
2. The name must not exceed 128 characters.
3. The name must be unique in the workspace.

39. Write the syntax to delete the IP access list?

The syntax to delete the IP access list is:

DELETE /ip-access-lists/

40. What are the different elements to specify within the JSON request body while replacing an IP access list?

The different elements to specify within JSON request body while replacing an IP access list are:

• label
• list_type
• ip_addresses
• enabled

41. Write the syntax to replace an IP access list?

The syntax to replace an IP access list is:

PUT /ip-access-lists/

42. Write the syntax to update an IP access list?

The syntax to update an IP access list is:

PATCH /ip-access-lists/

43. What do clusters do at the network level?

At the network, level clusters try to connect with the control panel proxy throughout the cluster reaction.

44. What are the things involved when pushing the data pipeline to a staging environment?

The four things involved when pushing the data pipeline to a staging environment are:

1. Notebooks
2. Libraries
3. Clusters and Jobs configuration
4. Results

45. List the stages of a CI/CD pipeline?

There are four stages of a CI/CD pipeline they are:

1. Source
2. Build
3. Staging
4. Production