Prepare for your exam certification with our Databricks-Certified-Professional-Data-Engineer Certified Databricks
Free Databricks Databricks-Certified-Professional-Data-Engineer Exam 2025 Practice Materials Collection
The Databricks Certified Professional Data Engineer Exam certification exam covers a range of topics, including data ingestion, transformation, and storage, ETL processes, data modeling, and machine learning. Candidates are tested on their ability to use Databricks tools and technologies to solve real-world data engineering problems. Databricks-Certified-Professional-Data-Engineer exam also evaluates the candidate's understanding of best practices for data engineering, including security, scalability, and cost optimization. By passing the Databricks Certified Professional Data Engineer exam, candidates can demonstrate their proficiency in Databricks data engineering technologies and enhance their job prospects in the field.
NEW QUESTION # 19
Which of the following describes a scenario in which a data engineer will want to use a Job cluster instead of
an all-purpose cluster?
- A. A data engineer needs to manually investigate a production error
- B. An automated workflow needs to be run every 30 minutes
- C. A data team needs to collaborate on the development of a machine learning model
- D. A Databricks SQL query needs to be scheduled for upward reporting
- E. An ad-hoc analytics report needs to be developed while minimizing compute costs
Answer: B
NEW QUESTION # 20
Review the following error traceback:
Which statement describes the error being raised?
- A. There is no column in the table named heartrateheartrateheartrate
- B. There is a syntax error because the heartrate column is not correctly identified as a column.
- C. The code executed was PvSoark but was executed in a Scala notebook.
- D. There is a type error because a column object cannot be multiplied.
- E. There is a type error because a DataFrame object cannot be multiplied.
Answer: A
Explanation:
The error being raised is an AnalysisException, which is a type of exception that occurs when Spark SQL cannot analyze or execute a query due to some logical or semantic error1. In this case, the error message indicates that the query cannot resolve the column name 'heartrateheartrateheartrate' given the input columns 'heartrate' and 'age'. This means that there is no column in the table named 'heartrateheartrateheartrate', and the query is invalid. A possible cause of this error is a typo or a copy-paste mistake in the query. To fix this error, the query should use a valid column name that exists in the table, such as 'heartrate'. Reference: AnalysisException
NEW QUESTION # 21
A data engineer, User A, has promoted a new pipeline to production by using the REST API to programmatically create several jobs. A DevOps engineer, User B, has configured an external orchestration tool to trigger job runs through the REST API. Both users authorized the REST API calls using their personal access tokens.
Which statement describes the contents of the workspace audit logs concerning these events?
- A. Because these events are managed separately, User A will have their identity associated with the job creation events and User B will have their identity associated with the job run events.
- B. Because the REST API was used for job creation and triggering runs, user identity will not be captured in the audit logs.
- C. Because User B last configured the jobs, their identity will be associated with both the job creation events and the job run events.
- D. Because the REST API was used for job creation and triggering runs, a Service Principal will be automatically used to identity these events.
- E. Because User A created the jobs, their identity will be associated with both the job creation events and the job run events.
Answer: A
Explanation:
The events are that a data engineer, User A, has promoted a new pipeline to production by using the REST API to programmatically create several jobs, and a DevOps engineer, User B, has configured an external orchestration tool to trigger job runs through the REST API. Both users authorized the REST API calls using their personal access tokens. The workspace audit logs are logs that record user activities in a Databricks workspace, such as creating, updating, or deleting objects like clusters, jobs, notebooks, or tables. The workspace audit logs also capture the identity of the user who performed each activity, as well as the time and details of the activity. Because these events are managed separately, User A will have their identity associated with the job creation events and User B will have their identity associated with the job run events in the workspace audit logs. Verified Reference: [Databricks Certified Data Engineer Professional], under "Databricks Workspace" section; Databricks Documentation, under "Workspace audit logs" section.
NEW QUESTION # 22
Which statement describes the correct use of pyspark.sql.functions.broadcast?
- A. It marks a DataFrame as small enough to store in memory on all executors, allowing a broadcast join.
- B. It marks a column as small enough to store in memory on all executors, allowing a broadcast join.
- C. It marks a column as having low enough cardinality to properly map distinct values to available partitions, allowing a broadcast join.
- D. It caches a copy of the indicated table on attached storage volumes for all active clusters within a Databricks workspace.
- E. It caches a copy of the indicated table on all nodes in the cluster for use in all future queries during the cluster lifetime.
Answer: A
Explanation:
https://spark.apache.org/docs/3.1.3/api/python/reference/api/pyspark.sql.functions.broadcast.html
NEW QUESTION # 23
Each configuration below is identical to the extent that each cluster has 400 GB total of RAM, 160 total cores and only one Executor per VM.
Given a job with at least one wide transformation, which of the following cluster configurations will result in maximum performance?
- A. * Total VMs: 8
* 50 GB per Executor
* 20 Cores / Executor - B. * Total VMs; 1
* 400 GB per Executor
* 160 Cores / Executor - C. * Total VMs: 4
* 100 GB per Executor
* 40 Cores/Executor - D. * Total VMs:2
* 200 GB per Executor
* 80 Cores / Executor
Answer: A
Explanation:
This is the correct answer because it is the cluster configuration that will result in maximum performance for a job with at least one wide transformation. A wide transformation is a type of transformation that requires shuffling data across partitions, such as join, groupBy, or orderBy. Shuffling can be expensive and time- consuming, especially if there are too many or too few partitions. Therefore, it is important to choose a cluster configuration that can balance thetrade-off between parallelism and network overhead. In this case, having 8 VMs with 50 GB per executor and 20 cores per executor will create 8 partitions, each with enough memory and CPU resources to handle the shuffling efficiently. Having fewer VMs with more memory and cores per executor will create fewer partitions, which will reduce parallelism and increase the size of each shuffle block.
Having more VMs with less memory and cores per executor will create more partitions, which will increase parallelism but also increase the network overhead and the number of shuffle files. Verified References:
[Databricks Certified Data Engineer Professional], under "Performance Tuning" section; Databricks Documentation, under "Cluster configurations" section.
NEW QUESTION # 24
A data analyst has noticed that their Databricks SQL queries are running too slowly. They claim that this issue
is affecting all of their sequentially run queries. They ask the data engineering team for help. The data
engineering team notices that each of the queries uses the same SQL endpoint, but the SQL endpoint is not
used by any other user.
Which of the following approaches can the data engineering team use to improve the latency of the data
analyst's queries?
- A. They can increase the cluster size of the SQL endpoint
- B. They can turn on the Serverless feature for the SQL endpoint and change the Spot In-stance Policy to
"Reliability Optimized" - C. They can turn on the Serverless feature for the SQL endpoint
- D. They can increase the maximum bound of the SQL endpoint's scaling range
- E. They can turn on the Auto Stop feature for the SQL endpoint
Answer: A
NEW QUESTION # 25
Which of the following python statement can be used to replace the schema name and table name in the query statement?
- A. 1.table_name = "sales"
2.schema_name = "bronze"
3.query = f"select * from schema_name.table_name" - B. 1.table_name = "sales"
2.schema_name = "bronze"
3.query = f"select * from { schema_name}.{table_name}" - C. 1.table_name = "sales"
2.schema_name = "bronze"
3.query = f"select * from + schema_name +"."+table_name" - D. 1.table_name = "sales"
2.schema_name = "bronze"
3.query = "select * from {schema_name}.{table_name}"
Answer: B
Explanation:
Explanation
Answer is
table_name = "sales"
query = f"select * from {schema_name}.{table_name}"
f strings can be used to format a string. f" This is string {python variable}"
https://realpython.com/python-f-strings/
NEW QUESTION # 26
The marketing team is looking to share data in an aggregate table with the sales organization, but the field names used by the teams do not match, and a number of marketing specific fields have not been approval for the sales org.
Which of the following solutions addresses the situation while emphasizing simplicity?
- A. Use a CTAS statement to create a derivative table from the marketing table configure a production jon to propagation changes.
- B. Create a new table with the required schema and use Delta Lake's DEEP CLONE functionality to sync up changes committed to one table to the corresponding table.
- C. Add a parallel table write to the current production pipeline, updating a new sales table that varies as required from marketing table.
- D. Create a view on the marketing table selecting only these fields approved for the sales team alias the names of any fields that should be standardized to the sales naming conventions.
Answer: D
Explanation:
Creating a view is a straightforward solution that can address the need for field name standardization and selective field sharing between departments. A view allows for presenting a transformed version of the underlying data without duplicating it. In this scenario, the view would only include the approved fields for the sales team and rename any fields as per their naming conventions.
Reference:
Databricks documentation on using SQL views in Delta Lake: https://docs.databricks.com/delta/quick-start.html#sql-views
NEW QUESTION # 27
A Spark job is taking longer than expected. Using the Spark UI, a data engineer notes that the Min, Median, and Max Durations for tasks in a particular stage show the minimum and median time to complete a task as roughly the same, but the max duration for a task to be roughly 100 times as long as the minimum.
Which situation is causing increased duration of the overall job?
- A. Task queueing resulting from improper thread pool assignment.
- B. Spill resulting from attached volume storage being too small.
- C. Skew caused by more data being assigned to a subset of spark-partitions.
- D. Credential validation errors while pulling data from an external system.
- E. Network latency due to some cluster nodes being in different regions from the source data
Answer: C
Explanation:
Explanation
This is the correct answer because skew is a common situation that causes increased duration of the overall job. Skew occurs when some partitions have more data than others, resulting in uneven distribution of work among tasks and executors. Skew can be caused by various factors, such as skewed data distribution, improper partitioning strategy, or join operations with skewed keys. Skew can lead to performance issues such as long-running tasks, wasted resources, or even task failures due to memory or disk spills. Verified References:
[Databricks Certified Data Engineer Professional], under "Performance Tuning" section; Databricks Documentation, under "Skew" section.
NEW QUESTION # 28
Which of the following statements are true about a lakehouse?
- A. Lakehouse only supports end-to-end streaming workloads and Data warehouses support Batch workloads
- B. Lakehouse do not support SQL
- C. Lakehouse supports Transactions
- D. Lakehouse does not support ACID
- E. Lakehouse only supports Machine learning workloads and Data warehouses support BI workloads
Answer: C
Explanation:
Explanation
What Is a Lakehouse? - The Databricks Blog
Text Description automatically generated
NEW QUESTION # 29
A data engineer is configuring a pipeline that will potentially see late-arriving, duplicate records.
In addition to de-duplicating records within the batch, which of the following approaches allows the data engineer to deduplicate data against previously processed records as it is inserted into a Delta table?
- A. Set the configuration delta.deduplicate = true.
- B. Rely on Delta Lake schema enforcement to prevent duplicate records.
- C. VACUUM the Delta table after each batch completes.
- D. Perform a full outer join on a unique key and overwrite existing data.
- E. Perform an insert-only merge with a matching condition on a unique key.
Answer: E
Explanation:
To deduplicate data against previously processed records as it is inserted into a Delta table, you can use the merge operation with an insert-only clause. This allows you to insert new records that do not match any existing records based on a unique key, while ignoring duplicate records that match existing records. For example, you can use the following syntax:
MERGE INTO target_table USING source_table ON target_table.unique_key = source_table.unique_key WHEN NOT MATCHED THEN INSERT * This will insert only the records from the source table that have a unique key that is not present in the target table, and skip the records that have a matching key. This way, you can avoid inserting duplicate records into the Delta table.
References:
* https://docs.databricks.com/delta/delta-update.html#upsert-into-a-table-using-merge
* https://docs.databricks.com/delta/delta-update.html#insert-only-merge
NEW QUESTION # 30
A junior data engineer has been asked to develop a streaming data pipeline with a grouped aggregation using DataFrame df. The pipeline needs to calculate the average humidity and average temperature for each non-overlapping five-minute interval. Incremental state information should be maintained for 10 minutes for late-arriving data.
Streaming DataFrame df has the following schema:
"device_id INT, event_time TIMESTAMP, temp FLOAT, humidity FLOAT"
Code block:
Choose the response that correctly fills in the blank within the code block to complete this task.
- A. slidingWindow("event_time", "10 minutes")
- B. awaitArrival("event_time", "10 minutes")
- C. delayWrite("event_time", "10 minutes")
- D. await("event_time + '10 minutes'")
- E. withWatermark("event_time", "10 minutes")
Answer: E
Explanation:
The correct answer is A. withWatermark("event_time", "10 minutes"). This is because the question asks for incremental state information to be maintained for 10 minutes for late-arriving data. The withWatermark method is used to define the watermark for late data. The watermark is a timestamp column and a threshold that tells the system how long to wait for late data. In this case, the watermark is set to 10 minutes. The other options are incorrect because they are not valid methods or syntax for watermarking in Structured Streaming. References:
* Watermarking: https://docs.databricks.com/spark/latest/structured-streaming/watermarks.html
* Windowed aggregations:
https://docs.databricks.com/spark/latest/structured-streaming/window-operations.html
NEW QUESTION # 31
A data engineer needs to dynamically create a table name string using three Python varia-bles: region, store,
and year. An example of a table name is below when region = "nyc", store = "100", and year = "2021":
nyc100_sales_2021
Which of the following commands should the data engineer use to construct the table name in Py-thon?
- A. f"{region}{store}_sales_2025"
- B. f"{region}+{store}+_sales_+2025"
- C. "{region}{store}_sales_2025"
- D. "{region}+{store}+_sales_+2025"
- E. "{region}+{store}+"_sales_"+2025"
Answer: A
NEW QUESTION # 32
You are currently working on reloading customer_sales tables using the below query
1. INSERT OVERWRITE customer_sales
2. SELECT * FROM customers c
3. INNER JOIN sales_monthly s on s.customer_id = c.customer_id
After you ran the above command, the Marketing team quickly wanted to review the old data that was in the table. How does INSERT OVERWRITE impact the data in the customer_sales table if you want to see the previous version of the data prior to running the above statement?
- A. Appends the data to the current version, you can time travel to previous versions
- B. Overwrites the data in the table, all historical versions of the data, you can not time travel to previous versions
- C. Overwrites the current version of the data but clears all historical versions of the data, so you can not time travel to previous versions.
- D. Overwrites the data in the table but preserves all historical versions of the data, you can time travel to previous versions
- E. By default, overwrites the data and schema, you cannot perform time travel
Answer: D
Explanation:
Explanation
The answer is, INSERT OVERWRITE Overwrites the current version of the data but preserves all historical versions of the data, you can time travel to previous versions.
1.INSERT OVERWRITE customer_sales
2.SELECT * FROM customers c
3.INNER JOIN sales s on s.customer_id = c.customer_id
Let's just assume that this is the second time you are running the above statement, you can still query the prior version of the data using time travel, and any DML/DDL except DROP TABLE creates new PARQUET files so you can still access the previous versions of data.
SQL Syntax for Time travel
SELECT * FROM table_name as of [version number]
with customer_sales example
SELECT * FROM customer_sales as of 1 -- previous version
SELECT * FROM customer_sales as of 2 -- current version
You see all historical changes on the table using DESCRIBE HISTORY table_name Note: the main difference between INSERT OVERWRITE and CREATE OR REPLACE TABLE(CRAS) is that CRAS can modify the schema of the table, i.e it can add new columns or change data types of existing columns. By default INSERT OVERWRITE only overwrites the data.
INSERT OVERWRITE can also be used to update the schema when
spark.databricks.delta.schema.autoMerge.enabled is set true if this option is not enabled and if there is a schema mismatch command INSERT OVERWRITEwill fail.
Any DML/DDL operation(except DROP TABLE) on the Delta table preserves the historical ver-sion of the data.
NEW QUESTION # 33
You are asked to setup an AUTO LOADER to process the incoming data, this data arrives in JSON format and get dropped into cloud object storage and you are required to process the data as soon as it arrives in cloud storage, which of the following statements is correct
- A. AUTO LOADER needs to be converted to a Structured stream process
- B. AUTO LOADER has to be triggered from an external process when the file arrives in the cloud storage
- C. AUTO LOADER can support file notification method so it can process data as it arrives
- D. AUTO LOADER is native to DELTA lake it cannot support external cloud object storage
- E. AUTO LOADER can only process continuous data when stored in DELTA lake
Answer: C
Explanation:
Explanation
Auto Loader supports two modes when ingesting new files from cloud object storage Directory listing: Auto Loader identifies new files by listing the input directory, and uses a directory polling approach.
File notification: Auto Loader can automatically set up a notification service and queue service that subscribe to file events from the input directory.
Diagram Description automatically generated
File notification is more efficient and can be used to process the data in real-time as data arrives in cloud object storage.
Choosing between file notification and directory listing modes | Databricks on AWS
NEW QUESTION # 34
A data engineer, User A, has promoted a new pipeline to production by using the REST API to programmatically create several jobs. A DevOps engineer, User B, has configured an external orchestration tool to trigger job runs through the REST API. Both users authorized the REST API calls using their personal access tokens.
Which statement describes the contents of the workspace audit logs concerning these events?
- A. Because these events are managed separately, User A will have their identity associated with the job creation events and User B will have their identity associated with the job run events.
- B. Because the REST API was used for job creation and triggering runs, user identity will not be captured in the audit logs.
- C. Because User B last configured the jobs, their identity will be associated with both the job creation events and the job run events.
- D. Because the REST API was used for job creation and triggering runs, a Service Principal will be automatically used to identity these events.
- E. Because User A created the jobs, their identity will be associated with both the job creation events and the job run events.
Answer: A
Explanation:
The events are that a data engineer, User A, has promoted a new pipeline to production by using the REST API to programmatically create several jobs, and a DevOps engineer, User B, has configured an external orchestration tool to trigger job runs through the REST API. Both users authorized the REST API calls using their personal access tokens. The workspace audit logs are logs that record user activities in a Databricks workspace, such as creating, updating, or deleting objects like clusters, jobs, notebooks, or tables. The workspace audit logs also capture the identity of the user who performed each activity, as well as the time and details of the activity. Because these events are managed separately, User A will have their identity associated with the job creation events and User B will have their identity associated with the job run events in the workspace audit logs. Verified References: [Databricks Certified Data Engineer Professional], under
"Databricks Workspace" section; Databricks Documentation, under "Workspace audit logs" section.
NEW QUESTION # 35
You are trying to create an object by joining two tables that and it is accessible to data scientist's team, so it does not get dropped if the cluster restarts or if the notebook is detached. What type of object are you trying to create?
- A. External view
- B. View
- C. Temporary view
- D. Global Temporary view with cache option
- E. Global Temporary view
Answer: B
Explanation:
Explanation
Answer is View, A view can be used to join multiple tables but also persist into meta stores so others can accesses it
NEW QUESTION # 36
The data architect has decided that once data has been ingested from external sources into the Databricks Lakehouse, table access controls will be leveraged to manage permissions for all production tables and views.
The following logic was executed to grant privileges for interactive queries on a production database to the core engineering group.
GRANT USAGE ON DATABASE prod TO eng;
GRANT SELECT ON DATABASE prod TO eng;
Assuming these are the only privileges that have been granted to the eng group and that these users are not workspace administrators, which statement describes their privileges?
- A. Group members have full permissions on the prod database and can also assign permissions to other users or groups.
- B. Group members are able to list all tables in the prod database but are not able to see the results of any queries on those tables.
- C. Group members are able to query all tables and views in the prod database, but cannot create or edit anything in the database.
- D. Group members are able to create, query, and modify all tables and views in the prod database, but cannot define custom functions.
- E. Group members are able to query and modify all tables and views in the prod database, but cannot create new tables or views.
Answer: C
Explanation:
Explanation
The GRANT USAGE ON DATABASE prod TO eng command grants the eng group the permission to use the prod database, which means they can list and access the tables and views in the database. The GRANT SELECT ON DATABASE prod TO eng command grants the eng group the permission to select data from the tables and views in the prod database, which means they can query the data using SQL or DataFrame API.
However, these commands do not grant the eng group any other permissions, such as creating, modifying, or deleting tables and views, or defining custom functions. Therefore, the eng group members are able to query all tables and views in the prod database, but cannot create or edit anything in the database. References:
Grant privileges on a database:
https://docs.databricks.com/en/security/auth-authz/table-acls/grant-privileges-database.html Privileges you can grant on Hive metastore objects:
https://docs.databricks.com/en/security/auth-authz/table-acls/privileges.html
NEW QUESTION # 37
A junior member of the data engineering team is exploring the language interoperability of Databricks notebooks. The intended outcome of the below code is to register a view of all sales that occurred in countries on the continent of Africa that appear in thegeo_lookuptable.
Before executing the code, runningSHOWTABLESon the current database indicates the database contains only two tables:geo_lookupandsales.
Which statement correctly describes the outcome of executing these command cells in order in an interactive notebook?
- A. Both commands will fail. No new variables, tables, or views will be created.
- B. Cmd 1 will succeed. Cmd 2 will search all accessible databases for a table or view named countries af: if this entity exists, Cmd 2 will succeed.
- C. Both commands will succeed. Executing show tables will show that countries at and sales at have been registered as views.
- D. Cmd 1 will succeed and Cmd 2 will fail, countries at will be a Python variable representing a PySpark DataFrame.
- E. Cmd 1 will succeed and Cmd 2 will fail, countries at will be a Python variable containing a list of strings.
Answer: E
Explanation:
This is the correct answer because Cmd 1 is written in Python and uses a list comprehension to extract the country names from the geo_lookup table and store them in a Python variable named countries af. This variable will contain a list of strings, not a PySpark DataFrame or a SQL view. Cmd 2 is written in SQL and tries to create a view named sales af by selecting from the sales table where city is in countries af. However, this command will fail because countries af is not a valid SQL entity and cannot be used in a SQL query. To fix this, a better approach would be to use spark.sql() to execute a SQL query in Python and pass the countries af variable as a parameter. Verified References: [Databricks Certified Data Engineer Professional], under
"Language Interoperability" section; Databricks Documentation, under "Mix languages" section.
NEW QUESTION # 38
A table in the Lakehouse namedcustomer_churn_paramsis used in churn prediction by the machine learning team. The table contains information about customers derived from a number of upstream sources. Currently, the data engineering team populates this table nightly by overwriting the table with the current valid values derived from upstream data sources.
The churn prediction model used by the ML team is fairly stable in production. The team is only interested in making predictions on records that have changed in the past 24 hours.
Which approach would simplify the identification of these changed records?
- A. Apply the churn model to all rows in the customer_churn_params table, but implement logic to perform an upsert into the predictions table that ignores rows where predictions have not changed.
- B. Modify the overwrite logic to include a field populated by calling spark.sql.functions.
current_timestamp() as data are being written; use this field to identify records written on a particular date. - C. Convert the batch job to a Structured Streaming job using the complete output mode; configure a Structured Streaming job to read from the customer_churn_params table and incrementally predict against the churn model.
- D. Calculate the difference between the previous model predictions and the current customer_churn_params on a key identifying unique customers before making new predictions; only make predictions on those customers not in the previous predictions.
- E. Replace the current overwrite logic with a merge statement to modify only those records that have changed; write logic to make predictions on the changed records identified by the change data feed.
Answer: E
Explanation:
The approach that would simplify the identification of the changed records is to replace the current overwrite logic with a merge statement to modify only those records that have changed, and write logic to make predictions on the changed records identified by the change data feed. This approach leverages the Delta Lake features of merge and change data feed, which are designed to handle upserts and track row-level changes in a Delta table12. By using merge, the data engineering team can avoid overwriting the entire table every night, and only update or insert the records that have changed in the source data. By using change data feed, the ML team can easily access the change events that have occurred in the customer_churn_params table, and filter them by operation type (update or insert) and timestamp. This way, they can only make predictions on the records that have changed in the past 24 hours, and avoid re-processing the unchanged records.
The other options are not as simple or efficient as the proposed approach, because:
* Option A would require applying the churn model to all rows in the customer_churn_params table, which would be wasteful and redundant. It would also require implementing logic to perform an upsert into the predictions table, which would be more complex than using the merge statement.
* Option B would require converting the batch job to a Structured Streaming job, which would involve changing the data ingestion and processing logic. It would also require using the complete output mode, which would output the entire result table every time there is a change in the source data, which would be inefficient and costly.
* Option C would require calculating the difference between the previous model predictions and the current customer_churn_params on a key identifying unique customers, which would be computationally expensive and prone to errors. It would also require storing and accessing the previous predictions, which would add extra storage and I/O costs.
* Option D would require modifying the overwrite logic to include a field populated by calling spark.sql.
functions.current_timestamp() as data are being written, which would add extra complexity and overhead to the data engineering job. It would also require using this field to identify records written on a particular date, which would be less accurate and reliable than using the change data feed.
References: Merge, Change data feed
NEW QUESTION # 39
......
Pass Databricks Databricks-Certified-Professional-Data-Engineer Actual Free Exam Q&As Updated Dump: https://www.testsdumps.com/Databricks-Certified-Professional-Data-Engineer_real-exam-dumps.html
