Certified Data Engineer Professional Exam Questions

dbutils.widgets is the method used in Databricks notebooks to define and access parameters passed to the notebook.

dbutils.widgets.text creates a widget for the parameter (in this case, date) with a default value of "null". This widget allows the notebook to accept and handle the date parameter dynamically.

dbutils.widgets.get("date") retrieves the value of the date parameter passed when invoking the notebook, either through the Databricks Jobs API or manually via the UI.

In Databricks, "Can Restart" privileges allow a user to:

1. Start or restart a cluster that is in a terminated state.

2. Attach their workloads (notebooks, jobs) to the cluster and execute code.

This privilege is sufficient for users to work with pre-configured clusters without needing broader access, such as cluster management or creation capabilities.

New Job Cluster: This ensures that each job runs on a separate cluster, providing isolation and scalability for each job execution, which is ideal for production workloads.

Retries: Unlimited: With unlimited retries, the job can automatically recover from query failures, ensuring that the job keeps running until it succeeds.

Maximum Concurrent Runs: 1: Limiting the concurrent runs to 1 ensures that there is only one active run of the job at a time, which helps to avoid overlapping runs and keeps costs under control by preventing unnecessary resource consumption.

This is the correct answer because the query is using a GROUP BY clause on the sensor_id column, which means it will calculate the mean temperature for each sensor separately. The alert will trigger when the mean temperature for any sensor is greater than 120, which means at least one sensor had an average temperature above 120 for three consecutive minutes. The alert will stop when the mean temperature for all sensors drops below 120. Verified References: [Databricks Certified Data Engineer Professional], under “SQL Analytics” section; Databricks Documentation, under “Alerts” section.

In Databricks Repos, you are typically working with a Git repository where different branches contain different versions of your code. If the desired branch (dev-2.3.9) is not visible in the dropdown, it's possible that the branch hasn't been fetched to your local environment.

By pulling changes from the remote Git repository (using Repos), you will retrieve the latest branches and updates from the remote, which should include the dev-2.3.9 branch. Once the branch is available locally, you can then select it and access the correct version of the logic.

This is the simplest and most direct approach for resolving the issue, without the need for additional steps like merging or cloning the repository.

1. Connection Success: The code uses spark.read.format("jdbc") with the correct options including URL, dbtable, and password. The password is correctly passed (though hidden), so the connection will succeed.

2. Password Handling:

   • When you print the jdbc_options map, Databricks will automatically redact sensitive information like passwords for security reasons. This is why you'll see "REDACTED" instead of the actual password.

   • This is a security feature to prevent accidental exposure of credentials in logs or notebook outputs

1. .write.mode("append").saveAsTable(...): Appends each day's predictions to a managed Delta Lake table.

2. Supports historical tracking of predictions without overwriting previous records.

3. Suitable when predictions are made once per day and don't need real-time streaming.

4. Compute-efficient since it's a batch operation, not streaming.

5. Easy to query with time filters or time-series analysis.

1. The code applies .dropDuplicates(["customer_id", "order_id"]) only to the batch being loaded that day from the Parquet files.

2. The .write.mode("append") blindly appends those deduplicated records to the orders table.

3. There is no comparison or deduplication against the data already in the table.

4. Therefore, if the same customer_id + order_id combination already exists in the table (e.g., from an earlier batch or previous day), a duplicate will still be added.

• Cmd 1 (Python): This command collects a list of countries from the geo_lookup table where continent = 'AF'. It results in a Python list of strings — e.g., ['Egypt', 'Kenya', 'Nigeria'].

• Cmd 2 (SQL): Tries to use the Python variable countries_af inside a SQL cell, which will fail because SQL cells cannot access Python variables unless explicitly injected (e.g., via temp views or string formatting).

Delta Lake uses a transaction log (often referred to as the Delta log) to track metadata and provide ACID properties for the data. When executing a query with a filter, the Delta engine checks the Delta log, not just the Parquet file footers, for the min and max statistics of the latitude column (and other columns, if applicable).

This allows the engine to efficiently identify which files or partitions (based on the statistics stored in the Delta log) need to be read, rather than scanning all files. This process optimizes performance by eliminating unnecessary file reads, which is part of Delta Lake's predicate pushdown optimization.

So, the correct sequence involves scanning the Delta log to determine the relevant files based on stored statistics, before applying the query filter to the data in the relevant files.