A data engineer manages a production Lakeflow Declarative Pipeline that processes customer transaction data. The pipeline includes several data quality expectations such as transaction_amount > 0 and customer_id IS NOT NULL. These expectations are defined using the EXPECT clause in SQL.
The engineer aims to monitor the pipeline's data quality by analyzing the number of records that passed or failed each expectation during the latest pipeline update. The Lakeflow Declarative Pipelines event logs are stored in a Delta table named event_log_table.
For the most recent pipeline update, determine a programmatically appropriate approach to extract information like the name of each expectation, associated dataset, count of records that passed the expectation, and count of records that failed the expectation.
Which method retrieves the desired data quality metrics from the Lakeflow Declarative Pipelines event log?

• A. Access the event_log_table, filter for events where event_type = 'flow_progress', and parse details.flow_progress.data_quality.expectations field to extract the required metrics.
• B. Use the Lakeflow Declarative Pipelines UI to navigate to the specific pipeline, select the dataset, and view the Data Quality tab to manually retrieve the expectation metrics.
• C. Query the event_log_table for events with event_type = 'data_quality' and directly select the passed_records and failed_records fields.
• D. Access the event_log_table, filter for events where event_type = 'expectation_result', and extract the expectation metrics from the details field.

A data engineer is building a customer data pipeline in Lakeflow Spark Declarative Pipelines. The source is a cloud-based event stream with limited retention containing inserts, updates, and deletes for customer records. These changes are being applied using the AUTO CDC INTO syntax to maintain an SCD Type 1 table as the target table, customer_dim. How should the data engineer build a downstream job that streams from the customer_dim table to only act on updates and delete events, processing data incrementally?

• A. Read change data feed from customer_dim table and apply filters to incrementally act on the change events.
• B. When stored as SCD 1, the target of AUTO CDC INTO includes updates and deletes. Streaming from customer_dim can fail due to these operations. Instead, build another stream from the original source.
• C. Use ignoreChanges flag while streaming from customer_dim to avoid breaking the pipeline during updates and deletes.
• D. Streaming from customer_dim table would only be possible in the case of SCD 2 retention.

A junior data engineer seeks to leverage Delta Lake's Change Data Feed functionality to create a Type 1 table representing all of the values that have ever been valid for all rows in a bronze table created with the property delta.enableChangeDataFeed = true. They plan to execute the following code as a daily job:

Which statement describes the execution and results of running the above query multiple times?

• A. Each time the job is executed, the target table will be overwritten using the entire history of inserted or updated records, giving the desired result.
• B. Each time the job is executed, the differences between the original and current versions are calculated; this may result in duplicate entries for some records.
• C. Each time the job is executed, the entire available history of inserted or updated records will be appended to the target table, resulting in many duplicate entries.
• D. Each time the job is executed, only those records that have been inserted or updated since the last execution will be appended to the target table giving the desired result.
• E. Each time the job is executed, newly updated records will be merged into the target table, overwriting previous values with the same primary keys.

A data engineer is working in an interactive notebook with many transformations before outputting the result from display(df.collect() ). The notebook includes wide transformations and a cross join.
The data engineer is getting the following error: "The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached." Which action should the data engineer take?

• A. Check into the Spark UI to see how many jobs are assigned to each stage as they are employing fewer executors.
• B. Run the notebook on a single node cluster to keep driver from falling.
• C. Look at the compute metrics UI to see if the executors have higher than 90% memory utilization.
• D. Rewrite their code to avoid putting memory pressure on the driver node.

A Delta Lake table was created with the below query:

Realizing that the original query had a typographical error, the below code was executed:
ALTER TABLE prod.sales_by_stor RENAME TO prod.sales_by_store
Which result will occur after running the second command?

• A. The table reference in the metastore is updated and no data is changed.
• B. All related files and metadata are dropped and recreated in a single ACID transaction.
• C. A new Delta transaction log Is created for the renamed table.
• D. The table name change is recorded in the Delta transaction log.
• E. The table reference in the metastore is updated and all data files are moved.