Workbook Fails: <path_to_file> Is Not a Parquet File. Expected Magic Number at Tail

Problem

When you attempt to execute my workbook containing partitioned data, you notice a few select partitions are causing a failure of the job.

Upon closer inspection, you see the following:

Caused by: java.lang.RuntimeException: hdfs://<datameer_private_folder>/importjobs/<artifact_id>/<execution_id>/rewrite/data/<partition_date>/<exported_parquet_file>_0.parquet is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [-76, -93, 1, 0]

If the broken partition is excluded, the workbook will complete successfully. 

Cause

The error can be traced back to a Compaction Job that ran against the source data. If such a job fails, it currently only attempts to clean up the data from the previous attempt. If this fails and a new attempt is started without the clean up, you are left with an additional corrupted file in your output path.

Workaround

If you were to identify the source directories that have been impacted, the corrupted files can be removed to repair the partition.

Two ways to identify the broken file:

• It will contain an "_<attempt_value>" that will be lower than the other file in place.
• The corrupted file will typically be sized smaller than the intact file.

Solution

We have identified a solution and will be releasing updated code in the form of a maintenance patch.

For further inquiries, please reach out to Support and provide "DAP-37174" as a reference.