Output count of Union Sheet changes if some sheets are not kept

Problem

Unkept union sheets in hierarchical unions and map-side-join sheets output partial results. The problem only occurs in a very specific configuration that meets all of the requirements:

• The workbook contains at least 3 union sheets:
  • The results of two union sheets are the sources for the third union sheet
  • The union sheets are unkept
• The workbook contains at least 1 join sheet:
  • The join sheet must process as a map-side-join at runtime
  • The join sheet must process data before at least one of the union sheets

If all the above requirements are met, the problem is as follows: if two union sheets ("UnionA" and "UnionB") feed another union sheet ("UnionResult") downstream in the same workbook and the source union sheets (UnionA or UnionB) are not kept sheets, the final union (UnionResult) results lacks records of 3 of the 4 initial input sources.

Cause

This is identified as a defect and is tracked via an internal ticket with ID: DAP-37664

This issue affects the following versions of Datameer: 

• All versions of Datameer 6.3
• Datameer 6.4.0 through 6.4.12
• Datameer 7.1.0 through 7.1.8
• Datameer 7.2.0 through 7.2.3

Datameer Administrators may verify if any workbooks are affected by using the following SQL command: 

SELECT t.workbook_fk WorkbookID
FROM (
 SELECT
 SUM(sheet_type = "das.internal.UnionSheetType") union_sheets,
 SUM(sheet_type = "das.internal.UnionSheetType" AND keep) kept_union_sheets,
 SUM(sheet_type = "das.internal.JoinedSheetType") join_sheets,
 workbook_fk
 FROM sheet
 GROUP BY workbook_fk) t
WHERE t.union_sheets > 2 AND t.union_sheets != t.kept_union_sheets AND t.join_sheets > 0;

 If the output of this query is 0 rows, then no workbooks are affected. If the output of this query contains workbook IDs, they may be affected and need attention in affected versions of Datameer.

Solution

To resolve this issue, install the latest maintenance patch for Datameer. Specifically, the fix is incorporated into Datameer releases 6.4.13, 7.1.9 and 7.2.4 and later. 

As a temporary workaround:

• In affected workbooks, configure all union sheets to be kept. 

Please keep in mind that this work-around may impact execution time of Workbooks that contain newly saved sheets. Therefore it is recommended to schedule an update to the latest maintenance patch as soon as possible. 

Please contact Datameer Technical Support for further information.