How to Use Datameer Smart Sampling

Goal

Datameer offers many features that enable users to work with large data sets. One of those features is Smart Sampling. Smart Sampling reliably provides a view on your data that enables you to build an analytic model with real-time feedback.

Learn

For optimized performance and an accurate representation of your data, Datameer provides a default initial sample of 5000 records (which can be customized) when you start your analysis. This is a representative sample generated via the distributed reservoir sampling technique:

    * Each Task generates a ((k / n-tasks) * oversample- factor) sized sample

    * At the end of the job the samples are merged to generate a k sized final sample

During design of the analysis, using the initial representative sample, it may happen that you end up with an empty sheet (e.g., due to usage of filters.) Even though the full data set may contain records that would pass the filter. Smart sampling overcomes this problem by resampling the data, taking the designed operations up to that point into account. Simply run your partially designed workbook and the new smart sample will populate the sheet with records.

Unbiased Randomized Samples

Unbiased randomized samples do have limitations:

• Increasing the size can reduce some of them
• Outliers could be missed or included affecting the preview calculations
• Join and Filter can result in empty sheets when trying to match against an underrepresented subset of the records

In order to deal with the empty sheet problem, we bias the samples.

Filtered and Joined Sheets

For filtered sheets, we generate a biased sample based on which records pass through the filter by assigning a larger weight to records that pass.

• This requires computing the formulas and filter twice
• CPU time isn’t usually the bottleneck

For Join sheets we generate a biased sample by evenly distributing the join keys

• Lowers the chance of sampling a record if we have seen its join key too many times

Optimized Previews

The optimized previews (smart samples) can only be generated when the Workbook is being run, and are generated for the data source sheets and kept sheets. Optimized Preview generation can be disabled by setting the property: “das.sampling.lookahead.maxdepth” to 0

• This can be useful for critical, time constrained Workbooks

Datalinks

The initial sample for Data Links aren’t representative. Currently, the sample is just the first k records of the data source. Once the Data Link is used in a Workbook though, an Optimized Preview will be generated for it.