Import Job - JSON files under folder with 2 schemas
Hello,
We are importing web transaction logs into Datameer where the schema changes for certain types of event logs, i.e. events that were blocked with a web application firewall, where there are addition columns within the JSON file under a certain folder.
When we import the entire folder it is noticed that those events logs where there was suppose to have additional colunms were not captured (imported).
Please may I have your advice on how to solve this?
Much appreciated.
Anson
-
Hello Anson.
Datameer could handle one schema for an imported dataset. This schema is being detected at artifact's creation and could be reevaluated later, but still, only one version is valid. Each record/file that doesn't comply with the schema will be dropped from import.To be able to suggest a solution, I would need to know how the source data is organized. Whether the records with different schemas are always located in separate files or there might be a single file that has records with different schemas inside? What is the format of the source data, is it JSON?
Please sign in to leave a comment.
Comments
1 comment