Custom Function that performs lookup

Follow

Guhan Kashyap

• November 13, 2015 18:48

Can I write a custom function that takes in some columns as parameters and then does an hbase lookup to translate this to a different value?

0

• 

  Gido

  • November 16, 2015 09:59

  In general, this is possible. With the SimpleFunctionType you will be able to take a number of arguments and return a result value.

  https://documentation.datameer.com/documentation/current/Tutorial02+-+Building+Custom+Functions

  Within the function you need to cover the part of connecting to a remote HBase in Java then.

  0

  Comment actions Permalink
• 

  Guhan Kashyap

  • November 16, 2015 16:15

  Thanks Gido. I'm still confused about what the runtime support is. Can I connect to any HBase, HDFS source? Can I connect to other sources like Vertica? Are there limitations to what servers I can connect to or can I pretty much use any Java functionality?

  Also - are there other restrictions on the functions such as side effect restrictions?

  0

  Comment actions Permalink
• 

  Gido

  • November 17, 2015 10:54

  In respect to your most recent question, if you could use pretty much any Java functionality, the answer is yes.

  About your your question what the runtime support is, I am not sure. Can you provide more context regarding this?

  0

  Comment actions Permalink
• 

  Guhan Kashyap

  • November 17, 2015 15:10

  In terms of context - I was thinking of something like using Spark SQL in the custom functions - can that be done?

  0

  Comment actions Permalink
• 

  Joel Stewart

  • November 17, 2015 18:35

  Guhan - I'd be cautious about implementing a custom function that requires feedback from an outside system. This may cause a lot of lag when using the function since the result must be computed and displayed on the sample data. This may provide a poor user experience.

  0

  Comment actions Permalink

Please sign in to leave a comment.