datameer commands & hive queries

Follow

Lacramioara Astefanoaei

• August 28, 2017 09:14

Hi,

i'm just wondering if there is a way to translate datameer commands (applied to worksheets) into hive queries. For instance, say I have sheet0 with column A and sheet1 with column B which is obtained by groupBy(#data!A). In hive, this would be

==

%hive

select * from datasource
group by A

==

no? Is there a way to automatically generate such hive queries for the formulas in a worksheet? At least for a fragment of datameer commands, it would still be very helpful. I don't need that the resulting queries are optimised, I would just need a "correct" translation, or say, compiler. Any opinion on this?

Thank you,

Lacramioara 

0

• 

  Alan

  • August 28, 2017 14:31
  • Edited

  Lacramioara,

  Unfortunately, I believe this is out of the scope of workbook architecture.  Additionally, what's entered as far as formulae is concerned on a workbook doesn't directly reflect what will be generated.  Depending on the cluster size/number of nodes - we may break up queries into smaller chunks for parallel execution.  Remember, Datameer is a job compiler. We build a job and submit it to the hadoop cluster.  It's a primary goal to remove the need to know or have to interact with SQL.

  The closest you could get would be to monitor hive itself during job execution to see what SQL was used to retrieve the data.

  Sincerely,

  Alan Mark

  0

  Comment actions Permalink
• 

  Lacramioara Astefanoaei

  • August 28, 2017 15:01

  Thank you Alan for your sharing!

  0

  Comment actions Permalink

Please sign in to leave a comment.