Skip to content

M
meta_base_code
Commit 5a246adf authored by 郭羽's avatar 郭羽
Browse files
Options

reduce collect()

parent e1604635
Show whitespace changes
Inline Side-by-side
Showing with 3 additions and 3 deletions
task/search_strategy_d.py
View file @ 5a246adf
...@@ -71,7 +71,7 @@ spark.sql("CREATE TEMPORARY FUNCTION is_json AS 'com.gmei.hive.common.udf.UDFJso ...@@ -71,7 +71,7 @@ spark.sql("CREATE TEMPORARY FUNCTION is_json AS 'com.gmei.hive.common.udf.UDFJso
spark.sql("CREATE TEMPORARY FUNCTION arrayMerge AS 'com.gmei.hive.common.udf.UDFArryMerge'") spark.sql("CREATE TEMPORARY FUNCTION arrayMerge AS 'com.gmei.hive.common.udf.UDFArryMerge'")
task_list = [] task_list = []
task_days = 5 task_days = 2
for t in range(0, task_days): for t in range(0, task_days):
day_num = 0 - t day_num = 0 - t
...@@ -151,8 +151,8 @@ for t in range(0, task_days): ...@@ -151,8 +151,8 @@ for t in range(0, task_days):
spam_pv_df = spark.sql(sql_spam_pv_device_id) spam_pv_df = spark.sql(sql_spam_pv_device_id)
spam_pv_df.createOrReplaceTempView("spam_pv") spam_pv_df.createOrReplaceTempView("spam_pv")
spam_pv_df.show(1) spam_pv_df.show(1)
sql_res = spam_pv_df.collect() # sql_res = spam_pv_df.collect()
spam_pv_df.cache() # spam_pv_df.cache()
# for res in sql_res: # for res in sql_res:
# print(res) # print(res)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment