Skip to content
Projects
Groups
Snippets
Help
Loading...
Sign in
Toggle navigation
F
ffm-baseline
Project
Project
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
ML
ffm-baseline
Commits
35422345
Commit
35422345
authored
Sep 27, 2019
by
张彦钊
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
add
parent
6327ad41
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
3 additions
and
3 deletions
+3
-3
hello.py
hello.py
+3
-3
No files found.
hello.py
View file @
35422345
...
...
@@ -28,15 +28,15 @@ if __name__ == '__main__':
.
set
(
"spark.driver.maxResultSize"
,
"8g"
)
.
set
(
"spark.sql.avro.compression.codec"
,
"snappy"
)
spark
=
SparkSession
.
builder
.
config
(
conf
=
sparkConf
)
.
enableHiveSupport
()
.
getOrCreate
()
sql
=
"select params['exposure_cards'] from online.ml_community_precise_exposure_detail "
\
sql
=
"select params['exposure_cards']
,time_str
from online.ml_community_precise_exposure_detail "
\
"where action = 'page_precise_exposure' and page_name = 'search_result_welfare' "
\
"AND partition_date='20190926' limit 20"
df
=
spark
.
sql
(
sql
)
# df.show(6)
rdd
=
df
.
rdd
.
map
(
lambda
x
:(
x
[
0
]
))
.
map
(
lambda
x
:(
eval
(
x
[
0
])))
.
map
(
lambda
x
:(
ctr
(
x
[
0
])
))
rdd
=
df
.
rdd
.
map
(
lambda
x
:(
x
[
0
]
,
x
[
1
]))
.
map
(
lambda
x
:(
eval
(
x
[
0
]),
x
[
1
]))
.
map
(
lambda
x
:(
ctr
(
x
[
0
]),
x
[
1
]
))
print
(
rdd
.
collect
())
spark
.
createDataFrame
(
rdd
)
.
toDF
(
"n"
)
.
show
(
6
)
spark
.
createDataFrame
(
rdd
)
.
show
(
6
)
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment