Skip to content
Projects
Groups
Snippets
Help
Loading...
Sign in
Toggle navigation
F
ffm-baseline
Project
Project
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
ML
ffm-baseline
Commits
8cc061c7
Commit
8cc061c7
authored
Dec 07, 2018
by
张彦钊
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
修改rdd
parent
a6d50b78
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
7 additions
and
3 deletions
+7
-3
Data2FFM.scala
eda/feededa/src/main/scala/com/gmei/Data2FFM.scala
+7
-3
No files found.
eda/feededa/src/main/scala/com/gmei/Data2FFM.scala
View file @
8cc061c7
...
...
@@ -60,7 +60,11 @@ object Data2FFM {
).
na
.
drop
()
esmm_data
.
show
(
8
)
println
(
esmm_data
.
count
())
val
column_list
=
esmm_data
.
columns
esmm_data
.
createOrReplaceTempView
(
"df"
)
val
sql1
=
"select * from df"
val
df
=
sc
.
sql
(
sql1
)
val
column_list
=
df
.
columns
val
max_stat_date
=
sc
.
sql
(
s
"""
...
...
@@ -76,13 +80,13 @@ object Data2FFM {
val
column_number
=
scala
.
collection
.
mutable
.
Map
[
String
,
Array
[
String
]]()
for
(
i
<-
column_list
){
column_number
(
i
)
=
esmm_data
.
select
(
i
).
distinct
().
collect
().
map
(
x
=>
x
(
0
).
toString
)
column_number
(
i
)
=
df
.
select
(
i
).
distinct
().
collect
().
map
(
x
=>
x
(
0
).
toString
)
}
val
a
=
column_number
(
"device_id"
).
toList
println
(
a
)
println
(
"dict"
)
val
rdd
=
esmm_data
.
rdd
.
repartition
(
200
)
val
rdd
=
df
.
rdd
.
repartition
(
200
)
.
map
(
x
=>
(
x
(
0
).
toString
,
x
(
1
).
toString
,
x
(
2
).
toString
,
x
(
3
).
toString
,
x
(
4
).
toString
,
x
(
5
).
toString
,
x
(
6
).
toString
,
x
(
7
).
toString
))
rdd
.
persist
()
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment