Skip to content
Projects
Groups
Snippets
Help
Loading...
Sign in
Toggle navigation
F
ffm-baseline
Project
Project
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
ML
ffm-baseline
Commits
0217ad56
Commit
0217ad56
authored
Dec 18, 2018
by
高雅喆
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
filter imp data where count > 1
parent
574c9a4f
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
14 additions
and
3 deletions
+14
-3
EsmmData.scala
eda/feededa/src/main/scala/com/gmei/EsmmData.scala
+14
-3
No files found.
eda/feededa/src/main/scala/com/gmei/EsmmData.scala
View file @
0217ad56
...
...
@@ -69,13 +69,24 @@ object EsmmData {
if
(
max_stat_date_str
!=
param
.
date
){
val
stat_date
=
param
.
date
println
(
stat_date
)
// val imp_data = sc.sql(
// s"""
// |select distinct stat_date,device_id,city_id as ucity_id,
// | cid_id,diary_service_id
// |from data_feed_exposure
// |where cid_type = 'diary'
// |and stat_date ='${stat_date}'
// """.stripMargin
// )
val
imp_data
=
sc
.
sql
(
s
"""
|select
distinct stat_date,device_id,city_id as ucity_id,
|
cid_id,diary_service_id
|select
* from
|
(select stat_date,device_id,city_id as ucity_id,
cid_id,diary_service_id
|from data_feed_exposure
|where cid_type = 'diary'
|and stat_date ='${stat_date}'
|group by stat_date,device_id,city_id,cid_id,diary_service_id having count(*) > 1) a
"""
.
stripMargin
)
// imp_data.show()
...
...
@@ -200,7 +211,7 @@ object EsmmData {
)
// union_data_scity_id.createOrReplaceTempView("union_data_scity_id")
union_data_scity_id
.
show
()
GmeiConfig
.
writeToJDBCTable
(
"jdbc:mysql://10.66.157.22:4000/jerry_test?user=root&password=3SYz54LS9#^9sBvC&rewriteBatchedStatements=true"
,
union_data_scity_id
,
table
=
"esmm_train_
data
"
,
SaveMode
.
Append
)
GmeiConfig
.
writeToJDBCTable
(
"jdbc:mysql://10.66.157.22:4000/jerry_test?user=root&password=3SYz54LS9#^9sBvC&rewriteBatchedStatements=true"
,
union_data_scity_id
,
table
=
"esmm_train_
test
"
,
SaveMode
.
Append
)
}
else
{
println
(
"esmm_train_data already have param.date data"
)
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment