Skip to content
Projects
Groups
Snippets
Help
Loading...
Sign in
Toggle navigation
D
DQMP
Project
Project
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
data
DQMP
Commits
ccd5210a
Commit
ccd5210a
authored
Dec 23, 2019
by
赵建伟
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
update codes
parent
01bc8030
Hide whitespace changes
Inline
Side-by-side
Showing
13 changed files
with
169 additions
and
8 deletions
+169
-8
TblResultCheckRefer.java
...n/java/com/gmei/data/dqmp/domain/TblResultCheckRefer.java
+24
-0
TblResultCheckUnblank.java
...java/com/gmei/data/dqmp/domain/TblResultCheckUnblank.java
+21
-0
TblResultCheckUnique.java
.../java/com/gmei/data/dqmp/domain/TblResultCheckUnique.java
+21
-0
TblResultMonitorSpecial.java
...va/com/gmei/data/dqmp/domain/TblResultMonitorSpecial.java
+21
-0
TblResultMonitorVolatility.java
...com/gmei/data/dqmp/domain/TblResultMonitorVolatility.java
+20
-0
MailService.java
src/main/java/com/gmei/data/dqmp/service/MailService.java
+1
-1
CheckServiceImpl.java
...ava/com/gmei/data/dqmp/service/impl/CheckServiceImpl.java
+20
-0
MailServiceImpl.java
...java/com/gmei/data/dqmp/service/impl/MailServiceImpl.java
+5
-1
MonitorServiceImpl.java
...a/com/gmei/data/dqmp/service/impl/MonitorServiceImpl.java
+20
-0
application-dev.yml
src/main/resources/application-dev.yml
+3
-0
application-prod.yml
src/main/resources/application-prod.yml
+3
-0
application-test.yml
src/main/resources/application-test.yml
+3
-0
generator.xml
src/main/resources/generator.xml
+7
-6
No files found.
src/main/java/com/gmei/data/dqmp/domain/TblResultCheckRefer.java
View file @
ccd5210a
...
...
@@ -130,4 +130,27 @@ public class TblResultCheckRefer {
public
void
setCreateTime
(
String
createTime
)
{
this
.
createTime
=
createTime
==
null
?
null
:
createTime
.
trim
();
}
@Override
public
String
toString
()
{
StringBuilder
sb
=
new
StringBuilder
();
sb
.
append
(
getClass
().
getSimpleName
());
sb
.
append
(
" ["
);
sb
.
append
(
"Hash = "
).
append
(
hashCode
());
sb
.
append
(
", id="
).
append
(
id
);
sb
.
append
(
", checkDbName="
).
append
(
checkDbName
);
sb
.
append
(
", checkTbName="
).
append
(
checkTbName
);
sb
.
append
(
", checkColName="
).
append
(
checkColName
);
sb
.
append
(
", referDbName="
).
append
(
referDbName
);
sb
.
append
(
", referTbName="
).
append
(
referTbName
);
sb
.
append
(
", referColName="
).
append
(
referColName
);
sb
.
append
(
", allNum="
).
append
(
allNum
);
sb
.
append
(
", matchedNum="
).
append
(
matchedNum
);
sb
.
append
(
", matchedRate="
).
append
(
matchedRate
);
sb
.
append
(
", partitionDate="
).
append
(
partitionDate
);
sb
.
append
(
", sqlId="
).
append
(
sqlId
);
sb
.
append
(
", createTime="
).
append
(
createTime
);
sb
.
append
(
"]"
);
return
sb
.
toString
();
}
}
\ No newline at end of file
src/main/java/com/gmei/data/dqmp/domain/TblResultCheckUnblank.java
View file @
ccd5210a
...
...
@@ -100,4 +100,24 @@ public class TblResultCheckUnblank {
public
void
setCreateTime
(
String
createTime
)
{
this
.
createTime
=
createTime
==
null
?
null
:
createTime
.
trim
();
}
@Override
public
String
toString
()
{
StringBuilder
sb
=
new
StringBuilder
();
sb
.
append
(
getClass
().
getSimpleName
());
sb
.
append
(
" ["
);
sb
.
append
(
"Hash = "
).
append
(
hashCode
());
sb
.
append
(
", id="
).
append
(
id
);
sb
.
append
(
", dbName="
).
append
(
dbName
);
sb
.
append
(
", tbName="
).
append
(
tbName
);
sb
.
append
(
", colName="
).
append
(
colName
);
sb
.
append
(
", allNum="
).
append
(
allNum
);
sb
.
append
(
", unblankNum="
).
append
(
unblankNum
);
sb
.
append
(
", unblankRate="
).
append
(
unblankRate
);
sb
.
append
(
", partitionDate="
).
append
(
partitionDate
);
sb
.
append
(
", sqlId="
).
append
(
sqlId
);
sb
.
append
(
", createTime="
).
append
(
createTime
);
sb
.
append
(
"]"
);
return
sb
.
toString
();
}
}
\ No newline at end of file
src/main/java/com/gmei/data/dqmp/domain/TblResultCheckUnique.java
View file @
ccd5210a
...
...
@@ -100,4 +100,24 @@ public class TblResultCheckUnique {
public
void
setCreateTime
(
String
createTime
)
{
this
.
createTime
=
createTime
==
null
?
null
:
createTime
.
trim
();
}
@Override
public
String
toString
()
{
StringBuilder
sb
=
new
StringBuilder
();
sb
.
append
(
getClass
().
getSimpleName
());
sb
.
append
(
" ["
);
sb
.
append
(
"Hash = "
).
append
(
hashCode
());
sb
.
append
(
", id="
).
append
(
id
);
sb
.
append
(
", dbName="
).
append
(
dbName
);
sb
.
append
(
", tbName="
).
append
(
tbName
);
sb
.
append
(
", colName="
).
append
(
colName
);
sb
.
append
(
", allNum="
).
append
(
allNum
);
sb
.
append
(
", uniqueNum="
).
append
(
uniqueNum
);
sb
.
append
(
", uniqueRate="
).
append
(
uniqueRate
);
sb
.
append
(
", partitionDate="
).
append
(
partitionDate
);
sb
.
append
(
", sqlId="
).
append
(
sqlId
);
sb
.
append
(
", createTime="
).
append
(
createTime
);
sb
.
append
(
"]"
);
return
sb
.
toString
();
}
}
\ No newline at end of file
src/main/java/com/gmei/data/dqmp/domain/TblResultMonitorSpecial.java
View file @
ccd5210a
...
...
@@ -100,4 +100,24 @@ public class TblResultMonitorSpecial {
public
void
setCreateTime
(
String
createTime
)
{
this
.
createTime
=
createTime
==
null
?
null
:
createTime
.
trim
();
}
@Override
public
String
toString
()
{
StringBuilder
sb
=
new
StringBuilder
();
sb
.
append
(
getClass
().
getSimpleName
());
sb
.
append
(
" ["
);
sb
.
append
(
"Hash = "
).
append
(
hashCode
());
sb
.
append
(
", id="
).
append
(
id
);
sb
.
append
(
", dbName="
).
append
(
dbName
);
sb
.
append
(
", tbName="
).
append
(
tbName
);
sb
.
append
(
", timeColName="
).
append
(
timeColName
);
sb
.
append
(
", indicatorType="
).
append
(
indicatorType
);
sb
.
append
(
", indicatorValue="
).
append
(
indicatorValue
);
sb
.
append
(
", startTime="
).
append
(
startTime
);
sb
.
append
(
", partitionDate="
).
append
(
partitionDate
);
sb
.
append
(
", sqlId="
).
append
(
sqlId
);
sb
.
append
(
", createTime="
).
append
(
createTime
);
sb
.
append
(
"]"
);
return
sb
.
toString
();
}
}
\ No newline at end of file
src/main/java/com/gmei/data/dqmp/domain/TblResultMonitorVolatility.java
View file @
ccd5210a
...
...
@@ -90,4 +90,23 @@ public class TblResultMonitorVolatility {
public
void
setCreateTime
(
String
createTime
)
{
this
.
createTime
=
createTime
==
null
?
null
:
createTime
.
trim
();
}
@Override
public
String
toString
()
{
StringBuilder
sb
=
new
StringBuilder
();
sb
.
append
(
getClass
().
getSimpleName
());
sb
.
append
(
" ["
);
sb
.
append
(
"Hash = "
).
append
(
hashCode
());
sb
.
append
(
", id="
).
append
(
id
);
sb
.
append
(
", dbName="
).
append
(
dbName
);
sb
.
append
(
", tbName="
).
append
(
tbName
);
sb
.
append
(
", colName="
).
append
(
colName
);
sb
.
append
(
", indicatorType="
).
append
(
indicatorType
);
sb
.
append
(
", indicatorValue="
).
append
(
indicatorValue
);
sb
.
append
(
", partitionDate="
).
append
(
partitionDate
);
sb
.
append
(
", sqlId="
).
append
(
sqlId
);
sb
.
append
(
", createTime="
).
append
(
createTime
);
sb
.
append
(
"]"
);
return
sb
.
toString
();
}
}
\ No newline at end of file
src/main/java/com/gmei/data/dqmp/service/MailService.java
View file @
ccd5210a
...
...
@@ -2,6 +2,6 @@ package com.gmei.data.dqmp.service;
public
interface
MailService
{
void
sendSimpleMail
(
String
to
,
String
subject
,
String
content
);
void
sendSimpleMail
(
String
content
);
}
src/main/java/com/gmei/data/dqmp/service/impl/CheckServiceImpl.java
View file @
ccd5210a
...
...
@@ -63,12 +63,19 @@ public class CheckServiceImpl implements CheckService {
for
(
TblSqlCheckSingle
tblCheckSql
:
sqlList
)
{
String
sql
=
tblCheckSql
.
getSqlContent
();
Integer
id
=
tblCheckSql
.
getId
();
Double
threshold
=
tblCheckSql
.
getThreshold
();
int
rs
=
0
;
if
(
Constants
.
CHECK_UNIQUE
.
equals
(
tblCheckSql
.
getCheckType
()))
{
TblResultCheckUnique
uniqueCheckResult
=
getUniqueCheckResult
(
sql
);
if
(
uniqueCheckResult
==
null
)
{
logger
.
error
(
"Find result is empty!"
);
return
;
}
else
{
Double
uniqueRate
=
uniqueCheckResult
.
getUniqueRate
();
if
(
uniqueRate
<=
threshold
)
{
mailService
.
sendSimpleMail
(
uniqueCheckResult
.
toString
());
dingdingService
.
sendMsgToDingding
(
uniqueCheckResult
.
toString
());
}
}
uniqueCheckResult
.
setSqlId
(
id
);
rs
=
tblResultCheckUniqueMapper
.
insert
(
uniqueCheckResult
);
...
...
@@ -77,6 +84,12 @@ public class CheckServiceImpl implements CheckService {
if
(
tblResultCheckUnblank
==
null
)
{
logger
.
error
(
"Find result is empty!"
);
return
;
}
else
{
Double
unblankRate
=
tblResultCheckUnblank
.
getUnblankRate
();
if
(
unblankRate
<=
threshold
)
{
mailService
.
sendSimpleMail
(
tblResultCheckUnblank
.
toString
());
dingdingService
.
sendMsgToDingding
(
tblResultCheckUnblank
.
toString
());
}
}
tblResultCheckUnblank
.
setSqlId
(
id
);
rs
=
tblResultCheckUnblankMapper
.
insert
(
tblResultCheckUnblank
);
...
...
@@ -103,12 +116,19 @@ public class CheckServiceImpl implements CheckService {
for
(
TblSqlCheckDuplex
tblCheckSql
:
sqlList
)
{
String
sql
=
tblCheckSql
.
getSqlContent
();
Integer
id
=
tblCheckSql
.
getId
();
Double
threshold
=
tblCheckSql
.
getThreshold
();
int
rs
=
0
;
if
(
Constants
.
CHECK_REFER
.
equals
(
tblCheckSql
.
getCheckType
()))
{
TblResultCheckRefer
referCheckResult
=
getReferCheckResult
(
sql
);
if
(
referCheckResult
==
null
)
{
logger
.
error
(
"Find result is empty!"
);
return
;
}
else
{
Double
matchedRate
=
referCheckResult
.
getMatchedRate
();
if
(
matchedRate
<=
threshold
)
{
mailService
.
sendSimpleMail
(
referCheckResult
.
toString
());
dingdingService
.
sendMsgToDingding
(
referCheckResult
.
toString
());
}
}
referCheckResult
.
setSqlId
(
id
);
rs
=
tblResultCheckReferMapper
.
insert
(
referCheckResult
);
...
...
src/main/java/com/gmei/data/dqmp/service/impl/MailServiceImpl.java
View file @
ccd5210a
...
...
@@ -19,9 +19,13 @@ public class MailServiceImpl implements MailService{
@Value
(
"${mail.fromMail.addr}"
)
private
String
from
;
@Value
(
"${mail.toMail.addr}"
)
private
String
to
;
@Value
(
"${mail.toMail.subject}"
)
private
String
subject
;
@Override
public
void
sendSimpleMail
(
String
to
,
String
subject
,
String
content
)
{
public
void
sendSimpleMail
(
String
content
)
{
SimpleMailMessage
message
=
new
SimpleMailMessage
();
message
.
setFrom
(
from
);
message
.
setTo
(
to
);
...
...
src/main/java/com/gmei/data/dqmp/service/impl/MonitorServiceImpl.java
View file @
ccd5210a
...
...
@@ -23,6 +23,8 @@ import com.gmei.data.dqmp.mapper.TblResultMonitorVolatilityMapper;
import
com.gmei.data.dqmp.mapper.TblSqlMonitorSpecialMapper
;
import
com.gmei.data.dqmp.mapper.TblSqlMonitorVolatilityMapper
;
import
com.gmei.data.dqmp.pool.JdbcConnectPool
;
import
com.gmei.data.dqmp.service.DingdingService
;
import
com.gmei.data.dqmp.service.MailService
;
import
com.gmei.data.dqmp.service.MonitorService
;
import
com.gmei.data.dqmp.utils.DateUtils
;
...
...
@@ -39,6 +41,10 @@ public class MonitorServiceImpl implements MonitorService {
private
TblResultMonitorVolatilityMapper
tblResultMonitorVolatilityMapper
;
@Autowired
private
TblResultMonitorSpecialMapper
tblResultMonitorSpecialMapper
;
@Autowired
private
MailService
mailService
;
@Autowired
private
DingdingService
dingdingService
;
/**
* 监控并持久化波动型监控结果
...
...
@@ -54,11 +60,18 @@ public class MonitorServiceImpl implements MonitorService {
for
(
TblSqlMonitorVolatility
tblCheckSql
:
sqlList
)
{
String
sql
=
tblCheckSql
.
getSqlContent
();
Integer
id
=
tblCheckSql
.
getId
();
Double
threshold
=
tblCheckSql
.
getThreshold
();
int
rs
=
0
;
TblResultMonitorVolatility
tblResultMonitorVolatility
=
getMonitorVolatilityResult
(
sql
,
indicatorType
);
if
(
tblResultMonitorVolatility
==
null
)
{
logger
.
error
(
"Find result is empty!"
);
return
;
}
else
{
Long
indicatorValue
=
tblResultMonitorVolatility
.
getIndicatorValue
();
if
(
indicatorValue
<=
threshold
)
{
mailService
.
sendSimpleMail
(
tblResultMonitorVolatility
.
toString
());
dingdingService
.
sendMsgToDingding
(
tblResultMonitorVolatility
.
toString
());
}
}
tblResultMonitorVolatility
.
setSqlId
(
id
);
rs
=
tblResultMonitorVolatilityMapper
.
insert
(
tblResultMonitorVolatility
);
...
...
@@ -82,11 +95,18 @@ public class MonitorServiceImpl implements MonitorService {
for
(
TblSqlMonitorSpecial
tblCheckSql
:
sqlList
)
{
String
sql
=
tblCheckSql
.
getSqlContent
();
Integer
id
=
tblCheckSql
.
getId
();
Double
threshold
=
tblCheckSql
.
getThreshold
();
int
rs
=
0
;
TblResultMonitorSpecial
tblResultMonitorSpecial
=
getMonitorSpecialResult
(
sql
);
if
(
tblResultMonitorSpecial
==
null
)
{
logger
.
error
(
"Find result is empty!"
);
return
;
}
else
{
Long
indicatorValue
=
tblResultMonitorSpecial
.
getIndicatorValue
();
if
(
indicatorValue
<=
threshold
)
{
mailService
.
sendSimpleMail
(
tblResultMonitorSpecial
.
toString
());
dingdingService
.
sendMsgToDingding
(
tblResultMonitorSpecial
.
toString
());
}
}
tblResultMonitorSpecial
.
setSqlId
(
id
);
rs
=
tblResultMonitorSpecialMapper
.
insert
(
tblResultMonitorSpecial
);
...
...
src/main/resources/application-dev.yml
View file @
ccd5210a
...
...
@@ -52,6 +52,9 @@ spring:
mail
:
fromMail
:
addr
:
youjiantongji@wanmeizhensuo.com
toMail
:
addr
:
zhaojianwei@igengmei.com
subject
:
数据质量预警
#---dingding send config---
dingding
:
...
...
src/main/resources/application-prod.yml
View file @
ccd5210a
...
...
@@ -52,6 +52,9 @@ spring:
mail
:
fromMail
:
addr
:
youjiantongji@wanmeizhensuo.com
toMail
:
addr
:
zhaojianwei@igengmei.com
subject
:
数据质量预警
#---dingding send config---
dingding
:
...
...
src/main/resources/application-test.yml
View file @
ccd5210a
...
...
@@ -52,6 +52,9 @@ spring:
mail
:
fromMail
:
addr
:
youjiantongji@wanmeizhensuo.com
toMail
:
addr
:
zhaojianwei@igengmei.com
subject
:
数据质量预警
#---dingding send config---
dingding
:
...
...
src/main/resources/generator.xml
View file @
ccd5210a
...
...
@@ -6,6 +6,7 @@
<!-- jdbc路径 -->
<classPathEntry
location=
"/Users/apple/.m2/repository/mysql/mysql-connector-java/5.1.38/mysql-connector-java-5.1.38.jar"
/>
<context
id=
"context"
targetRuntime=
"MyBatis3"
>
<plugin
type=
"org.mybatis.generator.plugins.ToStringPlugin"
></plugin>
<plugin
type=
"org.mybatis.generator.plugins.RenameExampleClassPlugin"
>
<property
name=
"searchString"
value=
"Example$"
/>
<property
name=
"replaceString"
value=
"Criteria"
/>
...
...
@@ -35,23 +36,23 @@
<!-- 需要生成的数据库表 -->
<!-- <table tableName="tbl_client_version_info" domainObjectName="TblClientVersionInfo"/> -->
<table
tableName=
"tbl_sql_check_single"
domainObjectName=
"TblSqlCheckSingle"
/>
<
!-- <
table tableName="tbl_sql_check_single" domainObjectName="TblSqlCheckSingle"/>
<table tableName="tbl_sql_check_duplex" domainObjectName="TblSqlCheckDuplex"/>
<table tableName="tbl_sql_monitor_volatility" domainObjectName="TblSqlMonitorVolatility"/>
<table
tableName=
"tbl_sql_monitor_special"
domainObjectName=
"TblSqlMonitorSpecial"
/>
<!-- <table tableName="tbl_result_check_unique" domainObjectName="TblResultCheckUnique"/>
<table tableName="tbl_sql_monitor_special" domainObjectName="TblSqlMonitorSpecial"/>
-->
<!-- <table tableName="tbl_result_check_unique" domainObjectName="TblResultCheckUnique"/>
-->
<table
tableName=
"tbl_result_check_unblank"
domainObjectName=
"TblResultCheckUnblank"
/>
<table
tableName=
"tbl_result_check_refer"
domainObjectName=
"TblResultCheckRefer"
/>
<table
tableName=
"tbl_result_monitor_volatility"
domainObjectName=
"TblResultMonitorVolatility"
/>
<table
tableName=
"tbl_result_monitor_special"
domainObjectName=
"TblResultMonitorSpecial"
/>
<table tableName="tbl_indicator_operator_info" domainObjectName="TblIndicatorOperatorInfo"/> -->
<
!-- <
table tableName="tbl_indicator_operator_info" domainObjectName="TblIndicatorOperatorInfo"/> -->
<!-- <table tableName="tbl_rule_template" domainObjectName="TblRuleTemplate"/> -->
<table
tableName=
"tbl_rule_info_simple"
domainObjectName=
"TblRuleInfoSimple"
>
<
!-- <
table tableName="tbl_rule_info_simple" domainObjectName="TblRuleInfoSimple">
<generatedKey column="id" sqlStatement="MySql" identity="true"/>
</table>
<table tableName="tbl_rule_info_duplex" domainObjectName="TblRuleInfoDuplex">
<generatedKey column="id" sqlStatement="MySql" identity="true"/>
</table>
</table>
-->
<!-- <table tableName="tbl_sys_param_info" domainObjectName="TblSysParamInfo"/> -->
</context>
</generatorConfiguration>
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment