Skip to content
Projects
Groups
Snippets
Help
Loading...
Sign in
Toggle navigation
B
bi-report
Project
Project
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
data
bi-report
Commits
3d61b15d
Commit
3d61b15d
authored
Aug 20, 2020
by
魏艺敏
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
push codes
parent
866d4528
Show whitespace changes
Inline
Side-by-side
Showing
23 changed files
with
372 additions
and
21 deletions
+372
-21
clue_daily.sql
pm/clue_daily/etl/clue_daily.sql
+6
-11
clue_daily.sql
pm/clue_daily/report/clue_daily.sql
+10
-10
en-cn.properties
pm/sign_daliy/en-cn.properties
+2
-0
create_sign_daily.sql
pm/sign_daliy/etl/create_sign_daily.sql
+34
-0
sign_daily.sql
pm/sign_daliy/etl/sign_daily.sql
+249
-0
sign_daily.zip
pm/sign_daliy/job/sign_daily.zip
+0
-0
step1_1.job
pm/sign_daliy/job/step1_1.job
+4
-0
step1_10.job
pm/sign_daliy/job/step1_10.job
+4
-0
step1_11.job
pm/sign_daliy/job/step1_11.job
+4
-0
step1_12.job
pm/sign_daliy/job/step1_12.job
+4
-0
step1_13.job
pm/sign_daliy/job/step1_13.job
+4
-0
step1_2.job
pm/sign_daliy/job/step1_2.job
+4
-0
step1_3.job
pm/sign_daliy/job/step1_3.job
+4
-0
step1_4.job
pm/sign_daliy/job/step1_4.job
+4
-0
step1_5.job
pm/sign_daliy/job/step1_5.job
+4
-0
step1_6.job
pm/sign_daliy/job/step1_6.job
+4
-0
step1_7.job
pm/sign_daliy/job/step1_7.job
+4
-0
step1_8.job
pm/sign_daliy/job/step1_8.job
+4
-0
step1_9.job
pm/sign_daliy/job/step1_9.job
+4
-0
step2.job
pm/sign_daliy/job/step2.job
+5
-0
step3.job
pm/sign_daliy/job/step3.job
+5
-0
readme.txt
pm/sign_daliy/readme.txt
+0
-0
sign_daily.sql
pm/sign_daliy/report/sign_daily.sql
+9
-0
No files found.
pm/clue_daily/etl/clue_daily.sql
View file @
3d61b15d
...
...
@@ -453,8 +453,7 @@ FROM
FROM
online
.
al_business_merchant_grade_day_repo
where
partition_date
=
regexp_replace
(
date_sub
(
current_date
,
1
),
'-'
,
''
)
)
a
where
((
rn
=
1
and
merchant_level
>=
4
)
or
merchant_id
in
(
'6329571003697540960'
))
and
merchant_id
not
in
(
'6385685250516781117'
)
--去除和增加是为了和王妍的线索通数据保持一致
where
rn
=
1
and
merchant_level
>=
3
)
b
on
msg
.
merchant_id
=
b
.
merchant_id
GROUP
BY
msg
.
partition_date
,
user_id
...
...
@@ -483,8 +482,7 @@ FROM
FROM
online
.
al_business_merchant_grade_day_repo
where
partition_date
=
regexp_replace
(
date_sub
(
current_date
,
1
),
'-'
,
''
)
)
a
where
((
rn
=
1
and
merchant_level
>=
4
)
or
merchant_id
in
(
'6329571003697540960'
))
and
merchant_id
not
in
(
'6385685250516781117'
)
--去除和增加是为了和王妍的线索通数据保持一致
where
rn
=
1
and
merchant_level
>=
3
)
b
on
msg
.
merchant_id
=
b
.
merchant_id
GROUP
BY
msg
.
partition_date
,
user_id
...
...
@@ -512,8 +510,7 @@ FROM
FROM
online
.
al_business_merchant_grade_day_repo
where
partition_date
=
regexp_replace
(
date_sub
(
current_date
,
1
),
'-'
,
''
)
)
a
where
((
rn
=
1
and
merchant_level
>=
4
)
or
merchant_id
in
(
'6329571003697540960'
))
and
merchant_id
not
in
(
'6385685250516781117'
)
--去除和增加是为了和王妍的线索通数据保持一致
where
rn
=
1
and
merchant_level
>=
3
)
b
on
a
.
merchant_id
=
b
.
merchant_id
GROUP
BY
partition_date
,
user_id
...
...
@@ -935,7 +932,7 @@ LEFT JOIN
)
t9
ON
t3
.
partition_date
=
t9
.
partition_date
and
t3
.
user_id
=
t9
.
user_id
LEFT
JOIN
(
---
拨打
电话线索
(
---
有效
电话线索
SELECT
partition_date
,
user_id
,
count
(
1
)
as
valid_call_num
FROM
...
...
@@ -1061,8 +1058,7 @@ LEFT JOIN
FROM
online
.
al_business_merchant_grade_day_repo
where
partition_date
=
regexp_replace
(
date_sub
(
current_date
,
1
),
'-'
,
''
)
)
a
where
((
rn
=
1
and
merchant_level
>=
4
)
or
merchant_id
in
(
'6329571003697540960'
))
and
merchant_id
not
in
(
'6385685250516781117'
)
--去除和增加是为了和王妍的线索通数据保持一致
where
rn
=
1
and
merchant_level
>=
3
)
b
on
t1
.
merchant_id
=
b
.
merchant_id
GROUP
BY
user_id
,
partition_date
...
...
@@ -1128,8 +1124,7 @@ LEFT JOIN
FROM
online
.
al_business_merchant_grade_day_repo
where
partition_date
=
regexp_replace
(
date_sub
(
current_date
,
1
),
'-'
,
''
)
)
a
where
((
rn
=
1
and
merchant_level
>=
4
)
or
merchant_id
in
(
'6329571003697540960'
))
and
merchant_id
not
in
(
'6385685250516781117'
)
--去除和增加是为了和王妍的线索通数据保持一致
where
rn
=
1
and
merchant_level
>=
3
)
b
on
t1
.
merchant_id
=
b
.
merchant_id
GROUP
BY
user_id
,
partition_date
...
...
pm/clue_daily/report/clue_daily.sql
View file @
3d61b15d
...
...
@@ -15,16 +15,16 @@ SELECT day_id `日期`
,
valid_call_dev
as
`有效电话线索设备数-总计`
,
valid_call_num
as
`有效电话线索人次-总计`
,
msg_dev_4
`私信咨询设备数-
4
级以上商户`
,
msg_num_4
`私信咨询人次-
4
级以上商户`
,
valid_msg_dev_4
`有效私信咨询设备数-
4
级以上商户`
,
valid_msg_num_4
as
`有效私信咨询人次-
4
级以上商户`
,
clue_dev_4
as
`授权电话线索设备数-
4
级以上商户`
,
clue_num_4
as
`授权电话线索人次-
4
级以上商户`
,
call_dev_4
as
`拨打电话线索设备数-
4
级以上商户`
,
call_num_4
as
`拨打电话线索人次-
4
级以上商户`
,
valid_call_dev_4
as
`有效电话线索设备数-
4
级以上商户`
,
valid_call_num_4
as
`有效电话线索人次-
4
级以上商户`
,
msg_dev_4
`私信咨询设备数-
3
级以上商户`
,
msg_num_4
`私信咨询人次-
3
级以上商户`
,
valid_msg_dev_4
`有效私信咨询设备数-
3
级以上商户`
,
valid_msg_num_4
as
`有效私信咨询人次-
3
级以上商户`
,
clue_dev_4
as
`授权电话线索设备数-
3
级以上商户`
,
clue_num_4
as
`授权电话线索人次-
3
级以上商户`
,
call_dev_4
as
`拨打电话线索设备数-
3
级以上商户`
,
call_num_4
as
`拨打电话线索人次-
3
级以上商户`
,
valid_call_dev_4
as
`有效电话线索设备数-
3
级以上商户`
,
valid_call_num_4
as
`有效电话线索人次-
3
级以上商户`
,
msg_dev_kyc
`私信咨询设备数-kyc`
,
msg_num_kyc
`私信咨询人次-kyc`
...
...
pm/sign_daliy/en-cn.properties
0 → 100644
View file @
3d61b15d
sign_daily
=
签到用户质量日报
\ No newline at end of file
pm/sign_daliy/etl/create_sign_daily.sql
0 → 100644
View file @
3d61b15d
--***************************************************************
--*脚本名称:
--*功能: 签到用户质量日报
--*业务名称: pm
--*输入数据:
--*作者: weiyimin@igengmei.com
--*更新时间:
--***************************************************************
--设置全局变量&UDF
SET
mapreduce
.
job
.
queuename
=
data
;
--使用bl数据库
USE
pm
;
--创建BL层内部表
CREATE
TABLE
IF
NOT
EXISTS
pm
.
tl_pm_sign_deepaction_d
(
day_id
string
comment
'{"chs_name":"当天日期","description":"","etl":"","value":"","remark":""}'
,
device_os_type
string
comment
'{"chs_name":"设备类型","description":"","etl":"","value":"","remark":""}'
,
user_type
string
comment
'{"chs_name":"用户类型","description":"","etl":"","value":"","remark":""}'
,
active_num
bigint
comment
'{"chs_name":"活跃设备数","description":"","etl":"","value":"","remark":""}'
,
wel_pv
bigint
comment
'{"chs_name":"商品核心页浏览pv","description":"","etl":"","value":"","remark":""}'
,
wel_uv
bigint
comment
'{"chs_name":"商品核心页浏览uv","description":"","etl":"","value":"","remark":""}'
,
clue_num
bigint
comment
'{"chs_name":"有效线索人次","description":"","etl":"","value":"","remark":""}'
)
comment
'签到用户质量日报'
PARTITIONED
BY
(
PARTITION_DAY
STRING
comment
'分区日期'
)
ROW
FORMAT
DELIMITED
FIELDS
TERMINATED
BY
'
\t
'
COLLECTION
ITEMS
TERMINATED
BY
'
\0
02'
MAP
KEYS
TERMINATED
BY
'
\0
03'
LINES
TERMINATED
BY
'
\n
'
STORED
AS
TEXTFILE
;
\ No newline at end of file
pm/sign_daliy/etl/sign_daily.sql
0 → 100644
View file @
3d61b15d
SET
mapreduce
.
job
.
queuename
=
data
;
SET
mapreduce
.
map
.
memory
.
mb
=
8192
;
SET
mapreduce
.
map
.
java
.
opts
=-
Xmx8000m
;
SET
mapreduce
.
reduce
.
memory
.
mb
=
8192
;
SET
mapreduce
.
reduce
.
java
.
opts
=-
Xmx8000m
;
set
hive
.
auto
.
convert
.
join
=
true
;
SET
mapred
.
reduce
.
tasks
=
20
;
SET
role
admin
;
--线索日报
ADD
JAR
hdfs
:
///
user
/
hive
/
share
/
lib
/
udf
/
hive
-
udf
-
1
.
0
-
SNAPSHOT
.
jar
;
CREATE
TEMPORARY
FUNCTION
convup
AS
'com.gmei.hive.common.udf.UDFConvUpgrade'
;
CREATE
TEMPORARY
FUNCTION
setencryption
AS
'com.gmei.hive.common.udf.UDFStringSetEncryption'
;
INSERT
OVERWRITE
TABLE
pm
.
tl_pm_sign_deepaction_d
PARTITION
(
PARTITION_DAY
=
${
partition_day
}
)
SELECT
mas
.
partition_date
as
day_id
,
device_os_type
,
case
when
c
.
device_id
is
null
then
'非签到用户'
when
c
.
sign_date
=
c
.
first_sign_date
then
'首日签到用户'
when
c
.
sign_date
>
c
.
first_sign_date
then
'非首日签到用户'
end
as
user_type
,
count
(
distinct
mas
.
device_id
)
as
active_num
,
sum
(
wel_pv
)
as
wel_pv
,
count
(
distinct
case
when
wel_pv
<>
0
then
a
.
cl_id
end
)
as
wel_uv
,
sum
(
CASE
WHEN
b
.
device_id
is
NOT
NULL
THEN
b
.
valid_pv
END
)
as
clue_num
FROM
(
SELECT
partition_date
,
device_os_type
,
device_id
FROM
online
.
ml_device_day_active_status
WHERE
partition_date
>=
REGEXP_REPLACE
(
trunc
(
date_sub
(
current_date
(),
1
),
'MM'
),
'-'
,
''
)
AND
partition_date
<=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
AND
active_type
IN
(
'4'
)
AND
first_channel_source_type
not
IN
(
'yqxiu1'
,
'yqxiu2'
,
'yqxiu3'
,
'yqxiu4'
,
'yqxiu5'
,
'mxyc1'
,
'mxyc2'
,
'mxyc3'
,
'wanpu'
,
'jinshan'
,
'jx'
,
'maimai'
,
'zhuoyi'
,
'huatian'
,
'suopingjingling'
,
'mocha'
,
'mizhe'
,
'meika'
,
'lamabang'
,
'js-az1'
,
'js-az2'
,
'js-az3'
,
'js-az4'
,
'js-az5'
,
'jfq-az1'
,
'jfq-az2'
,
'jfq-az3'
,
'jfq-az4'
,
'jfq-az5'
,
'toufang1'
,
'toufang2'
,
'toufang3'
,
'toufang4'
,
'toufang5'
,
'toufang6'
,
'TF-toufang1'
,
'TF-toufang2'
,
'TF-toufang3'
,
'TF-toufang4'
,
'TF-toufang5'
,
'tf-toufang1'
,
'tf-toufang2'
,
'tf-toufang3'
,
'tf-toufang4'
,
'tf-toufang5'
,
'benzhan'
,
'promotion_aso100'
,
'promotion_qianka'
,
'promotion_xiaoyu'
,
'promotion_dianru'
,
'promotion_malioaso'
,
'promotion_malioaso-shequ'
,
'promotion_shike'
,
'promotion_julang_jl03'
,
'promotion_zuimei'
)
AND
first_channel_source_type
not
LIKE
'promotion
\_
jf
\_
%'
)
mas
LEFT
JOIN
(
SELECT
partition_date
,
cl_id
,
wel_pv
FROM
(
SELECT
partition_date
,
cl_id
,
count
(
1
)
as
wel_pv
FROM
online
.
bl_hdfs_maidian_updates
WHERE
partition_date
>=
REGEXP_REPLACE
(
trunc
(
date_sub
(
current_date
(),
1
),
'MM'
),
'-'
,
''
)
AND
partition_date
<=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
AND
action
=
'page_view'
AND
page_name
in
(
'welfare_detail'
,
'organization_detail'
,
'expert_detail'
)
GROUP
BY
partition_date
,
cl_id
)
a
LEFT
JOIN
(
-- 去掉疑似机构刷量的PV和UV
select
distinct
device_id
from
ml
.
ml_d_ct_dv_devicespam_d
where
partition_day
=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
)
spam_pv
on
a
.
cl_id
=
spam_pv
.
device_id
WHERE
spam_pv
.
device_id
IS
NULL
)
a
on
a
.
partition_date
=
mas
.
partition_date
and
a
.
cl_id
=
mas
.
device_id
LEFT
JOIN
(
--有效线索分日数据(按用户维度)
SELECT
t2
.
device_id
,
date_day
,
count
(
1
)
as
valid_pv
FROM
(
SELECT
NVL
(
T1
.
USER_ID
,
T2
.
USER_ID
)
AS
USER_ID
,
NVL
(
T1
.
merchant_id
,
T2
.
merchant_id
)
AS
merchant_id
,
NVL
(
T1
.
DATE_MONTH
,
T2
.
CALL_MONTH
)
AS
DATE_MONTH
,
least
(
NVL
(
DATE_DAY
,
'99999999'
),
NVL
(
FIRST_CALL_DATE
,
'99999999'
))
AS
DATE_DAY
,
NVL
(
T1
.
CONSULT_NUM
,
0
)
AS
CONSULT_NUM
,
NVL
(
T1
.
PAY_NUM
,
0
)
AS
PAY_NUM
,
NVL
(
T2
.
call_num
,
0
)
AS
CALL_NUM
FROM
(
SELECT
NVL
(
T3
.
USER_ID
,
T4
.
USER_ID
)
AS
USER_ID
,
NVL
(
T3
.
merchant_id
,
T4
.
merchant_id
)
AS
merchant_id
,
NVL
(
T3
.
CONSULT_MONTH
,
T4
.
PAY_MONTH
)
AS
DATE_MONTH
,
least
(
NVL
(
first_consult_date
,
'99999999'
),
NVL
(
FIRST_PAY_DATE
,
'99999999'
))
AS
DATE_DAY
,
NVL
(
T3
.
CONSULT_NUM
,
0
)
AS
CONSULT_NUM
,
NVL
(
T4
.
PAY_NUM
,
0
)
AS
PAY_NUM
FROM
(
SELECT
user_id
,
merchant_id
,
SUBSTR
(
partition_day
,
1
,
6
)
AS
consult_month
,
min
(
partition_day
)
as
first_consult_date
,
1
as
CONSULT_NUM
FROM
ml
.
ml_c_et_msg_conversation_dimen_inc_d
--新仓库私信表
WHERE
partition_day
>=
REGEXP_REPLACE
(
trunc
(
date_sub
(
current_date
(),
1
),
'MM'
),
'-'
,
''
)
AND
partition_day
<=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
and
is_valid
=
'true'
GROUP
BY
user_id
,
merchant_id
,
SUBSTR
(
partition_day
,
1
,
6
)
)
T3
FULL
JOIN
(
SELECT
T
.
USER_ID
,
T
.
merchant_id
,
regexp_replace
(
SUBSTR
(
T
.
PAY_DATE
,
1
,
7
),
'-'
,
''
)
AS
PAY_MONTH
,
MIN
(
regexp_replace
(
SUBSTR
(
T
.
PAY_DATE
,
1
,
10
),
'-'
,
''
))
AS
FIRST_PAY_DATE
,
1
AS
PAY_NUM
FROM
ONLINE
.
ML_TRADE_ORDER_DETAIL_DAY
T
WHERE
T
.
PARTITION_DATE
=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
AND
regexp_replace
(
SUBSTR
(
T
.
pay_date
,
1
,
10
),
'-'
,
''
)
>=
REGEXP_REPLACE
(
trunc
(
date_sub
(
current_date
(),
1
),
'MM'
),
'-'
,
''
)
AND
regexp_replace
(
SUBSTR
(
T
.
pay_date
,
1
,
10
),
'-'
,
''
)
<=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
AND
T
.
is_pure_user
=
'true'
GROUP
BY
T
.
USER_ID
,
T
.
merchant_id
,
regexp_replace
(
SUBSTR
(
T
.
PAY_DATE
,
1
,
7
),
'-'
,
''
)
)
T4
ON
T3
.
USER_ID
=
T4
.
USER_ID
AND
T3
.
merchant_id
=
T4
.
merchant_id
AND
T3
.
CONSULT_MONTH
=
T4
.
PAY_MONTH
GROUP
BY
NVL
(
T3
.
USER_ID
,
T4
.
USER_ID
),
NVL
(
T3
.
merchant_id
,
T4
.
merchant_id
),
NVL
(
T3
.
CONSULT_MONTH
,
T4
.
PAY_MONTH
),
least
(
NVL
(
first_consult_date
,
'99999999'
),
NVL
(
FIRST_PAY_DATE
,
'99999999'
)),
NVL
(
T3
.
CONSULT_NUM
,
0
),
NVL
(
T4
.
PAY_NUM
,
0
)
)
T1
FULL
JOIN
(
SELECT
user_id
,
merchant_id
,
regexp_replace
(
SUBSTR
(
a
.
partition_date
,
1
,
6
),
'-'
,
''
)
AS
CALL_MONTH
,
MIN
(
partition_date
)
as
FIRST_CALL_DATE
,
1
AS
call_num
FROM
(
SELECT
sub_id
,
REGEXP_REPLACE
(
SUBSTR
(
call_time
,
0
,
10
),
'-'
,
''
)
as
partition_date
FROM
tl
.
tl_gm_sl_ali_virtual_phone_call_detail
--通话记录表,call_type呼叫类型
WHERE
partition_day
=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
AND
REGEXP_REPLACE
(
SUBSTR
(
call_time
,
0
,
10
),
'-'
,
''
)
>=
REGEXP_REPLACE
(
trunc
(
date_sub
(
current_date
(),
1
),
'MM'
),
'-'
,
''
)
AND
REGEXP_REPLACE
(
SUBSTR
(
call_time
,
0
,
10
),
'-'
,
''
)
<=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
AND
start_time
<
release_time
--电话拨通
and
start_time
>
call_time
)
a
LEFT
JOIN
(
SELECT
id
,
sub_id
FROM
tl
.
tl_gm_sl_ali_virtual_phone_binding
--阿里记录表
WHERE
partition_day
=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
)
b
ON
a
.
sub_id
=
b
.
sub_id
LEFT
JOIN
(
SELECT
id
,
platform_binding_id
FROM
tl
.
tl_gm_sl_virtual_phone_binding
WHERE
partition_day
=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
)
c
ON
b
.
id
=
c
.
platform_binding_id
LEFT
JOIN
(
SELECT
phone_binding_id
,
lead_task_id
FROM
tl
.
tl_gm_sl_lead_task_phone_binding
WHERE
partition_day
=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
)
d
ON
c
.
id
=
d
.
phone_binding_id
JOIN
(
SELECT
id
,
user_id
,
merchant_id
FROM
tl
.
tl_gm_sl_lead_task
--线索任务表(用户点击授权后记入该表)
WHERE
partition_day
=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
AND
source
=
'2'
--用户行为电话授权
)
e
ON
d
.
lead_task_id
=
e
.
id
GROUP
BY
user_id
,
merchant_id
,
regexp_replace
(
SUBSTR
(
a
.
partition_date
,
1
,
6
),
'-'
,
''
)
)
T2
ON
T1
.
USER_ID
=
T2
.
USER_ID
AND
T1
.
merchant_id
=
T2
.
merchant_id
AND
T1
.
DATE_MONTH
=
T2
.
CALL_MONTH
)
t1
JOIN
(
SELECT
t
.
user_id
,
partition_date
,
if
(
size
(
t
.
device_list
)
>
0
,
device_list
[
0
],
''
)
device_id
FROM
online
.
ml_user_updates
t
WHERE
t
.
partition_date
>=
REGEXP_REPLACE
(
trunc
(
date_sub
(
current_date
(),
1
),
'MM'
),
'-'
,
''
)
and
t
.
partition_date
<=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
)
t2
on
t1
.
user_id
=
t2
.
user_id
and
t1
.
date_day
=
t2
.
partition_date
GROUP
BY
t2
.
device_id
,
date_day
)
b
ON
b
.
date_day
=
mas
.
partition_date
and
b
.
device_id
=
mas
.
device_id
LEFT
JOIN
(
SELECT
device_id
,
sign_date
,
first_sign_date
FROM
(
SELECT
user_id
,
REGEXP_REPLACE
(
substr
(
sign_time
,
1
,
10
),
'-'
,
''
)
as
sign_date
FROM
online
.
tl_hdfs_api_user_sign_v2_view
--20190601之后使用的用户签到表
WHERE
partition_date
=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
and
regexp_replace
(
substr
(
sign_time
,
1
,
10
),
'-'
,
''
)
>=
'20190601'
and
regexp_replace
(
substr
(
sign_time
,
1
,
10
),
'-'
,
''
)
<=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
and
sign_type
<>
'2'
--2表示补签
union
all
SELECT
user_id
,
REGEXP_REPLACE
(
substr
(
sign_time
,
1
,
10
),
'-'
,
''
)
as
sign_date
FROM
online
.
tl_hdfs_api_sign_record_view
--20190601之前使用的用户签到表
WHERE
partition_date
=
'20190531'
and
regexp_replace
(
substr
(
sign_time
,
1
,
10
),
'-'
,
''
)
>=
REGEXP_REPLACE
(
trunc
(
date_sub
(
current_date
(),
1
),
'MM'
),
'-'
,
''
)
and
regexp_replace
(
substr
(
sign_time
,
1
,
10
),
'-'
,
''
)
<
'20190601'
and
sign_type
<>
'3'
--3表示补签
)
a
left
join
(
SELECT
user_id
,
REGEXP_REPLACE
(
min
(
sign_date
),
'-'
,
''
)
as
first_sign_date
FROM
(
SELECT
user_id
,
substr
(
sign_time
,
1
,
10
)
as
sign_date
FROM
online
.
tl_hdfs_api_user_sign_v2_view
--20190601之后使用的用户签到表
WHERE
partition_date
=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
and
regexp_replace
(
substr
(
sign_time
,
1
,
10
),
'-'
,
''
)
>=
'20190601'
and
regexp_replace
(
substr
(
sign_time
,
1
,
10
),
'-'
,
''
)
<=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
and
sign_type
<>
'2'
--2表示补签
union
all
SELECT
user_id
,
substr
(
sign_time
,
1
,
10
)
as
sign_date
FROM
online
.
tl_hdfs_api_sign_record_view
--20190601之前使用的用户签到表 ,缺少20190101-20190226以及201905整月的数据
WHERE
partition_date
=
'20190531'
and
regexp_replace
(
substr
(
sign_time
,
1
,
10
),
'-'
,
''
)
>=
REGEXP_REPLACE
(
trunc
(
date_sub
(
current_date
(),
1
),
'MM'
),
'-'
,
''
)
and
regexp_replace
(
substr
(
sign_time
,
1
,
10
),
'-'
,
''
)
<
'20190601'
and
sign_type
<>
'3'
--3表示补签
)
first
group
by
user_id
)
a_first
on
a
.
user_id
=
a_first
.
user_id
LEFT
JOIN
(
SELECT
user_id
--医生账号
FROM
online
.
tl_hdfs_doctor_view
WHERE
partition_date
=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
)
b
ON
a
.
user_id
=
b
.
user_id
JOIN
(
select
t
.
user_id
,
partition_date
,
if
(
size
(
t
.
device_list
)
>
0
,
device_list
[
0
],
''
)
device_id
from
online
.
ml_user_updates
t
where
t
.
partition_date
>=
REGEXP_REPLACE
(
trunc
(
date_sub
(
current_date
(),
1
),
'MM'
),
'-'
,
''
)
and
t
.
partition_date
<=
regexp_replace
(
date_sub
(
current_date
(),
1
),
'-'
,
''
)
)
c
on
a
.
user_id
=
c
.
user_id
and
a
.
sign_date
=
c
.
partition_date
where
b
.
user_id
is
null
group
by
device_id
,
sign_date
,
first_sign_date
)
c
on
c
.
device_id
=
mas
.
device_id
and
c
.
sign_date
=
mas
.
partition_date
group
by
mas
.
partition_date
,
device_os_type
,
case
when
c
.
device_id
is
null
then
'非签到用户'
when
c
.
sign_date
=
c
.
first_sign_date
then
'首日签到用户'
when
c
.
sign_date
>
c
.
first_sign_date
then
'非首日签到用户'
end
order
by
1
,
2
,
3
pm/sign_daliy/job/sign_daily.zip
0 → 100644
View file @
3d61b15d
File added
pm/sign_daliy/job/step1_1.job
0 → 100644
View file @
3d61b15d
#step1_1.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive online bl_hdfs_maidian_updates
\ No newline at end of file
pm/sign_daliy/job/step1_10.job
0 → 100644
View file @
3d61b15d
#step1_3.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive tl tl_gm_sl_virtual_phone_binding
\ No newline at end of file
pm/sign_daliy/job/step1_11.job
0 → 100644
View file @
3d61b15d
#step1_3.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive tl tl_gm_sl_lead_task_phone_binding
\ No newline at end of file
pm/sign_daliy/job/step1_12.job
0 → 100644
View file @
3d61b15d
#step1_3.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive tl tl_gm_sl_lead_task
\ No newline at end of file
pm/sign_daliy/job/step1_13.job
0 → 100644
View file @
3d61b15d
#step1_13.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive online ml_user_updates
\ No newline at end of file
pm/sign_daliy/job/step1_2.job
0 → 100644
View file @
3d61b15d
#step1_2.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive online ml_device_day_active_status
\ No newline at end of file
pm/sign_daliy/job/step1_3.job
0 → 100644
View file @
3d61b15d
#step1_3.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive ml ml_d_ct_dv_devicespam_d
\ No newline at end of file
pm/sign_daliy/job/step1_4.job
0 → 100644
View file @
3d61b15d
#step1_3.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive online ML_TRADE_ORDER_DETAIL_DAY
\ No newline at end of file
pm/sign_daliy/job/step1_5.job
0 → 100644
View file @
3d61b15d
#step1_3.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive online tl_hdfs_api_user_sign_v2_view
\ No newline at end of file
pm/sign_daliy/job/step1_6.job
0 → 100644
View file @
3d61b15d
#step1_3.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive ml ml_c_et_msg_conversation_dimen_inc_d
\ No newline at end of file
pm/sign_daliy/job/step1_7.job
0 → 100644
View file @
3d61b15d
#step1_3.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive online tl_hdfs_doctor_view
\ No newline at end of file
pm/sign_daliy/job/step1_8.job
0 → 100644
View file @
3d61b15d
#step1_3.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive tl tl_gm_sl_ali_virtual_phone_call_detail
\ No newline at end of file
pm/sign_daliy/job/step1_9.job
0 → 100644
View file @
3d61b15d
#step1_3.job
type=command
command=sh /home/bi/bi-report/lib/shell/waitsuccess.sh hive tl tl_gm_sl_ali_virtual_phone_binding
\ No newline at end of file
pm/sign_daliy/job/step2.job
0 → 100644
View file @
3d61b15d
#step2.job
type=command
dependencies=step1_1,step1_2,step1_3,step1_4,step1_5,step1_6,step1_7,step1_8,step1_9,step1_10,step1_11,step1_12,step1_13
command=sh /home/bi/bi-report/lib/shell/hive clue_daily
\ No newline at end of file
pm/sign_daliy/job/step3.job
0 → 100644
View file @
3d61b15d
#step3.job
type=command
dependencies=step2
command=curl -X GET http://localhost:8553/api/report/email/clue_daily/liudi@igengmei.com,wangxin@igengmei.com,zhaoyang@igengmei.com/weiyimin@igengmei.com
\ No newline at end of file
pm/sign_daliy/readme.txt
0 → 100644
View file @
3d61b15d
pm/sign_daliy/report/sign_daily.sql
0 → 100644
View file @
3d61b15d
SELECT
day_id
as
`日期`
,
round
((
sum
(
CASE
WHEN
device_os_type
=
'ios'
AND
user_type
=
'非首日签到用户'
THEN
wel_pv
END
)
/
count
(
distinct
CASE
WHEN
device_os_type
=
'ios'
and
user_type
=
'非首日签到用户'
THEN
mas
.
device_id
END
))
/
(
sum
(
CASE
WHEN
device_os_type
=
'ios'
THEN
wel_pv
END
)
/
count
(
distinct
CASE
WHEN
device_os_type
=
'ios'
THEN
mas
.
device_id
END
)),
2
)
as
`单个iOS非首日签到贡献商品核心页PV/单个iOS老贡献商品核心页PV`
,
round
((
sum
(
CASE
WHEN
device_os_type
=
'ios'
AND
user_type
=
'非首日签到用户'
THEN
clue_num
END
)
/
count
(
distinct
CASE
WHEN
device_os_type
=
'ios'
AND
user_type
=
'非首日签到用户'
THEN
mas
.
device_id
END
))
/
(
sum
(
CASE
WHEN
device_os_type
=
'ios'
AND
b
.
device_id
is
NOT
NULL
THEN
b
.
valid_pv
END
)
/
count
(
distinct
CASE
WHEN
device_os_type
=
'ios'
THEN
mas
.
device_id
END
)),
2
)
as
`单个iOS非首日签到贡献有效线索人次/单个iOS老贡献有效线索人次`
,
round
((
sum
(
CASE
WHEN
device_os_type
=
'android'
AND
user_type
=
'非首日签到用户'
THEN
wel_pv
END
)
/
count
(
distinct
CASE
WHEN
device_os_type
=
'android'
and
user_type
=
'非首日签到用户'
THEN
mas
.
device_id
END
))
/
(
sum
(
CASE
WHEN
device_os_type
=
'android'
THEN
wel_pv
END
)
/
count
(
distinct
CASE
WHEN
device_os_type
=
'android'
THEN
mas
.
device_id
END
)),
2
)
as
`单个android非首日签到贡献商品核心页PV/单个android老贡献商品核心页PV`
,
round
((
sum
(
CASE
WHEN
device_os_type
=
'android'
AND
user_type
=
'非首日签到用户'
THEN
clue_num
END
)
/
count
(
distinct
CASE
WHEN
device_os_type
=
'android'
AND
user_type
=
'非首日签到用户'
THEN
mas
.
device_id
END
))
/
(
sum
(
CASE
WHEN
device_os_type
=
'android'
AND
b
.
device_id
is
NOT
NULL
THEN
b
.
valid_pv
END
)
/
count
(
distinct
CASE
WHEN
device_os_type
=
'android'
THEN
mas
.
device_id
END
)),
2
)
as
`单个android非首日签到贡献有效线索人次/单个android老贡献有效线索人次`
FROM
pm
.
tl_pm_sign_deepaction_d
where
partition_day
in
(
'20200901'
,
'202001001'
,
'20201101'
,
'20201201'
,
'20210101'
,
regexp_replace
(
date_sub
(
current_date
,
1
),
'-'
,
''
))
order
by
`日期`
,
`系统`
,
`新老`
,
`是否灰度`
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment