Skip to content
Projects
Groups
Snippets
Help
Loading...
Sign in
Toggle navigation
C
crawler
Project
Project
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
backend
crawler
Commits
94a03ced
Commit
94a03ced
authored
Sep 21, 2020
by
litaolemo
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
udpate
parent
21dc7f75
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
6 additions
and
6 deletions
+6
-6
update_data_in_target_releasers_multi_process_by_date_from_redis.py
...a_in_target_releasers_multi_process_by_date_from_redis.py
+6
-6
No files found.
crawler_sys/framework/update_data_in_target_releasers_multi_process_by_date_from_redis.py
View file @
94a03ced
...
@@ -26,7 +26,7 @@ from crawler.crawler_sys.site_crawler_by_redis import crawler_weibo
...
@@ -26,7 +26,7 @@ from crawler.crawler_sys.site_crawler_by_redis import crawler_weibo
import
sys
import
sys
from
crawler.crawler_sys.utils.output_results
import
output_result
from
crawler.crawler_sys.utils.output_results
import
output_result
import
argparse
,
copy
,
datetime
,
time
import
argparse
,
copy
,
datetime
,
time
from
crawler.crawler_sys.framework.es_target_releasers
import
get_releaserUrls_from_es
#
from crawler.crawler_sys.framework.es_target_releasers import get_releaserUrls_from_es
from
crawler.crawler_sys.utils.parse_bool_for_args
import
parse_bool_for_args
from
crawler.crawler_sys.utils.parse_bool_for_args
import
parse_bool_for_args
import
redis
,
json
import
redis
,
json
from
concurrent.futures
import
ProcessPoolExecutor
from
concurrent.futures
import
ProcessPoolExecutor
...
@@ -262,12 +262,12 @@ if __name__ == "__main__":
...
@@ -262,12 +262,12 @@ if __name__ == "__main__":
platform_str
=
platform
+
"_process"
platform_str
=
platform
+
"_process"
rds_1
.
set
(
platform_str
,
0
)
rds_1
.
set
(
platform_str
,
0
)
for
processe
in
range
(
processes_num
):
for
processe
in
range
(
processes_num
):
#
start_crawler(processe,name)
start_crawler
(
processe
,
name
)
# print(kwargs_dict)
# print(kwargs_dict)
future
=
executor
.
submit
(
start_crawler
,
processe
,
name
)
#
future = executor.submit(start_crawler,processe,name)
futures
.
append
(
future
)
#
futures.append(future)
print
(
'Processe
%
s start'
%
processe
)
#
print('Processe %s start' % processe)
executor
.
shutdown
(
True
)
#
executor.shutdown(True)
# while True:
# while True:
# now = datetime.datetime.now()
# now = datetime.datetime.now()
# if now.hour >= exit_hour:
# if now.hour >= exit_hour:
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment