crawler
发布者页爬虫
- 部署在BJ-GM-Prod-Cos-faiss001/srv/apps/ crontab -e
- 切换权限 sudo su - gmuser
- source /root/anaconda3/bin/activate
- 创建虚拟环境 conda activate crawler_env/conda deactivate
- 抓取程序 nohup python /srv/apps/crawler/crawler_sys/framework/update_data_in_target_releasers_multi_process_by_date_from_redis.py > /data/log/fect_task.log &
- 写入抓取url程序 python /srv/apps/crawler/crawler_sys/framework/write_releasers_to_redis.py -p weibo -d 1 -proxies 2
- ##搜索页爬虫 pass
数据周报
- 切换权限 sudo su - gmuser
- source /root/anaconda3/bin/activate
- python crawler/crawler_sys/utils/get_query_result.py