https://overkazaf.github.io/reverse_engineering/tags/%E7%A4%BE%E4%BA%A4%E5%AA%92%E4%BD%93/Android 与 Web 逆向工程实战手册Hugo 0.147.7 & FixIt v0.4.0-alpha.3-20251224074056-4fec1cdazh-CNoverkazaf@gmail.com (+5)overkazaf@gmail.com (+5)Sat, 26 Apr 2025 00:00:00 +0000https://overkazaf.github.io/reverse_engineering/web/06-engineering/distributed_scraping/Sat, 26 Apr 2025 00:00:00 +0000overkazaf@gmail.com (+5)https://overkazaf.github.io/reverse_engineering/web/06-engineering/distributed_scraping/<h1 class="heading-element" id="分布式爬虫架构"><span>分布式爬虫架构</span> <a href="#%e5%88%86%e5%b8%83%e5%bc%8f%e7%88%ac%e8%99%ab%e6%9e%b6%e6%9e%84" class="heading-mark"> <svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg> </a> </h1><h2 class="heading-element" id="概述"><span>概述</span> <a href="#%e6%a6%82%e8%bf%b0" class="heading-mark"> <svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg> </a> </h2><p>随着数据规模的不断扩大，单机爬虫在性能、稳定性和效率上遇到了瓶颈。分布式爬虫架构通过将爬取任务分散到多台机器上并行执行，可以大幅提升爬取效率，实现千万级甚至亿级的数据采集。</p>https://overkazaf.github.io/reverse_engineering/android/01-recipes/automation/proxy_pool_design/Sat, 17 Aug 2024 00:00:00 +0000overkazaf@gmail.com (+5)https://overkazaf.github.io/reverse_engineering/android/01-recipes/automation/proxy_pool_design/<h1 class="heading-element" id="代理池设计与-scrapy-集成"><span>代理池设计与 Scrapy 集成</span> <a href="#%e4%bb%a3%e7%90%86%e6%b1%a0%e8%ae%be%e8%ae%a1%e4%b8%8e-scrapy-%e9%9b%86%e6%88%90" class="heading-mark"> <svg class="octicon octicon-link" viewBox="0 0 16 16" version="1.1" width="16" height="16" aria-hidden="true"><path d="m7.775 3.275 1.25-1.25a3.5 3.5 0 1 1 4.95 4.95l-2.5 2.5a3.5 3.5 0 0 1-4.95 0 .751.751 0 0 1 .018-1.042.751.751 0 0 1 1.042-.018 1.998 1.998 0 0 0 2.83 0l2.5-2.5a2.002 2.002 0 0 0-2.83-2.83l-1.25 1.25a.751.751 0 0 1-1.042-.018.751.751 0 0 1-.018-1.042Zm-4.69 9.64a1.998 1.998 0 0 0 2.83 0l1.25-1.25a.751.751 0 0 1 1.042.018.751.751 0 0 1 .018 1.042l-1.25 1.25a3.5 3.5 0 1 1-4.95-4.95l2.5-2.5a3.5 3.5 0 0 1 4.95 0 .751.751 0 0 1-.018 1.042.751.751 0 0 1-1.042.018 1.998 1.998 0 0 0-2.83 0l-2.5 2.5a1.998 1.998 0 0 0 0 2.83Z"></path></svg> </a> </h1><blockquote> <p><strong>📚 前置知识</strong></p>