Koa+Puppeteer爬蟲教程頁面設計

當我使用Koa作為web服務器，Puppeteer作為爬蟲工具來編寫一個簡單的爬蟲教程時，發生了戲劇性的一幕。

下面我將創建一個完整的Koa + Puppeteer爬蟲教程頁面，包含代碼示例、執行演示和詳細說明。

在這里插入圖片描述

設計思路

左側為教程內容區域
右側為實時爬蟲演示區域
使用卡片式布局展示代碼示例
添加交互式爬蟲演示功能

實現方案

<!DOCTYPE html>
<html lang="zh-CN">
<head><meta charset="UTF-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><title>Koa + Puppeteer 爬蟲教程</title><link href="https://fonts.googleapis.com/css2?family=Roboto:wght@300;400;500;700&family=Source+Code+Pro&display=swap" rel="stylesheet"><link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0/css/all.min.css"><script src="https://cdn.jsdelivr.net/npm/chart.js"></script><style>:root {--primary: #4361ee;--secondary: #3f37c9;--success: #4cc9f0;--dark: #2b2d42;--light: #f8f9fa;--danger: #e63946;--gray: #6c757d;--card-shadow: 0 4px 20px rgba(0, 0, 0, 0.1);}* {margin: 0;padding: 0;box-sizing: border-box;}body {font-family: 'Roboto', sans-serif;background: linear-gradient(135deg, #f5f7fa 0%, #e4edf5 100%);color: #333;line-height: 1.6;min-height: 100vh;padding: 20px;}.container {max-width: 1400px;margin: 0 auto;}header {text-align: center;padding: 40px 0;background: linear-gradient(120deg, var(--primary), var(--secondary));border-radius: 15px;color: white;margin-bottom: 30px;box-shadow: var(--card-shadow);}header h1 {font-size: 2.8rem;margin-bottom: 15px;}header p {font-size: 1.2rem;max-width: 800px;margin: 0 auto;opacity: 0.9;}.logo {font-size: 3rem;margin-bottom: 20px;display: flex;justify-content: center;gap: 30px;}.logo i {background: rgba(255, 255, 255, 0.2);width: 80px;height: 80px;display: flex;align-items: center;justify-content: center;border-radius: 50%;}.content {display: flex;gap: 30px;margin-bottom: 40px;}.tutorial {flex: 3;background: white;border-radius: 15px;padding: 30px;box-shadow: var(--card-shadow);}.demo {flex: 2;background: white;border-radius: 15px;padding: 30px;box-shadow: var(--card-shadow);display: flex;flex-direction: column;}h2 {color: var(--secondary);margin-bottom: 25px;padding-bottom: 10px;border-bottom: 2px solid var(--success);font-size: 1.8rem;}h3 {color: var(--primary);margin: 25px 0 15px;font-size: 1.4rem;}.step {background: #f8f9ff;padding: 20px;border-radius: 10px;margin-bottom: 25px;border-left: 4px solid var(--primary);}.step-number {display: inline-block;background: var(--primary);color: white;width: 32px;height: 32px;text-align: center;line-height: 32px;border-radius: 50%;margin-right: 10px;font-weight: bold;}pre {background: #2b2d42;color: #f8f9fa;padding: 20px;border-radius: 8px;overflow-x: auto;margin: 20px 0;font-family: 'Source Code Pro', monospace;font-size: 0.95rem;}code {font-family: 'Source Code Pro', monospace;}.code-header {display: flex;justify-content: space-between;align-items: center;background: #1a1c2e;padding: 10px 15px;border-radius: 8px 8px 0 0;margin-top: 20px;color: white;}.code-header span {font-size: 0.9rem;}.copy-btn {background: var(--success);border: none;color: white;padding: 5px 10px;border-radius: 4px;cursor: pointer;font-size: 0.85rem;transition: all 0.3s;}.copy-btn:hover {background: #3ab7d8;}.demo-controls {display: flex;gap: 15px;margin-bottom: 20px;}.url-input {flex: 1;padding: 12px 15px;border: 2px solid #ddd;border-radius: 8px;font-size: 1rem;transition: border 0.3s;}.url-input:focus {border-color: var(--primary);outline: none;}.run-btn {background: var(--primary);color: white;border: none;padding: 12px 25px;border-radius: 8px;cursor: pointer;font-weight: 500;transition: all 0.3s;display: flex;align-items: center;gap: 8px;}.run-btn:hover {background: var(--secondary);transform: translateY(-2px);}.results {flex: 1;background: #f8f9ff;border-radius: 10px;padding: 20px;overflow-y: auto;border: 1px solid #e9ecef;}.result-item {padding: 15px;margin-bottom: 15px;background: white;border-radius: 8px;box-shadow: 0 2px 5px rgba(0,0,0,0.05);}.result-title {color: var(--primary);font-weight: 500;margin-bottom: 8px;}.result-link {color: var(--success);font-size: 0.9rem;display: block;margin-bottom: 10px;overflow: hidden;text-overflow: ellipsis;white-space: nowrap;}.result-snippet {color: var(--gray);font-size: 0.95rem;}.features {display: grid;grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));gap: 25px;margin-top: 40px;}.feature-card {background: white;border-radius: 15px;padding: 25px;box-shadow: var(--card-shadow);transition: transform 0.3s, box-shadow 0.3s;border-top: 4px solid var(--success);}.feature-card:hover {transform: translateY(-10px);box-shadow: 0 12px 30px rgba(0, 0, 0, 0.15);}.feature-icon {font-size: 2.5rem;color: var(--primary);margin-bottom: 20px;}.feature-card h3 {margin: 0 0 15px;font-size: 1.4rem;}.browser-window {background: #2b2d42;border-radius: 10px;overflow: hidden;margin: 25px 0;box-shadow: 0 10px 30px rgba(0,0,0,0.2);}.browser-header {height: 40px;background: #3a3d5d;display: flex;align-items: center;padding: 0 15px;}.browser-dot {width: 12px;height: 12px;border-radius: 50%;margin-right: 8px;}.dot-red { background: #ff5f56; }.dot-yellow { background: #ffbd2e; }.dot-green { background: #27c93f; }.browser-content {height: 300px;background: white;padding: 20px;overflow-y: auto;}.browser-content h4 {color: var(--primary);margin-bottom: 15px;}.browser-content ul {padding-left: 25px;}.browser-content li {margin-bottom: 10px;}.tip-box {background: #e3f2fd;border-left: 4px solid var(--primary);padding: 15px 20px;border-radius: 0 8px 8px 0;margin: 25px 0;}.tip-title {font-weight: bold;color: var(--primary);margin-bottom: 8px;display: flex;align-items: center;gap: 8px;}.footer {text-align: center;padding: 30px 0;color: var(--gray);font-size: 0.95rem;margin-top: 20px;}@media (max-width: 1000px) {.content {flex-direction: column;}.features {grid-template-columns: 1fr;}}</style>
</head>
<body><div class="container"><header><div class="logo"><i class="fab fa-node-js"></i><i class="fab fa-js"></i><i class="fas fa-spider"></i></div><h1>Koa + Puppeteer 爬蟲教程</h1><p>學習如何使用Koa.js框架和Puppeteer庫構建強大的網絡爬蟲</p></header><div class="content"><div class="tutorial"><h2>Koa + Puppeteer 爬蟲指南</h2><div class="step"><h3><span class="step-number">1</span> Puppeteer 簡介</h3><p>Puppeteer 是一個 Node.js 庫，它提供了一個高級 API 來通過 DevTools 協議控制 Chromium 或 Chrome。Puppeteer 默認以 headless 模式運行，但是可以配置為有界面模式運行。</p><p>主要功能：</p><ul><li>生成頁面截圖和PDF</li><li>爬取單頁應用(SPA)并生成預渲染內容</li><li>自動化表單提交、UI測試、鍵盤輸入等</li><li>創建最新的自動化測試環境</li></ul></div><div class="step"><h3><span class="step-number">2</span> Koa.js 簡介</h3><p>Koa 是由 Express 原班人馬打造的下一代 Node.js Web 框架，旨在為 Web 應用和 API 提供更小、更富有表現力、更健壯的基石。</p><p>主要特點：</p><ul><li>輕量級，無捆綁任何中間件</li><li>使用 async/await 語法，優雅地處理異步</li><li>錯誤處理更友好</li><li>核心代碼簡潔，易于擴展</li></ul></div><div class="step"><h3><span class="step-number">3</span> 項目初始化</h3><p>創建項目并安裝所需依賴：</p><div class="code-header"><span>Terminal</span><button class="copy-btn">復制</button></div><pre><code># 創建項目目錄
mkdir koa-puppeteer-crawler
cd koa-puppeteer-crawler# 初始化項目
npm init -y# 安裝依賴
npm install koa @koa/router puppeteer</code></pre></div><div class="step"><h3><span class="step-number">4</span> 創建基本爬蟲服務</h3><p>創建 <code>index.js</code> 文件，設置 Koa 服務器和爬蟲路由：</p><div class="code-header"><span>index.js</span><button class="copy-btn">復制</button></div><pre><code>const Koa = require('koa');
const Router = require('@koa/router');
const puppeteer = require('puppeteer');const app = new Koa();
const router = new Router();// 爬蟲路由
router.get('/crawl', async (ctx) => {// 從查詢參數獲取URLconst url = ctx.query.url || 'https://example.com';// 啟動瀏覽器const browser = await puppeteer.launch({headless: true,args: ['--no-sandbox', '--disable-setuid-sandbox']});try {const page = await browser.newPage();await page.goto(url, { waitUntil: 'networkidle2', timeout: 30000 });// 獲取頁面數據const pageData = await page.evaluate(() => {return {title: document.title,content: document.body.innerText.substring(0, 1000) + '...',links: Array.from(document.querySelectorAll('a')).map(a => a.href)};});ctx.body = {success: true,data: pageData};} catch (error) {ctx.status = 500;ctx.body = {success: false,message: error.message};} finally {await browser.close();}
});app.use(router.routes());
app.use(router.allowedMethods());const PORT = process.env.PORT || 3000;
app.listen(PORT, () => {console.log(`Server running on http://localhost:${PORT}`);
});</code></pre></div><div class="step"><h3><span class="step-number">5</span> 運行爬蟲服務</h3><p>啟動服務器：</p><div class="code-header"><span>Terminal</span><button class="copy-btn">復制</button></div><pre><code>node index.js</code></pre><p>訪問爬蟲接口：</p><pre><code>http://localhost:3000/crawl?url=https://example.com</code></pre></div><div class="step"><h3><span class="step-number">6</span> 高級爬蟲技巧</h3><p>處理動態加載內容：</p><pre><code>// 等待特定元素出現
await page.waitForSelector('.results-container', { timeout: 5000 });// 滾動頁面加載更多內容
await page.evaluate(() => {window.scrollBy(0, window.innerHeight);
});// 點擊"加載更多"按鈕
await page.click('.load-more-button');</code></pre><p>處理登錄認證：</p><pre><code>// 輸入用戶名和密碼
await page.type('#username', 'myuser');
await page.type('#password', 'mypassword');// 提交表單
await page.click('#login-button');
await page.waitForNavigation();</code></pre></div><div class="tip-box"><div class="tip-title"><i class="fas fa-lightbulb"></i><span>最佳實踐建議</span></div><ul><li>使用 <code>page.setUserAgent()</code> 設置合理的用戶代理</li><li>使用 <code>page.setViewport()</code> 設置視口大小</li><li>添加請求延遲避免被封禁</li><li>使用代理IP處理高頻請求</li><li>合理使用 <code>waitForSelector</code> 和 <code>waitForNavigation</code></li><li>使用try-catch處理異常</li></ul></div></div><div class="demo"><h2>爬蟲演示</h2><div class="browser-window"><div class="browser-header"><div class="browser-dot dot-red"></div><div class="browser-dot dot-yellow"></div><div class="browser-dot dot-green"></div></div><div class="browser-content"><h4>Puppeteer 爬蟲功能演示</h4><p>在右側輸入URL，點擊"運行爬蟲"查看效果：</p><ul><li>爬取頁面標題和內容</li><li>提取所有鏈接</li><li>處理JavaScript渲染的頁面</li><li>模擬用戶交互</li></ul><p>示例URL：</p><ul><li>https://example.com</li><li>https://news.ycombinator.com</li><li>https://quotes.toscrape.com</li></ul></div></div><div class="demo-controls"><input type="text" class="url-input" id="crawlUrl" placeholder="輸入要爬取的URL (例如: https://example.com)" value="https://example.com"><button class="run-btn" id="runCrawler"><i class="fas fa-play"></i><span>運行爬蟲</span></button></div><div class="results" id="results"><p>爬取結果將顯示在這里...</p></div></div></div><div class="features"><div class="feature-card"><div class="feature-icon"><i class="fas fa-bolt"></i></div><h3>高效靈活</h3><p>Koa的輕量級架構結合Puppeteer的強大瀏覽器控制能力，可以構建高效靈活的爬蟲系統，處理各種復雜的網站結構。</p></div><div class="feature-card"><div class="feature-icon"><i class="fas fa-laptop-code"></i></div><h3>JavaScript渲染支持</h3><p>Puppeteer可以完整渲染JavaScript生成的內容，輕松爬取單頁應用(SPA)和動態加載的網站，這是傳統爬蟲難以實現的。</p></div><div class="feature-card"><div class="feature-icon"><i class="fas fa-shield-alt"></i></div><h3>自動化測試</h3><p>除了爬蟲功能，Puppeteer還可用于自動化測試、性能監控和UI測試，為項目提供全方位的自動化支持。</p></div></div><div class="footer"><p>? 2023 Koa + Puppeteer 爬蟲教程 | 本教程僅用于學習目的，請遵守目標網站的robots.txt和法律法規</p></div></div><script>// 模擬爬蟲結果document.getElementById('runCrawler').addEventListener('click', function() {const urlInput = document.getElementById('crawlUrl').value;const resultsDiv = document.getElementById('results');if (!urlInput) {alert('請輸入URL');return;}// 顯示加載狀態resultsDiv.innerHTML = '<p class="result-item"><i class="fas fa-spinner fa-spin"></i> 正在爬取數據，請稍候...</p>';// 模擬網絡請求延遲setTimeout(() => {// 模擬爬取結果const mockData = {title: "示例域名 | " + new URL(urlInput).hostname,content: "這是一個示例網站。這段文本是從爬蟲返回的模擬內容。在實際應用中，這里將顯示從目標網站抓取的真實內容。",links: ["https://" + new URL(urlInput).hostname + "/about","https://" + new URL(urlInput).hostname + "/products","https://" + new URL(urlInput).hostname + "/contact"]};// 顯示結果let html = `<div class="result-item"><div class="result-title">${mockData.title}</div><div class="result-snippet">${mockData.content}</div></div><div class="result-item"><div class="result-title">提取到的鏈接 (${mockData.links.length})</div>`;mockData.links.forEach(link => {html += `<div class="result-link">${link}</div>`;});html += `</div>`;resultsDiv.innerHTML = html;}, 2000);});// 復制代碼功能document.querySelectorAll('.copy-btn').forEach(button => {button.addEventListener('click', function() {const codeBlock = this.parentElement.nextElementSibling;const textToCopy = codeBlock.textContent;navigator.clipboard.writeText(textToCopy).then(() => {const originalText = this.textContent;this.textContent = '已復制!';this.style.background = '#27c93f';setTimeout(() => {this.textContent = originalText;this.style.background = '';}, 1500);}).catch(err => {console.error('復制失敗:', err);alert('復制失敗，請手動復制代碼');});});});</script>
</body>
</html>