0
100% Satisfaction Guarantee
Amazing Value
Fast Shipping
Expert Customer Service
网络爬虫案例教程(Python·微课视频版)
$39.33
从实战出发,适合初学者,紧跟 章节案例,理清代码实现,很快掌握编写Python爬虫程序。
本书主要以纯案例驱动的形式,分别使用和结合使用requests库、selenium库,从互联网上收集数据。主要内容包括收集静态网页数据,网页解析,JSON数据,JSON解析,多级翻页,cookie和session,验证登录,多线程,数据的存储,收集到的数据包括静态网页数据、json、图片、音频、视频等数据,并对获取的数据进行清洗。适合Python初学者,也适合研究Python的广大科研人员、学者、工程技术人员。本教材采用纯的案例,和简单的注释,其他一些先修的知识是在其他课程有开设。
韩莹,防灾科技学院教师,讲授“数据科学与大数据”课程,本教材采用纯的案例,和简单的注释,其他一些先修的知识是在其他课程有开设。
第1章绪论
1.1网络爬虫的基本概念
1.2网络爬虫的基本流程
1.2.1发起请求
1.2.2获取响应内容
1.2.3解析内容
1.2.4持久化保存数据
1.3网络爬虫的合法性问题
1.4反爬虫技术
1.4.1Useragent控制访问
1.4.2IP限制访问
1.4.3设置请求间隔
1.4.4通过参数加密和JavaScript脚本
1.4.5通过robots.txt来限制爬虫
1.5网络爬虫的预备知识
1.5.1统一资源定位器
1.5.2超文本传输协议
1.5.3超文本标记语言
1.6开发语言和开发环境
1.6.1开发语言
1.6.2第三方请求库
1.6.3开发工具
第2章Requests库
2.1安装Requests库
2.2Requests库发送请求
2.3查看响应内容
2.3.1查看响应状态码
2.3.2查看响应的文本信息
2.3.3解决乱码问题
2.3.4二进制码响应内容
2.3.5JSON响应内容
2.4定制请求头部Headers
2.5Chrome浏览器开发者工具面板
2.5.1打开开发者工具面板
2.5.2Elements面板
2.5.3Network面板
2.6GET请求单个网页的爬取案例
2.6.1不带参数的GET请求
2.6.2携带参数的GET请求
第3章JSON数据爬取
3.1Ajax
3.1.1Ajax技术
3.1.2分析数据来源
3.2JSON
3.2.1JSON语法规则
3.2.2访问JSON数据
3.2.3JSON文件读写操作
3.2.4JSON数据校验和格式化
3.3Ajax异步动态加载的数据爬虫
3.3.1带参数的POST请求爬虫
3.3.2多个网页多链接GET请求爬虫综合案例
3.4POST请求的两种参数格式
3.4.1Form Data类型
3.4.2Request Payload类型
第4章XPath解析及网页数据爬取
4.1XPath简介及安装
4.2XPath节点
4.2.1基本值节点
4.2.2节点关系
4.3XPath语法
4.3.1选取节点语法
4.3.2谓语
4.3.3选取未知节点
4.3.4选取若干路径
4.3.5初步使用XPath案例
4.4XPath表达式
4.4.1定位XPath搜索框
4.4.2在网页上写XPath表达式
4.5爬取HTML文档数据案例
4.6爬取多页HTML文档数据案例
4.6.1翻页在参数里
4.6.2翻页在URL中
4.7图片爬虫案例
4.7.1单张图片爬取
4.7.2多页多幅图片爬虫案例
4.7.3多类多页多图爬虫案例
第5章IP代理
5.1IP代理的作用
5.2IP代理使用方法
5.3搭建IP池
5.3.1获取单页IP
5.3.2获取多页IP
5.3.3检测IP有效性
5.3.4建立IP池
5.4付费IP代理使用
第6章Selenium库
6.1Selenium安装及环境配置
6.1.1Selenium安装
6.1.2环境配置
6.1.3环境测试
6.2Selenium简单使用及配置
6.2.1打开网页
6.2.2规避伪装机制
6.2.3常见的配置项
6.3Selenium的元素定位操作
6.3.1查看页面元素
6.3.2通过ID定位元素
6.3.3通过name定位元素
6.3.4通过class定位元素
6.3.5通过tag定位元素
6.3.6通过link定位元素
6.3.7通过partial_link定位元素
6.3.8通过XPath定位元素
6.3.9通过CSS定位元素
6.3.10通过By定位元素
6.4Selenium等待机制
6.4.1固定等待
6.4.2隐式等待
6.4.3显式等待WebDriverWait
6.5Selenium控制浏览器
6.5.1浏览器的常见操作
6.5.2不同窗口之间切换
6.5.3鼠标事件
6.5.4键盘事件
6.5.5定位Frame/IFrame
6.5.6页面下拉
6.5.7窗口截图
6.5.8文件上传
6.6Selenium爬虫案例
6.6.1单页爬取案例
6.6.2多页爬取案例
第7章Requests与Selenium结合使用
7.1Selenium模拟登录
7.1.1Selenium程序模拟登录
7.1.2手动输入数据模拟登录
7.2Cookie与Session机制
7.2.1Cookie机制
7.2.2Session机制
7.3Requests、Cookie、Selenium结合使用
7.4Selenium和Requests结合下载音乐
7.4.1单首音乐下载
7.4.2多首音乐下载
第8章异步爬虫
8.1基本概念
8.2串行下载多个视频
8.3使用线程池下载多个视频
8.3.1Multiprocessing
8.3.2Threading
8.4使用协程下载多个视频
第9章正则表达式
9.1正则函数
9.1.1re.match函数
9.1.2re.search函数
9.1.3re.sub函数
9.1.4re.compile函数
9.1.5re.findall函数
9.1.6re.finditer函数
9.1.7re.split函数
9.2正则表达式模式及实例
9.3正则表达式实例
9.3.1匹配字符串
9.3.2匹配字符组
9.3.3区间匹配
9.3.4特殊字符匹配
9.3.5取反
9.3.6快捷匹配数字和字符
9.3.7匹配空白字符
9.3.8单词边界
9.3.9快捷方式取反
9.3.10开始和结束
9.3.11匹配任意字符
9.3.12可选字符
9.3.13重复
9.3.14重复区间
9.3.15开闭区间
9.4正则表达式进阶
9.4.1分组
9.4.2或者条件
9.4.3分组的回溯引用
9.4.4断言
第10章数据清洗
10.1数据分析流程
10.2数据清洗的概念及流程
10.3数据清洗常用方法
10.3.1读取数据
10.3.2初步探索数据
10.3.3简单处理数据
10.3.4重复值处理
10.3.5异常值处理
10.3.6处理缺失值
10.3.7爬取数据
10.3.8增加特征值
10.3.9格式与内容清洗
10.3.10数据持久化保存
第11章综合爬虫案例
11.1数据爬取
11.1.1单线程爬取“前程无忧”
11.1.2多线程爬取“英才网”
11.1.3多线程爬取“前程无忧”
11.2简单数据清洗
11.2.1导入库
11.2.2初识数据
11.2.3简单数据处理
11.2.4处理重复值
11.2.5处理空值
11.2.6字段内容清洗
11.2.7提取并增加特征值
11.2.8处理异常值
11.2.9数据保存
参考文献
现阶段,随着大数据和人工智能技术的研究不断深入发展,网络数据采集与清洗越来越具有较为广泛的应用范围。本书是为“数据采集与清洗”课程编写的教材,其内容选取符合教学大纲要求,以案例驱动展开,兼顾内容的广度和深度,适用面广。本书的第1章主要阐述网络爬虫的基本概念、基本流程、爬虫合法性问题、反爬虫技术以及爬虫的预备知识。第2~9章主要讲解爬虫的各种技术,分别讨论Requests库、JSON数据爬取、XPath解析及网页数据爬取、IP代理、Selenium库、Selenium与Requests结合使用、异步爬虫、正则表达式等基本技术的实现; 爬取的内容包括HTML文档、JSON数据、图片、音频、视频,以及这些类型数据的持久化保存。第10章讲解数据的简单清洗。第11章讲解一个综合案例,结合Requests和Selenium,分别采用单线程和多线程实现对不同网站的数据爬取,并进行相应的数据清洗操作。全书采用Python为主要描述语言。Python是一种面向对象的高级通用脚本编程语言,其语法简洁,简单易懂。相比其他语言,Python更容易配置,在字符处理方面灵活,并且在爬虫程序开发中具有先天的优势,是开发者的编程语言。Python初被用于编写Shell(自动化脚本),伴随着版本不断更新以及语言新功能的加入,其作为爬虫编程语言优势更加突出。因此,越来越多的开发者选择Python用于大型爬虫项目开发。Python自带有多种爬取模板,可以利用Requests和Selenium第三方库模拟人工浏览器访问的行为,实现起来便捷; 爬虫程序爬取网页之后,需要对网页进行诸如过滤标签、提取文本等处理。Python拥有简短的文档处理功能,能够用简短代码处理绝大部分文档。从课程性质上来讲,“数据采集与清洗”是一门专业选修课,侧重于应用。它的教学要求是: 理解互联网大数据采集的主要技术; 掌握各种典型爬虫的技术原理、技术框架、实现方法、主要开源包的使用; 理解对爬虫采集到的网页数据的处理方法及文本处理,并会使用Python进行技术实现。本书的学习过程通过案例驱动的方式展开,帮助读者贯穿爬虫、数据清洗的过程,培养读者掌握从互联网上采集数据的技术,能够独立完成数据采集和清洗工作,对培养学生的数据处理能力、信息分析与应用能力、信息表达能力具有重要作用,为后期的生产实习、毕业设计以及未来的工作奠定一定的实践基础。本书内容以实战为主,适合高等院校相关专业的学生阅读,可以作为数据科学与大数据专业的本科或专科教材,也可以作为信息类相关专业的选修教材,也适合Python培训机构作为实训教材。讲课学时可设置为30~40学时。本书文字通俗,简单易懂,便于自学,也可供从事大数据处理等工作的科技人员参考。只需要掌握Python程序设计便可以学习本书。
配套资源为了方便教学,本书配有微课视频、教学课件、源代码。(1) 获取微课视频方式: 读者可以先扫描本书封底的文泉云盘防盗码,再扫描书中相应的视频二维码,即可观看教学视频。(2) 其他资源可先扫描本书封底的文泉云盘防盗码,再扫描下方二维码,即可获取。
教学课件
源代码
Shipping Overview:
• Shipping: Standard Domestic Shipping within the United States charges USD 4.99. Standard International Shipping from United Kingdom, Germany and Japan to the United States charges 14.99.
• Order Processing: Please allow 1-2 business days for order processing and preparation before shipment.
• Domestic Shipping: Orders within the U.S. are shipped via USPS or FedEx, depending on the origin of the product. The average transit time is 3-7 business days.
• International Shipping: Currently, we only ship within the USA.
• Tracking Information: Every order is trackable. You will receive a tracking number once your order has been shipped. Products may be shipped from various global fulfillment centers.
Shipping Delays:
Please note that shipping times may vary due to factors beyond our control, such as weather conditions, natural disasters, or peak holiday periods. While we strive to ensure timely delivery, the exact arrival time cannot be guaranteed and is managed by the shipping carrier.
Shipping Options:
Standard Delivery: Most orders are shipped within 3-7 business days. Larger items may utilize LTL shipping for safe handling.
Handling Time: We handle shipments on business days (Monday - Friday), with a preparation time of 1-2 days.
Additional Charges: Some items require additional shipping charges due to their size, weight, or special handling. These charges are specified on the product pages and are not eligible for shipping discounts.
Exclusions: Gift cards, packaging, taxes, and prior purchases do not count toward the minimum purchase requirement for free shipping. This offer is valid only for shipments to U.S. addresses, including Puerto Rico.
Delivery Details:
Estimates: Standard shipping within the US typically takes 3-7 business days. These are estimates and not guarantees.
Shipping Restrictions: We ship to all 50 states, Washington, DC, U.S. territories, and APO/FPO/DPO addresses. Shipping options vary based on the delivery address.
Remote Areas: Shipments to remote areas may incur additional charges or require pickup from a nearby shipping partner’s location.
Shipping Confirmation:
You will receive a shipping confirmation email with a tracking number as soon as your order is dispatched. If you do not receive this email immediately, please be assured that your items will arrive within the estimated delivery window provided at checkout.
Order Modifications:
If you need to cancel or modify your order, please contact our customer support immediately.
Issues with Delivery:
If your order shows as delivered but you have not received it, please contact the shipping carrier directly to resolve the issue. For persistent problems, contact our customer service at cs@everymarket.com.
Customer Support:
Our team is available 24/7 to assist you with any questions or concerns regarding your order. We are committed to ensuring a smooth shopping experience.
Return & Refund Policy Overview
Please review our return and refund policies below to ensure a smooth transaction process.
Return Policy
Duration: You have 30 days from receiving your item to initiate a return.
Condition: Items must be unworn, unwashed, with original tags and packaging intact.
Shipping Costs: Customers are responsible for return shipping costs.
Packaging: Ensure returned items are well-packaged to avoid damage during transit.
Tracking: Use a trackable and insured shipping method as we are not liable for items lost or damaged on return.
Initiating a Return: Contact us at cs@everymarket.com to start your return. We will provide a return shipping label and instructions upon approval. Returns without prior approval will not be accepted.
How to Return
Method: Returns must be sent back by mail to the address provided in the return instructions.
Return Label
Defective Products: Return labels are provided for defective items.
Non-Defective Returns: Customers are responsible for obtaining their return shipping label.
Product Conditions for Return
Eligible Products: Returns are only accepted for items in new condition.
Nonreturnable Items
Certain items are not eligible for return:
Electronic devices after 30 days (e.g., computers, laptops, Kindles)
Gift cards, prepaid game cards
Perishable goods, live insects, some jewelry, some health and personal care items
Customized or personalized products
Items with special shipping restrictions
Refund Policy
Window: Eligible products may be returned within 30 days of delivery for a refund.
Refund Method: Refunds are processed to the original payment method or as store credit for items purchased with gift cards.
Processing Time: Refunds are processed within 3-5 business days after we receive the return; please allow additional time for shipping and bank processing.
Claims
Inspect your order upon arrival and report any defects, damages, or incorrect items immediately to allow us to address the issue. For claims, contact our support team with details of the issue.
Exchange Policy
For the quickest service, return your original item and place a new order for the desired product once your return is accepted.
Return Address
EveryMarket Customer Service 2101 E Terra Ln, O'Fallon, MO 63366
Customer Support
Available 24/7 for any questions or assistance needed:
Phone: +1 636-312-5925
Email: cs@everymarket.com
Oops!
Sorry, it looks like some products are not available in selected quantity.
OK