For faster navigation, this Iframe is preloading the Wikiwand page for
网页抓取.
网页抓取(英語:web scraping)是一种从网页上获取页面内容的计算机软件技术。通常透過软件使用低级别的超文本传输协议模仿人类的正常访问。
网页抓取和网页索引极其相似,其中网页索引指的是大多数搜索引擎采用使用的机器人或网络爬虫等技术。与此相反,网页抓取更侧重于转换网络上非结构化数据(常见的是HTML格式)成为能在一个中央数据库和电子表格中储存和分析的结构化数据。网页抓取也涉及到网络自动化,它利用计算机软件模拟了人的浏览。网页抓取的用途包括在线的价格比较,联系人抓取,气象数据监测,网页变化检测,科研,混搭和Web数据集成。
技术层面
网络抓取用于自动化获取万维网上的信息
- 人工复制与粘贴:最好的网页抓取技术也比不上人类的手工复制与粘贴,尤其是在某些网站采取技术手段禁止自动化网页抓取的情况下,人工的复制与粘贴就成了唯一的解决方案。
- 文本搜索与正则表达式:文本搜索并且配合正则表达式可以有效的从页面上提取需要的内容。在基于UNIX的系统上可以使用grep,在其他平台或其他编程语言(例如Perl,Python)中也有相应的命令或语法。
- 基于HTTP编程:无论是静态网页还是动态网页均可以通过发送HTTP请求给服务器来获得,所以可以通过直接进行socket编程来实现。
- HTML语法分析器:很多网站都是使用数据库来存储他们的数据,用户访问的时候再通过程序自动按照指定的格式生成,由于生成的这些网页都采用了相同的的格式或者模板等,所以可以通过对获取到的HTML页面使用语法分析器进行语法分析,然后就可以使用HTML标签来提取需要的内容。使用HTML语法分析器同文本搜索与正则表达式相比较程序更加的健壮,也免于构造复杂的正则表达式。
{{bottomLinkPreText}}
{{bottomLinkText}}
This page is based on a Wikipedia article written by
contributors (read/edit).
Text is available under the
CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.
{{current.index+1}} of {{items.length}}
Thanks for reporting this video!
This browser is not supported by Wikiwand :(
Wikiwand requires a browser with modern capabilities in order to provide you with the best reading experience.
Please download and use one of the following browsers:
An extension you use may be preventing Wikiwand articles from loading properly.
If you're using HTTPS Everywhere or you're unable to access any article on Wikiwand, please consider switching to HTTPS (https://www.wikiwand.com).
An extension you use may be preventing Wikiwand articles from loading properly.
If you are using an Ad-Blocker, it might have mistakenly blocked our content.
You will need to temporarily disable your Ad-blocker to view this page.
✕
This article was just edited, click to reload
Please click Add in the dialog above
Please click Allow in the top-left corner,
then click Install Now in the dialog
Please click Open in the download dialog,
then click Install
Please click the "Downloads" icon in the Safari toolbar, open the first download in the list,
then click Install
{{::$root.activation.text}}
Follow Us
Don't forget to rate us
Oh no, there's been an error
Please help us solve this error by emailing us at
support@wikiwand.com
Let us know what you've done that caused this error, what browser you're using, and whether you have any special extensions/add-ons installed.
Thank you!