Mac中Python 3环境下安装scrapy的方法教程

站长资源 2024/11/25 佚名

2 0 1

前言

最近抽空想学习一下python的爬虫框架scrapy，在mac下安装的时候遇到了问题，逐一解决了问题，分享一下，话不多说了，来一起看看详细的介绍吧。

步骤如下：

1. 从官网下载最新版本Python 3.6.3（本地快速下载安装：https://www.jb51.net/softs/583651.html）

# 在Mac上Python3环境下安装scrapy

2. 安装 Python3

在终端输入python3出现下面的内容表示安装成功

"help", "copyright", "credits" or "license" for more information.
>

输入quit()退出编辑模式

3. 输入 pip install scrapy 执行 scrapy 安装

"/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/basecommand.py", line 215, in main
 status = self.run(options, args)
 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/commands/install.py", line 342, in run
 prefix=options.prefix_path,
 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_set.py", line 784, in install
 **kwargs
 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 851, in install
 self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/req/req_install.py", line 1064, in move_wheel_files
 isolated=self.isolated,
 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 345, in move_wheel_files
 clobber(source, lib_dir, True)
 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/wheel.py", line 316, in clobber
 ensure_dir(destdir)
 File "/Library/Python/2.7/site-packages/pip-9.0.1-py2.7.egg/pip/utils/__init__.py", line 83, in ensure_dir
 os.makedirs(path)
 File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 157, in makedirs
 mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/lxml'

出现 OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/lxml' 错误

4. 尝试重新安装lxml,执行 sudo pip install lxml

"htmlcode">

"htmlcode">

"/usr/local/bin/scrapy", line 7, in <module>
 from scrapy.cmdline import execute
 File "/Library/Python/2.7/site-packages/scrapy/cmdline.py", line 9, in <module>
 from scrapy.crawler import CrawlerProcess
 File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 7, in <module>
 from twisted.internet import reactor, defer
 File "/Library/Python/2.7/site-packages/twisted/internet/reactor.py", line 38, in <module>
 from twisted.internet import default
 File "/Library/Python/2.7/site-packages/twisted/internet/default.py", line 56, in <module>
 install = _getInstallFunction(platform)
 File "/Library/Python/2.7/site-packages/twisted/internet/default.py", line 50, in _getInstallFunction
 from twisted.internet.selectreactor import install
 File "/Library/Python/2.7/site-packages/twisted/internet/selectreactor.py", line 18, in <module>
 from twisted.internet import posixbase
 File "/Library/Python/2.7/site-packages/twisted/internet/posixbase.py", line 18, in <module>
 from twisted.internet import error, udp, tcp
 File "/Library/Python/2.7/site-packages/twisted/internet/tcp.py", line 28, in <module>
 from twisted.internet._newtls import (
 File "/Library/Python/2.7/site-packages/twisted/internet/_newtls.py", line 21, in <module>
 from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
 File "/Library/Python/2.7/site-packages/twisted/protocols/tls.py", line 63, in <module>
 from twisted.internet._sslverify import _setAcceptableProtocols
 File "/Library/Python/2.7/site-packages/twisted/internet/_sslverify.py", line 38, in <module>
 TLSVersion.TLSv1_1: SSL.OP_NO_TLSv1_1,
AttributeError: 'module' object has no attribute 'OP_NO_TLSv1_1'


需要更新 OpenSSL 库，执行 sudo pip install --upgrade pyopenssl


"PyPy" (from cryptography>=1.9->pyopenssl)
 Downloading cffi-1.11.2-cp27-cp27m-macosx_10_6_intel.whl (238kB)
 100% |████████████████████████████████| 245kB 2.2MB/s 
Collecting enum34; python_version < "3" (from cryptography>=1.9->pyopenssl)
 Downloading enum34-1.1.6-py2-none-any.whl
Collecting idna>=2.1 (from cryptography>=1.9->pyopenssl)
 Downloading idna-2.6-py2.py3-none-any.whl (56kB)
 100% |████████████████████████████████| 61kB 3.1MB/s 
Collecting asn1crypto>=0.21.0 (from cryptography>=1.9->pyopenssl)
 Downloading asn1crypto-0.23.0-py2.py3-none-any.whl (99kB)
 100% |████████████████████████████████| 102kB 2.7MB/s 
Collecting ipaddress; python_version < "3" (from cryptography>=1.9->pyopenssl)
 Downloading ipaddress-1.0.18-py2-none-any.whl
Collecting pycparser (from cffi>=1.7; platform_python_implementation != "PyPy"->cryptography>=1.9->pyopenssl)
 Downloading pycparser-2.18.tar.gz (245kB)
 100% |████████████████████████████████| 256kB 3.6MB/s 
Installing collected packages: pycparser, cffi, enum34, idna, asn1crypto, ipaddress, cryptography, pyopenssl
 Running setup.py install for pycparser ... done
 Found existing installation: pyOpenSSL 0.13.1
 DEPRECATION: Uninstalling a distutils installed project (pyopenssl) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
 Uninstalling pyOpenSSL-0.13.1:
 Successfully uninstalled pyOpenSSL-0.13.1
Successfully installed asn1crypto-0.23.0 cffi-1.11.2 cryptography-2.1.1 enum34-1.1.6 idna-2.6 ipaddress-1.0.18 pycparser-2.18 pyopenssl-17.3.0


更新 OpenSSL 成功，再次尝试执行 scrapy


"scrapy <command> -h" to see more info about a command


出现上面内容，表明安装成功。现在可以通过 scrapy 创建一个爬虫项目了
7. 进入到你项目的目录,执行 scrapy startproject firstscrapy创建 firstscrapy 爬虫项目


"text-align: center">


出现上面内容表明项目创建成功，但是使用的是2.7版本的Python怎么切换到3.6版本呢？
8. 使用 PyCharm IDE 打开刚才的项目，执行 command + , 打开偏好设置菜单，在Project里面选择 Projiect interpreter 来切换你需要依赖的Python库的版本，配置结束。

总结
以上就是这篇文章的全部内容了，希望本文的内容对大家的学习或者工作具有一定的参考学习价值，如果有疑问大家可以留言交流，谢谢大家对的支持。

                                
                                
                                    mac,scrapy,python3,python,scrapy,安装,python3,scrapy,安装