python3 学习（2）：在网站地图爬虫时的cannot use a string pattern on a bytes-like object 问题的解决方法，,python3.6.

python3 学习（2）：在网站地图爬虫时的cannot use a string pattern on a bytes-like object 问题的解决方法，,python3.6.

python3.6.5 + pycharm

注意：

一、python3里的 urllib2 已经没有了，改为了urllbi.request，因此，直接导入import urllib.request即可。

二、必须对正则表达式里的引用变量进行格式变换.decode(‘utf-8‘)，否则会报错说 不能在一个字节类的对象上使用字符串格式。

如下代码所示。

### 一、 网站地图爬虫，控制用户代理的设定，可捕获异常、重试下载并设置用户代理。wswp: web scraping with pythonimport urllib.request  ## -- written by LiSongbodef Rocky_dnload(url,user_agent=‘wswp‘,num_retries = 2):    print(‘Downloading:‘,url)    LiSongbo_he={‘User-agent‘:user_agent}    request = urllib.request.Request(url, headers=LiSongbo_he)    try:  ## -- written by LiSongbohtml = urllib.request.urlopen(request).read()    except urllib.request.URLError as e:  ## -- written by LiSongboprint(‘Download error:‘,e.reason)        html = None        if num_retries > 0:  ## -- written by LiSongboif hasattr(e,‘code‘) and 500 <= e.code < 600:                return Rocky_dnload(url,user_agent,num_retries-1) ## retry 5xx HTTP errorsreturn htmlimport re  ## -- written by LiSongbodef Rocky_crawl_sitemap(url):  ## -- written by LiSongbositemap = Rocky_dnload(url)  ## download the sitmap file    # sitemap = sitemap.decode(‘utf-8‘) ## must add this .links = re.findall(‘<loc>(.*?)</loc>‘, sitemap)  ## extract the sitemap links from flag locfor link in links:  ## download each linkhtml = Rocky_dnload(link)  ## crape html hereRocky_crawl_sitemap(‘http://example.webscraping.com/sitemap.xml‘)

运行结果报错：

Downloading: http://example.webscraping.com/sitemap.xml
Traceback (most recent call last):
File "C:/Users/klooa/my_env/book9/test.py", line 25, in <module>
Rocky_crawl_sitemap(‘http://example.webscraping.com/sitemap.xml‘)
File "C:/Users/klooa/my_env/book9/test.py", line 22, in Rocky_crawl_sitemap
links = re.findall(‘<loc>(.*?)</loc>‘, sitemap) ## extract the sitemap links from flag loc
File "C:\Users\klooa\AppData\Local\Programs\Python\Python36\lib\re.py", line 222, in findall
return _compile(pattern, flags).findall(string)
TypeError: cannot use a string pattern on a bytes-like object

必须在 sitemap的下一行加上

sitemap = sitemap.decode(‘utf-8‘)

修改后的运行结果为：

Downloading: http://example.webscraping.com/sitemap.xml
Downloading: http://example.webscraping.com/places/default/view/Afghanistan-1
Downloading: http://example.webscraping.com/places/default/view/Aland-Islands-2
Downloading: http://example.webscraping.com/places/default/view/Albania-3
Downloading: http://example.webscraping.com/places/default/view/Algeria-4
Downloading: http://example.webscraping.com/places/default/view/American-Samoa-5
Downloading: http://example.webscraping.com/places/default/view/Andorra-6
Downloading: http://example.webscraping.com/places/default/view/Angola-7
Downloading: http://example.webscraping.com/places/default/view/Anguilla-8
Downloading: http://example.webscraping.com/places/default/view/Antarctica-9
Downloading: http://example.webscraping.com/places/default/view/Antigua-and-Barbuda-10
Downloading: http://example.webscraping.com/places/default/view/Argentina-11
Downloading: http://example.webscraping.com/places/default/view/Armenia-12
Download error: TOO MANY REQUESTS
Downloading: http://example.webscraping.com/places/default/view/Aruba-13

Download error: TOO MANY REQUESTS

……

## -- written by LiSongbo

python3 学习（2）：在网站地图爬虫时的cannot use a string pattern on a bytes-like object 问题的解决方法