几次调用后，urlib2.urlopen通过代理失败
编辑：经过多次摆弄后，似乎urlgrabber在urllib2失败的地方成功，即使告诉它在每个文件后关闭连接。似乎urllib2处理代理的方式可能有问题，或者我使用它的方式！
无论如何，这是在循环中检索文件的最简单的代码：
import urlgrabber
for i in range(1, 100):
url = "http://www.iana.org/domains/example/"
urlgrabber.urlgrab(url, proxies={'http':'http://<user>:<password>@<proxy url>:<proxy port>'}, keepalive=1, close_connection=1, throttle=0)
import urllib2
passmgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
passmgr.add_password(None, '<proxy url>:<proxy port>', '<my user name>', '<my password>')
authinfo = urllib2.ProxyBasicAuthHandler(passmgr)
proxy_support = urllib2.ProxyHandler({"http" : "<proxy http address>"})
opener = urllib2.build_opener(authinfo, proxy_support)
urllib2.install_opener(opener)
for i in range(100):
with open("e:/tmp/images/tst{}.htm".format(i), "w") as outfile:
f = urllib2.urlopen("http://www.iana.org/domains/example/")
outfile.write(f.read())
没有找到相关结果
已邀请:
2 个回复
杭难插
import urllib2 from keepalive import HTTPHandler keepalive_handler = HTTPHandler() opener = urllib2.build_opener(keepalive_handler) urllib2.install_opener(opener) fo = urllib2.urlopen('http://www.python.org')泉秘胁