R.py, a small subset of Python Requests

Hacker News

Share card

R.py, a small subset of Python Requests

i revisited the work that i started in 2022 re: writing a subset of Requests but using the standard library: https://github.com/gabrielsroka/r back then, i tried using urllib.request (from the standard library, not urllib3) but it lacks what Requests/urllib3 has -- connection pools and keep alive [or that's where i thought the magic was] -- so my code ran much slower. it turns out that urllib.request uses http.client, but it closes the connection. so by using http.client directly, i can keep the connection open [that's where the real magic is]. now my code runs as fast as Requests/urllib3, but in 5 lines of code instead of 4,000-15,000+ moral of the story: RTFM over and over and over again. """Fetch users from the Okta API and paginate.""" import http.client import json import re import urllib.parse # Set these: host = 'domain.okta.com' token = 'xxx' url = '/api/v1/users?' + urllib.parse.urlencode({'filter': 'profile.lastName eq "Doe"'}) headers = {'authorization': 'SSWS ' + token} conn = http.client.HTTPSConnection(host) while url: conn.request('GET', url, headers=headers) res = conn.getresponse() for user in json.load(res): print(user['id']) links = [link for link in res.headers.get_all('link') if 'rel="next"' in link] url = re.search('<https://[^/]+(.+)>', links[0]).group(1) if links else None https://docs.python.org/3/library/urllib.request.html https://docs.python.org/3/library/http.client.html