With the modern web, very few websites still used plain old unencrypted http protocol. It's a good news for end-user because https provides better privacy. For developers it could be hard to debug HTTPS exchanges because you cannot capture the network traffic with analyzer like tcpdump. In fact you could but you will only see the HTTPS session initialization not the underlying HTTP exchanges. Not enough to debug an HTTP client. To solve this problem, the only solution is to activate debugging from the client side.

Debugging using Python requests

Python requests is a nice http library, but there is no simple way to activate debugging. Internet provide many advice to activate debugging. It is always like this:

import http.client
# logging stuffs
http.client.HTTPConnection.debuglevel = 1
requests.get("https://myurl/path/myrsr")

In my opinion it is just a dirty hack that use an hidden feature of http.client, the low-level http library of Python. You should not use it because:

• official documentation says that http.client is normally not used directly
• the level of logging is not configurable
• debuglevel just activate some print statements. It does not use the logging system
• most annoying thing is that activating debuglevel is also printing form data. Just horrible if you are trying to debug POST requests. You terminal will be filled with tons of binary data :-)

A better solution

The solution is to use the headers provide by the requests object and adapt it to your need. Here is an example for the GET request:

import requests
import logging

logging.basicConfig(level=logging.DEBUG, format=' %(message)s')

logger = logging.getLogger('myclient')
logger.setLevel(logging.DEBUG)

def get_and_debug(url, **kw):
    " do the get and print debug info "
    r=requests.get(url, kw)
    for h, v in r.request.headers.items():
        logger.debug(" > {}: {}".format(h ,v))
    for h, v in r.headers.items():
        logger.debug(" < {}: {}".format( h ,v))
    logger.debug("Size of the response: {}".format( len(r.content) ))
    return r

get_and_debug("https://docs.python.org/3/_static/py.png")

The output is:

Starting new HTTPS connection (1): docs.python.org
 https://docs.python.org:443 "GET /3/_static/py.png HTTP/1.1" 200 695
  > User-Agent: python-requests/2.18.4
  > Accept-Encoding: gzip, deflate
  > Accept: */*
  > Connection: keep-alive
  < Connection: keep-alive
  < Content-Length: 695
  < Server: nginx
  < Content-Type: image/png
  < Last-Modified: Thu, 14 Feb 2019 11:40:31 GMT
  < ETag: "5c6553af-2b7"
  < X-Clacks-Overhead: GNU Terry Pratchett
  < Strict-Transport-Security: max-age=315360000; includeSubDomains; preload
  < Via: 1.1 varnish, 1.1 varnish
  < Accept-Ranges: bytes
  < Date: Wed, 29 Jan 2020 19:54:26 GMT
  < Age: 398453
  < X-Served-By: cache-jfk8144-JFK, cache-cdg20740-CDG
  < X-Cache: HIT, HIT
  < X-Cache-Hits: 649, 184
  < X-Timer: S1580327667.914627,VS0,VE0
 Size of the response: 695

You can use this method to wrap any top-level methods of requests you need to debug. If you still cannot debug HTTPS with this blog post, your last chance is to use website like neverssl :-).