Tutorial: Using Charles proxy to debug HTTP(S) communication between server and iOS apps

Abstract

In all the apps we have developed so far, we have witnessed the use of tightly coupled communication between the iOS app on the device and a remote web service. For example, an app might request a JSON encoded list of popular items from the web server that is displayed to the user in a nice table. No matter how much time is spent on defining and specifying communication protocol, during development (and sometimes even during production) there are moments when there are problems with these app vs. server interactions.

Especially when these problems occur on the device or during production, it can be hard to trace the problem without the proper tools. Using a good proxy all communication between app and server can be monitored, which makes tracing these issues much easier. In our tutorial we’ll focus on HTTP and HTTPS traffic only and we will be using the Charles proxy, which is available for Mac, Windows and Linux/UNIX. Knowledge on the HTTP protocol is expected.

Intercepting HTTP traffic

Install Charles. There is a free trial, but costs $50 if you want to get rid of the nagging splash screen.¬†Launch the app. It will immediately prompt you to ask for permission to configure the necessary network settings. This is needed in order for Charles to intercept network traffic. Choose “Grant Privileges” for the easy route.

By default, it will start recording HTTP traffic immediately, so you’ll see the table get filled with calls e.g. if you are browsing the web on your computer. You can try that for a start, for example by visiting google.com like I did here. On first start, the “Structure” tab will be selected. The Structure tab shows an expandable tree of hosts and paths that have been requested and intercepted during this Charles session. If you expand a visited hostname (e.g. www.google.com), you’ll get subpaths that have been visited, etc.

The other view is the “Sequence” view. Instead of presenting requests by hostname and path, it shows requests ordered by time. Personally, I usually use this view most of the times. If you select a request, you will get all the details for that request in the pane below the list of all requests (I selected a request to Twitter’s search.twitter.com in the next screen shot). Basically, the Request and the Response sections present the request sent by the app and the response received from the server.

On the bottom of the details viewer, there are some options that depend on what section of the details you are in. For example, in the Request section, you can view the Request’s Query String (previous screen shot) or the Request Headers (next screen shot).

One of the nice features of Charles is that is able to detect a few commonly used container formats, like JSON and XML, probably based on the Content-Type response header. For example in the case the response contains JSON, Charles will show a “JSON” tab with a view that formats the JSON response conveniently in a more readable way instead of just displaying raw text.

Proxying traffic from an iOS device to Charles

The next step is to use Charles to inspect traffic from an actual iOS device. The easiest way to do this, is to use Charles’ built-in proxy. First you need to tell your iOS to use the proxy. Go to Settings > Wifi > Select the Wifi network and make sure your machine that is running Charles is also in this network. Scroll down to the “HTTP Proxy” setting. (See next screen shot.) Select “Manual”. Enter the IP address of your machine that is running Charles. You can find the IP address quickly in Charles’ Help > Local IP Address menu. Enter “8888″, which is the default proxy port of Charles. Authentication is off by default.

Now open up an app that uses HTTP calls to a server (e.g. browse the web in Safari). Charles will detect that a new host is trying to use its proxy. You will need to accept it once by choosing “Allow”:

Once allowed, traffic will flow from your iOS device through Charles, which will forward it to its destination (while allowing you to inspect it). In the next screen you can see what traffic the Google Maps app is causing when I’m panning over the map on my iPhone (no JSON there but many binary responses):

Intercepting SSL traffic from an iOS device using Charles

Now you might think, wait a minute, what about SSL / HTTPS traffic? What if my fancy web service is using an encrypted connection? Well, fortunately there is a pretty simple way to inspect that as well. In this contrived example, we will look into encrypted traffic from Safari to https://mobile.twitter.com/. Open up the Proxy Settings menu from Charles. Add the hostname of the HTTPS server. In our case we added “mobile.twitter.com” and the default SSL port 443:

If you try visiting https://mobile.twitter.com/ from your proxied iPhone, Safari will present you a warning that it cannot verify the server’s identity because the Charles proxy is using untrusted, self-signed certificates.

You don’t want this, because the (correct) default behaviour of iOS (and NSURLConnection for example) is to disallow untrusted server connections. The solution is to add the Charles root certificate (the certificate that is used to generate SSL certificates on-the-fly when you are using the proxy) to the list of trusted root certificates in your iOS device. To do this you need to add a configuration profile containing Charles root certificate bundle. To save everyone some time, we created a signed configuration profile with Charles’ certificate bundle ¬†for everyone to use.

If you visit http://charles.noodlewerk.com/ using Safari on your iOS device, you will be prompted whether or not you want to install the profile and the containing bundle. Once installed, iOS will trust the Charles proxy and will act as normally. Note that in essence you are weakening the security by installing this profile. So it is wise to remove the profile once you are done debugging if you are using your device in real life :)