TLDR; A collection of tips for working with HTTP Proxies. Use a proxy against localhost or from mobile by using your machine name or private ip address from ifconfig (or ipconfig). Chain proxies to use features from all tools at the same time and make request manipulation more reliable. Charles proxy can validate HTML as you test.
Tips:
- Use a proxy against localhost by using your machine name or private ip address from ifconfig (or ipconfig).
- Mobile devices can use proxies on your main machine if you find your ip address and configure the tool to point to the proxy on that ip address.
- Use a browser plugin like Foxy Proxy Standard to make switching between proxies easy
- Charles proxy can validate HTML passively as you test
Working With Localhost
One issue we might experience when working with locally installed applications or testing apps running from a local docker container is that they run on localhost, and we might want to proxy them.
Generic methods
Instead of http://localhost:80
Use:
- the machine name e.g.
http://machinename:80 - your private ip address e.g.
http://192.168.1.132:80
There are a range of ip addresses that are private and not accessible from the internet
You can find your machine ipaddress by using ifconfig or ipconfig (if you are on Windows)
Using this ip address is the most generic approach for using a proxy on your local machine and should work regardless of which proxy you are using.
When I want to find the ip address on mac I combine the ifconfig with a grep to make viewing the output easier:
ifconfig | grep "inet "
Charles Proxy
https://www.charlesproxy.com/documentation/faqs/localhost-traffic-doesnt-appear-in-charles/
Charles proxy supports the address http://localhost.charlesproxy.com so you could add the port on the end of that and your traffic should appear in Charles.
Fiddler
https://docs.telerik.com/fiddler/Configure-Fiddler/Tasks/MonitorLocalTraffic
Fiddler supports two special case addresses to use instead of localhost
http://ipv4.fiddlerhttp://ipv6.fidder
Demo Video
Chaining HTTP Proxies
I chain HTTP debug proxies.
That way I can use features from all of them, at the same time:
- Fiddler Autoresponders
- BurpSuite passive sitemap building
- ZAP’s multiple breakpoints
- Scripting in zap to amend requests and response without the message being cut off
- etc.
I’ve also found this useful when scripting in the proxies, sometimes, for some reason the generated http request after amending from a script isn’t completely sent to the browser or server, but when I chain proxies and have the script amending the request or response then go through another proxy before arriving at its destination, that seems to help.
Fiddler
Fiddler hooks into the windows system seamlessly without any additional config. All other proxies I point at Fiddler as the down stream proxy.
When fiddler is running. Test your setup by pointing your browser through Fiddler.
BurpSuite
BurpSuite Options tab. Upstream Proxy Servers. Add an entry for Fiddler:
- Destination Host: *
- Proxy Host: localhost
- Proxy Port: 8888
At this point - test your setup again. Don’t chain everything together and then try and figure out where the problem is. Point your browser at BurpSuite and check you can see traffic through them all.
If you get stuck, use the Alerts tab in BurpSuite to check for errors.
ZAP
Tools Options Connection. Then point it at BurpSuite
- Port: 8082 (or whatever you bound BurpSuite too)
Find the port you have bound ZAP to in Tools Options Local Proxy
And point the browser at this port now.
End
Voila, you should have it all chained.
If not, just revisit the last step. And don’t panic.
Step by step. check each part of the journey. If its not working, it will probably just be some stupid error where you had previous config left over from a previous session.
Just remember to unwind them when you are done.
I have a video in the Technical Web Testing 101 course that shows this in more detail.
Links:
Mobile Devices
Viewing the HTTP traffic from your mobile browsers doesn’t take that long to set up, but there are a few gotchas to be aware of:
- You need to find the right IP address on your desktop
- You need to change your proxy settings on your phone
- Make sure your proxy allows external connections
- Make sure both Mobile Device and Proxy Machine are on the same network
The setup I normally use:
- Phone connected to wifi network
- Windows Desktop connected to same network via wired connection (or laptop connected via wireless)
As an example, using Fiddler as the debug proxy, on the desktop:
- Start Fiddler
- Check that Fiddler has “Allow remote computers to connect” (via Tools Fiddler Options Connections)
- Start a command prompt and use “ipconfig” to show you the current list of ip addresses for you computer
- Configure mobile device
Android Devices
On the mobile device, details below are for android (but the principle is the same for other operating systems):
- Open Settings
- Wi-Fi settings
- Long press on the wireless network you are using to access the connection settings
- Modify Network Config
- Proxy Settings - Manual
- change the Proxy HostName to the IP Address of your desktop
- Add the port for your debug proxy e.g. 8888 for Fiddler
Then your browser should be connecting to the proxy.
Other applications may not use the proxy in this way. You might need to setup port forwarding use adb to feed them through to your proxy (don’t leave comments asking how to do this check on google. I haven’t had to do this for some time so my memory of hacking around on the android device to get it working is hazy.)
On Android: Chrome, Firefox, Dolphin and the inbuilt browser all worked without issue. I had some hassle connecting Opera, but didn’t try and diagnose why.
Try it and see how you get on.
It makes a big difference to the visibility of your testing when working through mobile.
Additional Notes:
- For burpsuite, in “Proxy Options” edit the proxy listener to Bind to address “All interfaces”
How to Proxy iOS Devices
Connecting an iOS device to an HTTP Proxy is much the same as we demonstrated on Android devices.
On your iOS settings:
- In Wifi
- Select the (i) information icon next to your wifi network
- On the bottom of the screen is the HTTP Proxy settings
- Set this to Manual
- Type in the Server IP address and port of your proxy
Done.
Now your HTTP traffic should flow through the desktop proxy.
iOS does a pretty good job of caching so I either clear the cache before I start testing use the Safari Settings and “Clear Cookies and Data” or use the “Private” link at the bottom of Safari to start a new session.
I found iOS doesn’t like connecting to some networks, and so I setup a local hotspot with connectify.me and that made my life a little easier.
Remember to switch off the proxy when you are done.
Easily configuring Chrome and Firefox
One tool that has helped me take advantage of proxies more regularly is a browser plugin called foxy proxy.
With FoxyProxy installed, I start the proxy, click a button then choose the proxy configuration I want to use. Most of the proxies default to 8080 so one configuration usually meets most of my needs.
Without this I have to fiddle in browser settings or network settings and I would never use proxies.
This also makes it easy to switch off proxying and get back to normal.
If you work with HTTP Proxies, I thoroughly recommend installing a plugin to help switch between them and switch them off.
Validating HTML with Charles Proxy
TLDR; Charles proxy can use validator.w3.org and report results in the proxy GUI
After looking at my default test process for web and realising it didn’t include HTML validation by default, I decided to find the easiest way I could to add it in to my process. I generally find that the easier I can slot something into my process, the more likely I am to use it - just like the foxy proxy extension.
We all use Dev Tools by default now
Dev tools are bundled with every browser now. Previously we had to download plugins and add ons. But now that it is a right click away by default, we have no excuse.
But HTML validation doesn’t appear to be there. I can ‘audit’ the performance of the page in the Chrome dev tools, and the CSS usage checking is quite useful. But no HTML validation.
Which is the higher technical risk to your products functionality? CSS? Speed? HTML?
Sadly, the latter I think has caused more issues across devices than any of the others.
So what can help me get the basics right? And do so without trying too hard.
I mentioned validator.w3.org in a previous post. But that requires work. I have to check each page one by one.
A quick web search revealed a bunch of HTML validation tools. Some of which look quite good and I Will evaluate them in due course. But they require adding a new tool type into my workflow, which means I have to change how I test and that means I need to work harder to incorporate it.
What I really wanted was a proxy that validates HTML
- I had a quick search for Fiddler plugins to validate HTML but couldn’t see any.
- I couldn’t remember any of my existing proxies offering to validate HTML.
But a web search for proxies that validate html popped up Charles Proxy.
I already use Charles Proxy, but it never had any standout features that meant I used it regularly. So I must have missed the HTML validation functionality.
But there it is, clear as day.
What does the Validate option in Charles Proxy do?
Given the urls that you highlight in the ‘session’. Charles will open up a new ‘validation’ tab and pass each of those urls through to the w3.org validator and format the results for you.
Chain proxies allows us to take advantage of this even if we prefer to use one of the other proxies for observation and manipulation. I can always add Charles into the chain and use it for validation during the process.
Since Charles is cross browser I use this on my Windows and Mac testing.
Summary
- Understand your own workflow
- Try and find tools that fit into your existing workflow
- Learn what your existing tools already do
- Try to add automated validation into your workflow seamlessly
- Charles Proxy can use w3.org validator to validate HTML
And with that I’ve just increased my workload and the number of things to check and observe, but hopefully for the general betterment of my testing and my web sites and web apps.
