Over the years I’ve used literally thousands of proxies from all over the world for a huge variety of reasons. Often it’s just because of ordinary, everyday paranoia, I just want to keep my connection private and secure. When you’ve spent a lot of time poring over server logs at work you begin to realise exactly how much information about our lives we actually leak online.
Other times it’s for more practical purposes, completing tasks that are not possible with a single IP address from one computer. Proxies have a myriad of uses – research online, running social media accounts, buying stock from e-commerce stores and even trying to watch videos or TV programmes from ‘blocked locations’. In truth, the practical elepments of using proxies is arguably more important to me, after all my twenty Instagram accounts basically pay my mortgage! But without a bank of usable IP addresses, I couldn’t promote or sell through them using tools like Jarvee.
Proxies enable us to ‘bend the rules’, social media sites like Instagram don’t wont us running multiple accounts. Commercial sites want to be able to restrict who and how much they sell to – in order to maximise their profits. I want to enjoy the BBC when I’m travelling abroad yet without a UK IP address to use, the site won’t work.
Most large web sites want us to have a single digital identity which they can track, monitor and control.
A proxy allow you to bypass these blocks and filters, it allows you to run multiple identities without restrictions and to access whatever site you want irrespective of location. What is more they allow you to function online anonymously without being monitored too.
Why Some Don’t Like Proxies
In 2012 there were a staggering number of Australian Netflix subscribers and the numbers were rising all the time. This was quite surprising considering that Netflix wasn’t actually technically available in Australia yet, nobody should have been able to watch it at all! There were in fact thousands of people watching the US version of Netflix routed through a proxy situated in the USA. These people were still paying a subscription, yet had to resort to hiding their location in order to access the service.
Last year I wanted to go with a group of friends to see one of favorite bands, think aging but incredibly popular rock band! Now getting a few tickets to see this band was virtually impossible unless you went through ticket touts and paid several times the cover prices. Yet a few rotating proxies hired for a couple of weeks and a bot which automated the buying process from Ticketmaster solved this problem. Twelve tickets purchased at the proper price and a group of friends enjoyed the concert, no harm done?
Of course there are lots more mundane reasons why organisations hate proxies. Many states try and filter the internet for social, religious and political reasons, most of the methods involved are circumvented by using a proxy or VPN in another country. Lots of companies are understandably keen to stop their employees spending half their day on Facebook or Pinterest instead of working so routinely block these sites. Again a proxy server with a layer of encryption will thwart most of these attempts if they’re setup properly.
The list goes on and on, for every useful function or task you can achieve using a proxy, VPN or extra IP address then there’s likely a whole host of people who want to block them for the same reason. I for one couldn’t live without them!
So How are Proxies Being Blocked ?
The war being raged upon proxies has of course been happening for years. Whether it’s IT professionals seeking to secure their networks, large companies protecting their profits or State run telecom firms taking control of their population. However it’s not that easy and stopping the use of all these anonymous services can be extremely difficult.
For a start there’s literally millions of different proxies around plus a whole host of obfuscation techniques which can hide them and the connections they protect.
The issue can be split into two distinct parts –
- Detection – it’s actually not that easy to identify connections from proxies/VPN services, particularly in high traffic networks.
- Blocking – preventing access to a specific proxy on a certain IP address is fairly straight forward. However there will be thousands of these connections on popular sites. Often when I need them, my own subscription to Storm Proxies gives me access to several hundred IP addresses and connections.
Detecting Inbound Connections from Proxy Servers
The sophistication of proxy detection systems varies greatly depending on the organisation and the resources they put into it. The reality is that detecting, blocking and restricting access to these proxy services is a very time consuming process.
One of the simplest steps is to create a simple blacklist, that is a list of addresses of known proxy servers. There are many ways to build up these lists but most involve some manual effort. For example, there are many tools which people use to scrape the internet for open proxies which can be used to identify many free one. Ironically many individuals use these tools to find proxies to use themselves. The scripts are also used to create online dynamic sites which contain proxy lists – another useful source of addresses to block and potentially one that can be used dynamically to update a blacklist.
Identifying Popular/Known Services
Although there are hundreds of commercial providers of proxies, VPN services and IP addresses there are some very large sources of these. Identifying specific ranges being used by these services is actually fairly simple to do. Paying for a quick subscription will usually unlock much of the information in a few minutes. Locating the major players is often done by targeting the most aggressive, for instance the cheaper services who advertise a lot is an easy first step.
As well as using these manual ‘detective’ methods there are of course various more technical detection methods employed. For example a very common proxy connection is ‘The Onion Router’ or TOR as it’s more commonly known. This is a very effective proxy service which operates by bouncing your connection around a network of relays run by other volunteers in the network. As such it’s much more difficult to build up a list of effective IP addresses as this will be extremely dynamic and change almost constantly.
TOR is a powerful tool and a force for enabling free speech in many countries in the world where the internet is heavily monitored and filtered. However it obviously can be used for other purposes too, which is why it’s important that network administrators have some methods of detecting for the use of TOR. There are several reasonably effective rules you can add to intrusion detection systems to spot inbound and outbound TOR connections, including some freely available for Snort.
Other Proxy Detection methods
So apart from specific rules detecting patterns in the connection and the broader IP blacklisting methods, there are some other methods. The Chinese government have some very advanced methods implemented to reinforce the Great Chinese Firewall, although these are well beyond the reach of most commercial enterprises.
Netflix implemented a very effective block by simply banning all connections that originate from non-residential IP addresses. This meant that 99% of VPN/Proxies were of little use if you wanted to watch a particular geo-version of Netflix because most use IP addresses originating from commercial datacentres.
There are other methods like detecting specific signatures or problems with the user agent fields caused by a badly configured proxy. You can even scan the IP addresses of each client accessing the web site looking for open ports and proxy software. In reality though most of these are so time consuming that only state or military organisations would be able to implement these methods effectively. Who knows what the future might bring though and you can find lots of free projects on both the proxy hiding and detecting sides – have a look at the Frost Jedi Proxy Detector project for example.
For most of us, it’s relatively simple to avoid detection by using a decent proxy company who can configure their hardware properly and maintain a healthy stock of IP addresses from different sources – e.g residential, commercial and mobile. There are certainly several really effective and reliable suppliers who can provide the right proxies for your specific needs.
If you want a recommendation – may we suggest Storm Proxies, a well established company who provide loads of support and can almost certainly supply the right proxies for your needs. They are also crucially one of the few companies who own and control the hardware that their proxies run on – rather than leasing and using other peoples computers.
Try them, they have a 48 hour money back guarantee if there’s a problem.