In an ideal world, we’d all have our own dedicated network of super fast, highly secure proxies dedicated for our use only. In reality though this is extremely expensive and unless you’ve got a pretty huge budget then it’s pretty much unrealistic for most of us. The majority of us have to compromise somehow, perhaps on proxy type, speed or where the proxies are hosted – even with these limitations you can still spend serious amounts on getting even a half decent proxy infrastructure.
The major problem that faces most people is scale, proxies are normally needed to support automated tools and these can ‘burn’ through servers very quickly indeed. One of the classic mistakes that newcomers tend to make is to overuse their proxies by not configuring their software properly. This can get very costly when your expensive servers get their IPs blacklisted one by one. Although it’s not complicated, people often lose sight of the basics – that is the red flags that sites look for in detecting the IP address of a proxy server.
Here’s some of the important ones:
- Large number of queries coming from suspicious location
- Large number of queries coming in too quickly.
- Large number of queries from same web browsers
- Large number of duplicate/suspicious queries.
That is not a definitive list, but it’s the sort of things that many websites look out for when checking for proxies. When you think about it, they’re common sense especially if you consider the queries from the other side of the coin. First of all remember that especially in commercial setups, the proxy is there to make you look like a legitimate customers. If you’re trying to stock up on the latest Jordan’s to fill your eBay inventory then routing your internet connection through a proxy server based in Lagos, Nigeria is not a wise move. Firstly the country has a poor reputation for online fraud and secondly an E-commerce site set up to service the US market is not going to be interested in Nigerian based traffic.
|Rotating Proxy FAQ|
|What's the Best Proxy to Use?||If possible use private residential proxies that are specifically designed for your task e.g. Instagram proxies, Twitter proxies.
|What is Proxy Scraping ?||Using proxies to access popular sites repeatedly without being blocked or results tapered to your real IP address.|
|What is a Proxy Pool?||These are the advanced proxies which instead of using static IP addresses rotate through a pool of different IP addresses.|
|What are Residential Proxies?||Proxies which have residential IP addresses registered for home users. Normally only allocated to ISPs.|
|Do I need a VPN ?||Main difference between a proxy and VPN is the VPN encrypts all your communication which means activity can't be logged and is more secure.|
Indeed most commercial servers are already programmed to detect high risk countries and flag them for potential fraud. It’s not entirely fair but e-commerce sites have learnt to their cost where most fraudulent transactions come from and often simply block them. Indeed many entrepreneurs resell in these countries at hugely inflated prices simply because items are so hard to obtain legitimately.
So How Do I Choose the Best Rotating Proxies
The most basic answer is of course, simply by trial and error. The best proxies depend largely on the purpose you’re using them for – a network of proxies may be fabulous for Ticketmaster but may be completely blocked out by Google or Instagram. You might be lucky and find extremely cheap, generic proxies that work brilliantly for you – although this is more likely for scraping and research tools than with commercially focused software. In most circumstances though you’ll find the best proxy providers will specifically group and manage their proxy resources for specific purposes. If a block of IP addresses is constantly used to access a specific website by hundreds of paying customers then they’ll quickly get blacklisted. However if the proxies are carefully monitored and rotated properly they can often be used indefinitely for a specific purpose.
This is why you’ll see the best proxy networks will provide ‘sneaker proxies’ or ‘instagram proxies’ and so on. Large blocks of undefined proxies used for any purpose and not rotated properly soon become useless for any sites. If you have a very specific need for these servers it’s worth checking this before you buy rotating proxies from anyone.
There are a few proxy providers who really know what they’re doing and make a real effort to protect their resources. After all although anyone can create simple proxies based in datacentres very easily, residential proxies are much more difficult to obtain and cost lots of money to replace. These are proxies with residential IP addresses which are much less likely to be flagged or blocked by ordinary websites.
Beware Cheap Rotating Proxies with Residential Addresses
Indeed any company with these sorts of resources will be very careful to configure the other aspects which are likely to raise red flags. For example creating unique user agents and properly relaying the native referrer source from each individual IP address.
These companies have also discovered that the best way to protect their proxy resources is to ensure that they control the rotation of the servers and addresses too. Although most automated software allows you to switch IP addresses automatically there’s no guarantee that this will be configured correctly. By setting up backconnect rotating proxies, the provider can control how often each IP address is used. You can read more about backconnect rotating proxies on this site too.
There is another question that you may want to ask yourselves – where do these addresses come from? We’ve stressed that it’s difficult to get hold of residential IP addresses as they’re distributed by ISPs to home users. So how do the likes of Microleaves and Luminati manage to get hold of millions of these addresses. Well there’s only one real method to build up those sort of numbers and that’s to get them from ordinary users themselves. Microleaves for example is well known for rather nasty adware which installs itself on your computer which can be controlled via a browser plugin. This also gives them access to your internet connection and residential IP address. Luminati offer a free VPN and some other applications in exchange for using your connection too.
Ultimately, you will to a certain extent get what you pay for with proxies. You can of course find free proxies online but they are totally useless for most commercial tasks simply because they’re completely overloaded. Also a fair proportion of them are set up by cyber criminals and identity thieves to steal data so you should never use them for any commercial transactions, or anything involving usernames or passwords.
There is however another reason why many providers use rotating proxies and that’s because they don’t actually own any IP addresses of their own. Yes, there’s actually a thriving market in reselling other peoples connections to provide what looks like a professional service which supplies unlimited residential proxies (yet is actually little more than a reseller with a fancy website). These addresses are bought from a pool of addresses and connections created from a variety of means – here’s just a brief example.
- Application Vendors who’s API provides access to User’s internet connections.
- VPN services which supply ‘free options’ which share the host’s connections
- Malware and Virus farmed addresses from hacked accounts and computers (beware free proxy list api offers as this is where many addresses originate from)
It’s big business now and particularly residential addresses from normal internet connections have a significant value. If you’ve done any research into how to make residential proxies then you’ll appreciate how difficult and expensive it is to create them. If the price seems low you’re probably better avoiding them altogether as there’s a large chance that the IP addresses actually are hacked or abused internet connections shared by thousands.
Unfortunately, nowadays residential IP addresses are increasingly required as many sites block access from the cheap and available commercial IP address ranges. There is no way to obtain any volume of these without spending a lot of money, but they’re increased cost is definitely worth it.
There are actually only a handful of proxy providers who offer a reliable and extensive network of proxies. Indeed many companies merely lease or resell other peoples proxies and addresses too. The best ones will answer your questions and will usually offer a brief trial for you to test. Be aware that most can’t offer long trials as they are often abused and the IP addresses blocked.