Proxies in PHP: cURL and Guzzle with authentication

PHP's cURL extension has everything you need for proxies built in: the proxy address, the login and the protocol are each one option. This guide shows the plain cURL setup, the same thing in Guzzle, and the details that matter once you scale up: rotation, sticky sessions and running requests in parallel.

Your username and password are in the generator. The examples use the Residential gateway geo.crawlproxies.com (8080 for HTTP, 1080 for SOCKS5).

cURL: HTTP proxy

<?php
$ch = curl_init('https://ipinfo.io/json');
curl_setopt_array($ch, [
    CURLOPT_PROXY          => 'http://geo.crawlproxies.com:8080',
    CURLOPT_PROXYUSERPWD   => 'USERNAME:PASSWORD',
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_TIMEOUT        => 30,
]);

$body = curl_exec($ch);
if ($body === false) {
    echo 'Error: ' . curl_error($ch) . PHP_EOL;
} else {
    echo curl_getinfo($ch, CURLINFO_HTTP_CODE) . ' ' . $body . PHP_EOL;
}

CURLOPT_PROXYUSERPWD keeps the login out of the proxy URL, so a password with @ or : in it needs no escaping. HTTPS sites go through a CONNECT tunnel: the connection to the website stays encrypted end to end.

cURL: SOCKS5

curl_setopt_array($ch, [
    CURLOPT_PROXY        => 'geo.crawlproxies.com:1080',
    CURLOPT_PROXYTYPE    => CURLPROXY_SOCKS5_HOSTNAME,
    CURLOPT_PROXYUSERPWD => 'USERNAME:PASSWORD',
]);

CURLPROXY_SOCKS5_HOSTNAME lets the proxy resolve the website's hostname. With plain CURLPROXY_SOCKS5, DNS lookups happen on your own server.

Targeting and sticky sessions

Location and session options go in the username. A German IP:

CURLOPT_PROXYUSERPWD => 'USERNAME-country-de:PASSWORD',

The same IP for up to 30 minutes, for a login or a checkout flow:

$session = bin2hex(random_bytes(4));
CURLOPT_PROXYUSERPWD => "USERNAME-country-us-session-{$session}-time-1800:PASSWORD",

Every option is in the geo-targeting guide; when to use which mode is in sticky vs rotating sessions.

Making IPs rotate

With a rotating username, every new connection gets a new IP. cURL keeps connections open and reuses them when you reuse a handle, so a loop over one handle can keep the same IP. Ask for a fresh connection per request:

curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);   // new connection, so a new IP
curl_setopt($ch, CURLOPT_FORBID_REUSE, true);    // and don't keep it for the next request

Creating a new handle per request has the same effect.

Guzzle

Guzzle passes its proxy option straight to cURL, credentials included:

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;

$client = new Client([
    'timeout' => 30,
    'proxy'   => 'http://USERNAME:PASSWORD@geo.crawlproxies.com:8080',
]);

$res = $client->get('https://ipinfo.io/json');
echo $res->getStatusCode() . ' ' . $res->getBody() . PHP_EOL;

Set proxy on the client to use it for every request, or pass it per request ($client->get($url, ['proxy' => ...])) to give different requests different locations or sessions. If the password contains special characters, URL-encode it with rawurlencode() before putting it in the URL.

Requests in parallel

curl_multi runs many transfers at once from a single PHP process. Keep the batch size modest, around 10 at a time, and raise it while watching for 429 responses:

<?php
$urls = ['https://example.com/a', 'https://example.com/b', 'https://example.com/c'];

$mh = curl_multi_init();
$handles = [];
foreach ($urls as $url) {
    $ch = curl_init($url);
    curl_setopt_array($ch, [
        CURLOPT_PROXY          => 'http://geo.crawlproxies.com:8080',
        CURLOPT_PROXYUSERPWD   => 'USERNAME:PASSWORD',
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_TIMEOUT        => 30,
    ]);
    curl_multi_add_handle($mh, $ch);
    $handles[$url] = $ch;
}

do {
    $status = curl_multi_exec($mh, $running);
    if ($running) {
        curl_multi_select($mh);
    }
} while ($running && $status === CURLM_OK);

foreach ($handles as $url => $ch) {
    echo curl_getinfo($ch, CURLINFO_HTTP_CODE) . " $url" . PHP_EOL;
    curl_multi_remove_handle($mh, $ch);
}

With Guzzle, GuzzleHttp\Pool does the same with a concurrency setting.

Errors you might see

When something doesn't work, try the same line in a terminal first; the cURL cheat sheet has the commands. If it works there, the problem is in the PHP options.