Proxies in C#: HttpClient with authentication, SOCKS5 and rotation

.NET's HttpClient supports authenticated proxies out of the box through WebProxy, and since .NET 6 it speaks SOCKS5 too. This guide covers the setup, the connection-pooling detail that decides whether your IPs rotate, sticky sessions and a capped parallel crawl.

You need .NET 6 or newer. Your username and password are in the generator; the examples use the Residential gateway geo.crawlproxies.com.

HTTP proxy

using System.Net;

var proxy = new WebProxy("http://geo.crawlproxies.com:8080")
{
    Credentials = new NetworkCredential("USERNAME", "PASSWORD"),
};

var handler = new HttpClientHandler { Proxy = proxy, UseProxy = true };
using var client = new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(30) };

var body = await client.GetStringAsync("https://ipinfo.io/json");
Console.WriteLine(body);

HTTPS sites are reached through a CONNECT tunnel, so the connection to the website stays encrypted. Create the client once and reuse it: a new HttpClient per request wastes sockets.

SOCKS5

Change the address and the port; the credentials stay the same:

var proxy = new WebProxy("socks5://geo.crawlproxies.com:1080")
{
    Credentials = new NetworkCredential("USERNAME", "PASSWORD"),
};

Targeting and sticky sessions

Location and session options are part of the username. One client per identity keeps things simple:

static HttpClient ClientFor(string username) =>
    new(new HttpClientHandler
    {
        Proxy = new WebProxy("http://geo.crawlproxies.com:8080")
        {
            Credentials = new NetworkCredential(username, "PASSWORD"),
        },
        UseProxy = true,
    })
    { Timeout = TimeSpan.FromSeconds(30) };

var germany = ClientFor("USERNAME-country-de");
var sticky = ClientFor($"USERNAME-country-us-session-{Guid.NewGuid():N}-time-1800");

The sticky client keeps the same IP for up to 30 minutes, which is what a login or a checkout flow needs. See sticky vs rotating sessions and the geo-targeting guide for every option.

Making IPs rotate

With a rotating username, every new connection gets a new IP. HttpClient pools connections and reuses them, so a loop through one client can keep the same IP. To get a new IP per request, close the connection after each response:

client.DefaultRequestHeaders.ConnectionClose = true;   // new connection, so a new IP

That costs a new TCP and TLS handshake per request, so only turn it on when you need it.

Requests in parallel, with a cap

Parallel.ForEachAsync runs requests concurrently and limits how many are in flight:

var urls = new[] { "https://example.com/a", "https://example.com/b", "https://example.com/c" };

await Parallel.ForEachAsync(urls, new ParallelOptions { MaxDegreeOfParallelism = 10 }, async (url, ct) =>
{
    try
    {
        using var res = await client.GetAsync(url, ct);
        Console.WriteLine($"{(int)res.StatusCode} {url}");
    }
    catch (HttpRequestException e)
    {
        Console.WriteLine($"error {url}: {e.Message}");
    }
});

Start around 10 and raise it gradually while watching for 429 responses.

Errors you might see

The same username options work in every language: compare with Python, Node.js or Go.