Proxies in Go: net/http with authentication, SOCKS5 and rotation

Go's net/http has proxy support built in, including credentials, and since Go understands socks5:// URLs too, you don't need any third-party package for either. This guide covers the setup with crawlproxies, the keep-alive detail that decides whether your IPs rotate, sticky sessions, and a simple concurrent crawler.

Your username and password are in the generator. The examples use the Residential gateway geo.crawlproxies.com.

HTTP proxy

package main

import (
	"fmt"
	"io"
	"net/http"
	"net/url"
	"time"
)

func main() {
	proxyURL, err := url.Parse("http://USERNAME:PASSWORD@geo.crawlproxies.com:8080")
	if err != nil {
		panic(err)
	}

	client := &http.Client{
		Transport: &http.Transport{Proxy: http.ProxyURL(proxyURL)},
		Timeout:   30 * time.Second,
	}

	resp, err := client.Get("https://ipinfo.io/json")
	if err != nil {
		panic(err)
	}
	defer resp.Body.Close()
	body, _ := io.ReadAll(resp.Body)
	fmt.Println(resp.Status, string(body))
}

Go takes the username and password from the proxy URL and sends them as Proxy-Authorization. HTTPS sites are reached through a CONNECT tunnel, so the connection to the website stays encrypted.

Special characters: build the URL with url.UserPassword instead of string formatting if your credentials could contain characters like @ or :. It escapes them for you.

proxyURL := &url.URL{
	Scheme: "http",
	User:   url.UserPassword("USERNAME-country-us", "PASSWORD"),
	Host:   "geo.crawlproxies.com:8080",
}

SOCKS5

Change the scheme and the port. Nothing else:

proxyURL := &url.URL{
	Scheme: "socks5",
	User:   url.UserPassword("USERNAME", "PASSWORD"),
	Host:   "geo.crawlproxies.com:1080",
}
client := &http.Client{Transport: &http.Transport{Proxy: http.ProxyURL(proxyURL)}}

Making IPs rotate

With a rotating username, every new connection gets a new IP. But http.Transport keeps connections alive and reuses them, so a loop of requests through one client can all come from the same IP. If you want a fresh IP per request, turn keep-alive off:

transport := &http.Transport{
	Proxy:             http.ProxyURL(proxyURL),
	DisableKeepAlives: true, // a new connection, and so a new IP, for every request
}

That costs a little speed (a new TCP and TLS handshake each time), so only do it when you need it.

Sticky sessions

For the opposite (the same IP across a login or a multi-step flow), put a session id and a duration in seconds into the username and keep the connection reuse on:

func stickyClient(sessionID string) *http.Client {
	u := &url.URL{
		Scheme: "http",
		User:   url.UserPassword("USERNAME-country-us-session-"+sessionID+"-time-1800", "PASSWORD"),
		Host:   "geo.crawlproxies.com:8080",
	}
	return &http.Client{
		Transport: &http.Transport{Proxy: http.ProxyURL(u)},
		Timeout:   30 * time.Second,
	}
}

Create one client per account or per job. Sticky vs rotating sessions explains when to use which.

A bounded worker pool

Goroutines make it easy to fire off thousands of requests at once, which is also the easiest way to get rate-limited. Cap the concurrency with a buffered channel:

func crawl(client *http.Client, urls []string, workers int) {
	sem := make(chan struct{}, workers)
	var wg sync.WaitGroup

	for _, u := range urls {
		wg.Add(1)
		sem <- struct{}{}
		go func(u string) {
			defer wg.Done()
			defer func() { <-sem }()

			resp, err := client.Get(u)
			if err != nil {
				fmt.Println("error:", u, err)
				return
			}
			io.Copy(io.Discard, resp.Body) // read the body so the connection can be reused
			resp.Body.Close()
			fmt.Println(resp.StatusCode, u)
		}(u)
	}
	wg.Wait()
}

Add "sync" to your imports. Start with around 10 workers and raise it gradually while watching for 429 responses. Always read and close response bodies: an unread body keeps the connection busy and can stall the pool.

Errors you might see

That's everything Go needs. The same username options (country, state, city, session) are explained in the geo-targeting guide.