Playwright with authenticated proxies (Python and Node.js)

Playwright has proxy support built in, including the username and password that most proxy services need. This guide shows the setup in Python and Node.js, how to run several identities side by side with one IP each, and how to stop a headless browser from burning through your bandwidth.

Use the HTTP gateway for browsers: geo.crawlproxies.com:8080 on Residential. Playwright's username/password support is documented for HTTP proxies, so it's the most compatible choice. Your credentials are in the generator.

Python

pip install playwright
playwright install chromium

from playwright.sync_api import sync_playwright

PROXY = {
    "server": "http://geo.crawlproxies.com:8080",
    "username": "USERNAME-country-us",
    "password": "PASSWORD",
}

with sync_playwright() as p:
    browser = p.chromium.launch(proxy=PROXY)
    page = browser.new_page()
    page.goto("https://ipinfo.io/json")
    print(page.inner_text("body"))
    browser.close()

Note that the username and password are separate fields. Don't put them in the server URL; Playwright won't pick them up from there.

Node.js

npm i playwright
npx playwright install chromium

import { chromium } from "playwright";

const browser = await chromium.launch({
  proxy: {
    server: "http://geo.crawlproxies.com:8080",
    username: "USERNAME-country-us",
    password: "PASSWORD",
  },
});
const page = await browser.newPage();
await page.goto("https://ipinfo.io/json");
console.log(await page.innerText("body"));
await browser.close();

One IP per browser context

A browser context is an isolated session: its own cookies, storage and cache. Give each context its own proxy username and you get several independent identities from one browser process:

def new_identity(browser, name, country="us"):
    return browser.new_context(proxy={
        "server": "http://geo.crawlproxies.com:8080",
        "username": f"USERNAME-country-{country}-session-{name}-time-3600",
        "password": "PASSWORD",
    })

with sync_playwright() as p:
    browser = p.chromium.launch()
    for name in ("alice1", "bob2", "carol3"):
        ctx = new_identity(browser, name)
        page = ctx.new_page()
        page.goto("https://ipinfo.io/ip")
        print(name, page.inner_text("body").strip())
        ctx.close()
    browser.close()

Each name gets its own sticky IP for up to an hour (-time-3600), and reusing the same name later brings that session back while it's still alive. Use letters and numbers for session ids.

On older Playwright versions Chromium ignored per-context proxies unless the browser was also launched with one. If your contexts all show your own IP, launch with a placeholder: p.chromium.launch(proxy={"server": "http://per-context"}). Every context then overrides it.

Sticky for logins, rotating for crawling

• Logging in, filling a cart, multi-step forms: use a session id (sticky). Changing IP in the middle of a login is one of the fastest ways to get flagged.
• Visiting many unrelated pages: leave the session out and open a new context per page or per batch, so each one leaves from a fresh IP.

Sticky vs rotating sessions goes into more detail.

Save bandwidth: block what you don't need

Proxy plans are billed by the gigabyte, and a modern web page loads several megabytes of images, video and fonts that a scraper never looks at. Blocking them often cuts traffic by more than half:

BLOCK = {"image", "media", "font"}

def block_heavy(route):
    if route.request.resource_type in BLOCK:
        return route.abort()
    return route.continue_()

page.route("**/*", block_heavy)
page.goto("https://example.com/")

The Node.js version:

await page.route("**/*", (route) =>
  ["image", "media", "font"].includes(route.request().resourceType())
    ? route.abort()
    : route.continue()
);

Keep stylesheets and scripts. Many sites need them to render the content you're after, and some detect clients that never load them.

Timeouts and retries

Residential exits are real home connections, so give navigation a little more room than the default and retry the occasional failure:

page.set_default_navigation_timeout(60_000)

for attempt in range(3):
    try:
        page.goto("https://example.com/", wait_until="domcontentloaded")
        break
    except Exception as e:
        print("retrying after", type(e).__name__)

wait_until="domcontentloaded" returns as soon as the HTML is parsed instead of waiting for every last request, which is faster and uses less bandwidth.

Troubleshooting

Using Puppeteer instead? The setup is a little different: see Puppeteer with authenticated proxies.