Why Does My Selenium Scraper Keep Breaking?

NoSuchElementException after a site update, StaleElementReferenceException on dynamic pages, ChromeDriver version mismatches, CAPTCHA walls: a diagnostic for every recurring Selenium failure, with an error-to-fix routing table and the maintenance math that decides when to stop patching.

You wrote the scraper, it ran clean for a quarter, and now the cron log is a wall of NoSuchElementException. Nothing in your code changed. That is the defining experience of Selenium in production: the script is a fixed reference to a surface that moves. This guide walks the recurring failure modes in the order you're likely to meet them, each opening with the error as you'd paste it into a chatbot, and ends with the math for deciding when patching stops being worth it.

If you want the cross-tool view first, start with the anchor diagnostic.

Error-to-fix routing table

"Selenium NoSuchElementException after the website updated"

The most common Selenium failure, and the least mysterious. Your locator, find_element(By.CSS_SELECTOR, "#charge-amount") or an XPath chain, points at an element that is no longer where the DOM said it was. Two causes, easy to tell apart:

• The element renders late. Dynamic pages populate content after load. If the error is intermittent and retries succeed, it's a race. Fix it with explicit waits: WebDriverWait(driver, 10).until(EC.presence_of_element_located(...)), never time.sleep.
• The element moved. If the error started after a site deploy and is consistent, the selector drifted. A designer renamed a class, a form got restructured, a button moved into a shadow DOM. Re-inspect and update.

The patch takes minutes. The pattern is the expensive part: enterprise portals redesign meaningfully 1 to 3 times a year, and each break costs 4 to 16 hours of dev time once you count diagnosis, the fix, and re-validation. Prefer IDs and data-* attributes over positional XPath to lower the frequency, but understand what you're doing: reducing the blast radius of a dependency you cannot remove. The page's layout is not an API contract, and no locator strategy makes it one.

"Selenium StaleElementReferenceException on a dynamic page"

You found the element, then the framework re-rendered the DOM, and the reference you're holding points at a node that no longer exists. React, Vue, and Angular portals do this constantly; a table refresh or a websocket update is enough.

Short-term fixes: re-find the element immediately before each interaction, wrap interactions in retry logic, and wait for the page to reach a stable state before reading. All of these amount to the same concession: you are polling a surface that changes under you. The backend request that populated that table carried the same data in one stable JSON payload. That asymmetry is the core of the Supergood vs Selenium comparison.

"Selenium WebDriverException: ChromeDriver only supports Chrome version..."

Chrome auto-updated overnight, ChromeDriver didn't, and every session on the box now fails at startup. Selenium Manager (bundled since 4.6) mostly automates driver resolution now, and pinning browser versions in CI helps, but the failure class persists: your automation has a runtime dependency on a browser binary that a third party updates on their schedule. Every dependency bump is a potential outage that has nothing to do with your code or the target site. An API client has one dependency: an HTTP library.

"Selenium bot detection / CAPTCHA blocking my scraper"

Headless Chrome announces itself: navigator.webdriver is true, plugin lists are empty, canvas and WebGL fingerprints are distinctive, and detection vendors like Cloudflare and DataDome cross-reference all of it with IP reputation and behavioral signals. The escalation path is familiar: stealth patches, then residential proxies, then human-like mouse movement, and each rung costs more and breaks more often, because the detection side updates faster than the patch libraries.

Be honest about which situation you're in. If you're scraping a site you have no relationship with, that arms race is the business you're in. But most enterprise Selenium runs against portals you or your customers legitimately log into: a payer portal, a property management system, a supplier dashboard. There, disguising a browser is the wrong layer entirely. The portal's own frontend talks to backend endpoints over authenticated requests; automation that speaks to those endpoints directly, the way Supergood builds it, doesn't need to look human, because it isn't pretending to be one.

"My Selenium script hits an MFA screen now"

The portal added two-factor authentication, and the login flow your script has replayed for a year now dead-ends at a code prompt. Workarounds exist: session and cookie reuse, app passwords where the portal allows them, or plumbing TOTP secrets into the script. Each is fragile, and some violate the portal's terms. The MFA-in-Playwright guide covers the same landscape (the mechanics are identical for Selenium), and lands where every team lands: MFA is designed to stop exactly what a scraping script does. A managed integration layer with real service accounts, including the email and phone the MFA challenge goes to, is the version of this that doesn't break quarterly.

Latency: why Selenium is structurally slow at volume

Even a healthy Selenium script pays the UI-layer tax on every run. The browser must fully load the page, execute its JavaScript, and wait for dynamic content before a selector resolves; multi-second steps are normal, and a login-navigate-submit workflow of 30 seconds is typical. Compare the same work as backend API calls, completing in about a second:

The hours are real infrastructure: each session is a full Chromium process holding hundreds of MB of RAM, so concurrency is bounded by memory, and memory is what you rent. Hosted browser sessions floor out around $0.15 each, and a realistic multi-step workflow lands near $1.00 per completed call once retries and failures are counted. The full unit economics, including where retries take the real cost per completed action several times higher, are in the true cost of browser automation past 10,000 calls a month.

The quarterly maintenance math

One Selenium script against one actively developed portal, one quarter. Assumptions stated, arithmetic only: one selector-drift break (10 hours to diagnose, patch, and re-validate), one driver/browser version incident (2 hours), and 2 hours a month of babysitting flaky waits and retries.

Seventy-plus dev-hours a year to keep one script standing still is the recurring bill most teams never put on one line. If the portal has an official API covering your workflow, use it. If it doesn't, that's the gap Supergood fills: generated, maintained REST and MCP APIs at the network layer the portal's own frontend uses, where a UI redesign is a non-event.

Related reading

• Why your enterprise automation keeps breaking — the cross-tool anchor diagnostic
• Why does my Playwright automation keep failing? — same failure arc, Playwright's error surface
• Why does my Puppeteer script keep breaking? — the Chrome-native sibling
• Supergood vs Selenium — the direct comparison
• The true cost of browser automation past 10,000 calls a month — the cost model behind the latency table

FAQ

What do I do when the website changes and breaks my Selenium script? Re-inspect the page, update the selectors that moved, and prefer stable attributes over brittle XPath chains. Then count the pattern: portals redesign 1 to 3 times a year at 4 to 16 dev-hours per break. If this is the second or third time, the durable fix is moving the workflow onto an API integration with no selectors to drift.

Why does Selenium throw NoSuchElementException? The element isn't in the DOM when Selenium looks: either the page changed (drift) or it hasn't rendered yet (a timing race). Explicit waits fix races; updated locators fix drift; nothing fixes recurrence except removing the UI dependency.

Why is my Selenium scraper so slow? Every step pays for a full page render before a selector resolves. At 10,000 monthly runs, a 30-second workflow consumes ~83 hours of browser time and a Chromium process worth of memory per session. The equivalent API call is sub-second.

How do I stop Selenium being detected as a bot? Stealth patches and proxies buy time, but detection updates faster than patches. For portals you legitimately use, authenticated network-layer access is the stable answer, not a better disguise.

Should I keep fixing it or replace it? Patch a one-off. Replace the approach when breaks recur, volume passes a few thousand runs a month, or MFA and bot detection arrive. At that point the maintenance is structural, and an API integration removes the failure class.