Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 35 ยท Getting Started

How WebDriver Works

See how WebDriver works: your script, a driver program and the browser exchange JSON over HTTP using sessions and element ids defined by a W3C standard.

When you write driver.get("https://example.com"), nothing magical happens inside Python or Java. Your code does not touch the browser at all. It builds a small message, sends it over HTTP to another program, and waits for an answer. Understanding that conversation takes the mystery out of Selenium and explains a lot of its behavior: why commands take milliseconds, why a missing element raises an exception, why a crashed driver leaves browsers running.

This lesson follows one request from your script to the browser and back, and builds a tiny fake driver server you can talk to over real HTTP.

Four actors

Four actors, one conversation

Your script Calls driver.get, find_element, click.
Client library The Selenium binding turns calls into HTTP requests.
Driver program chromedriver or geckodriver listens on a local port.
Browser Does the real work and replies.

Your script never talks to the browser directly.

1 / 2

The client library (the Selenium package you install) is what you import. The driver is a separate program provided by the browser vendor or project, for example ChromeDriver for Chrome and geckodriver for Firefox. It implements the W3C WebDriver standard: a specification that defines the HTTP endpoints, the JSON shapes and the error codes. Because it is a standard, any client library can talk to any compliant driver, which is why Selenium supports so many languages.

sequenceDiagram
  participant S as Your script
  participant C as Client library
  participant D as Driver (local HTTP server)
  participant B as Browser
  S->>C: driver.get("https://example.com")
  C->>D: POST /session/{id}/url<br/>{"url": "https://example.com"}
  D->>B: navigate (browser-specific control)
  B-->>D: page loaded
  D-->>C: 200 {"value": null}
  C-->>S: returns

One call becomes an HTTP request to the driver, which controls the browser and replies with JSON.

The conversation, step by step

These are the main endpoints defined by the standard. Each one is a plain HTTP request:

Your call HTTP request Reply value
webdriver.Chrome() POST /session with the requested capabilities An object with a sessionId and the capabilities the browser agreed to
driver.get(url) POST /session/{id}/url with {"url": ...} null
driver.title GET /session/{id}/title The title string
find_element(By.CSS_SELECTOR, "#a") POST /session/{id}/element with {"using": "css selector", "value": "#a"} An element reference
element.click() POST /session/{id}/element/{elementId}/click null
element.text GET /session/{id}/element/{elementId}/text The visible text
driver.quit() DELETE /session/{id} null

Three ideas matter.

Sessions. The first request creates a session. The driver launches a browser and answers with a sessionId. Every later request includes that id in its path, so one driver can control several browsers at once. quit() ends the session and closes the browser.

Element references. When you find an element, you do not receive the element. You receive an opaque id, a string the driver made up, wrapped in an object whose key is the fixed text element-6066-11e4-a52e-4f735466cecf. Later calls like click put that id in the URL. That explains the infamous stale element reference error: if the page changes and the browser forgets that element, the id no longer points at anything.

The value wrapper. Every reply from the driver is a JSON object with a single value key. For errors, the value holds an error code (such as no such element), a message and a stacktrace, and the HTTP status is 4xx or 5xx. The client library reads the code and raises the matching exception in your language.

You normally never write these requests yourself. The client library does it for you. But reading them once makes error messages and logs much easier to understand.

A stand-in: a fake driver server

The code below is real Node.js using real HTTP. The part that is a stand-in is the driver: instead of controlling a browser, it keeps a pretend page in memory. The request paths, JSON bodies and the value wrapper follow the shape of the WebDriver standard, so the conversation looks like the real one.

Node.js A tiny fake driver and client over HTTP

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

Read the output from the top. First a POST /session gets a session id. Every later path contains it. The element lookup returns an object whose single key is the standard element key, and the click uses that id. The lookup of #missing comes back with status 404 and no such element, exactly the sort of reply a real driver sends before your library raises an exception.

From error codes to exceptions

Client libraries keep a table that maps driver error codes to exception classes. The names below match what Selenium's Python binding raises. The stand-in shows the idea of that table.

JS Mapping driver errors to exceptions

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Anything the table does not know falls back to a general WebDriverException. Knowing the code in the driver's reply is useful when you read logs: the plain-text code (no such element) is the same in every language.

Why this design matters to you

  • Any language works. Anything that can send HTTP and JSON can be a client.
  • Remote is free. The driver can run on another machine. Selenium Grid is built on this: the client sends the same requests to a router instead of a local driver.
  • Every command is a round trip. Many tiny commands add up. Prefer reading a whole value in one call over many small ones.
  • The driver and the browser must match. A driver built for one browser version can fail with another. Selenium Manager takes care of that for you.
  • Always call quit(). The driver and browser are separate processes and keep running until the session ends.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When your script calls driver.get(url), where does the request go first?

    The client library sends an HTTP request to the driver program (such as ChromeDriver), which then controls the browser.
  2. 2. What does a successful find_element call give your script?

    The driver replies with an element reference id. The WebElement object in your language stores it and includes it in later requests such as click.
  3. 3. Why can an element reference become stale?

    The id points at one specific node in the page. If the page replaces or removes that node, the id no longer matches anything and the driver reports a stale element reference.
  4. 4. What shape does every WebDriver reply have?

    Success and error replies are both JSON with a single value key. For errors the value holds the error code, message and stacktrace.

Summary

  • Selenium has four actors: your script, the client library, a driver program and the browser.
  • The client and driver talk JSON over HTTP, following the W3C WebDriver standard.
  • The first request creates a session and returns a session id used in every later path.
  • Found elements are opaque references, which is why they can go stale.
  • Replies are wrapped in value; errors carry a code that the library turns into an exception class.
  • The fake server above is a stand-in that shows the shape of real requests and replies.

Now put it to use with Setup and Your First Script.