Skip to content
Tutorials Center
My learning

Topics

Lesson 141 of 149 ยท Projects

Desktop Automation: Screen Capture, pyautogui, Text-to-Speech

Automate your desktop with Python. Take screenshots, record the screen to a GIF, control mouse and keyboard with pyautogui and speak text. Includes runnable stand-ins.

Most programs live inside their own window. Desktop automation is the opposite: a Python script looks at your screen, moves the mouse, presses keys and even talks. People use it to fill in the same form fifty times, to capture screenshots on a schedule, to record a short screen demo, or to read notifications aloud.

This project walks through three jobs: capturing and recording the screen, controlling the mouse and keyboard, and speaking text. There is one catch. A web page cannot control your desktop, so the real libraries (pyautogui, mss, pyttsx3) cannot run in the live editors on this site. The real code is shown read-only, and next to it you get runnable stand-ins: small classes with the same shape that work on a virtual screen drawn with Pillow. They are labelled as stand-ins every time, so you always know what is real and what is simulated. The logic you practice (loops, timing, coordinates, safety checks) is exactly the logic you will use for real.

Desktop automation is powerful and easy to get wrong. A script that clicks at fixed coordinates will happily click the wrong thing if a window moves. Test on harmless windows first, keep a way to stop it, and never automate anything you would not do by hand, such as spamming a service or bypassing its rules. Many apps forbid automated input in their terms of use.

Installing and a first look at the screen

On your computer, install the libraries in a virtual environment:

Terminal
python -m pip install pyautogui mss pyttsx3 pillow

pyautogui works on Windows, macOS and Linux (with X11). On macOS you must grant your terminal "Screen Recording" and "Accessibility" permissions in System Settings, or the calls fail silently or return black images. On Linux with Wayland, many of these tools are restricted by design.

The first things to learn are the screen size and the mouse position. Coordinates start at (0, 0) in the top-left corner; x grows to the right and y grows downward.

Python
import pyautogui

width, height = pyautogui.size()
print("screen:", width, "x", height)
print("mouse is at", pyautogui.position())

shot = pyautogui.screenshot()          # returns a Pillow Image
shot.save("screen.png")
print(shot.size, shot.getpixel((10, 10)))

Because the screenshot is a Pillow Image, everything from the Pillow lesson works on it: crop a region, convert to grayscale, resize or draw on it.

A virtual screen to practice on

The class below is a stand-in, not pyautogui. It keeps a small virtual desktop as a Pillow image and offers the same method names we need: size(), position(), moveTo(), click(), write() and screenshot(). Buttons on the virtual screen react to clicks, and typed text appears in a text box. This lets you write and test automation logic with no real mouse at all.

Python Stand-in: a virtual screen with a pyautogui-style API

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

With real pyautogui the same script would look like this (read-only):

Python
import pyautogui

pyautogui.click(150, 45)               # move and click
pyautogui.write("hello from Python", interval=0.05)
pyautogui.click(60, 170)
pyautogui.hotkey("ctrl", "s")          # keyboard shortcut (Cmd on macOS)
pyautogui.press("enter")
pyautogui.scroll(-300)                 # scroll down
pyautogui.moveTo(200, 200, duration=0.5)   # glide instead of jump

Notice what the stand-in taught you: if the text box does not have focus, typing is lost, and there is no error to warn you. Real automation has the same trap. Add small pauses, click to focus first, and check the result (for example with a screenshot) before moving on.

Safety: fail-safes and pauses

pyautogui has two built-in protections you should leave switched on.

  • The fail-safe. If you slam the mouse into any corner of the screen (top-left by default), pyautogui raises pyautogui.FailSafeException and stops. It is your emergency brake when a script goes wrong. Never set pyautogui.FAILSAFE = False casually.
  • The pause. pyautogui.PAUSE = 0.5 inserts half a second after every call. That gives slow programs time to react and gives you time to notice a problem.

A good script also validates coordinates before acting, limits the number of repetitions, and logs what it does. The stand-in already did the first one by raising a RuntimeError for a point outside the screen. The next TryIt adds a pause, a repetition limit and a stop switch to the loop of a task.

Python Repeat a task safely: pauses, limits and a stop switch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Screen capture and recording

A screen recorder is a loop: grab a frame, store it, wait until the next frame is due, repeat. Everything interesting is in the timing and in what you do with the frames.

Real capture. mss is fast and cross-platform; pyautogui.screenshot() is simpler but slower.

Python
import time
import mss
from PIL import Image

frames = []
with mss.mss() as grabber:
    monitor = grabber.monitors[1]               # the main monitor
    end = time.time() + 5                       # record for 5 seconds
    while time.time() < end:
        raw = grabber.grab(monitor)
        frames.append(Image.frombytes("RGB", raw.size, raw.bgra, "raw", "BGRX"))
        time.sleep(0.1)                         # about 10 frames per second

frames[0].save("demo.gif", save_all=True, append_images=frames[1:],
               duration=100, loop=0)

For real video files, a typical program writes frames with OpenCV's cv2.VideoWriter (convert each Pillow frame to a NumPy array first, as in the NumPy lesson, and swap RGB to BGR).

The next TryIt is a stand-in recorder. Its grab() draws a moving square instead of reading your monitor, but the recording loop, the frame timing and the animated GIF are all genuine Pillow code, so this part is a faithful rehearsal.

Python Stand-in: record frames and save an animated GIF

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the line next_due += interval. Scheduling the next frame from the plan rather than from "now" keeps the average frame rate steady even if one frame is slow. Sleeping for a fixed amount after each grab would make the recording drift slower and slower.

Two practical tips for recorders: record a region instead of the whole screen when you can (smaller frames are faster to grab and save), and be careful with GIFs, which get huge quickly. For anything longer than a few seconds, write a real video file.

Text-to-speech

Reading text aloud is useful for reminders, accessibility and fun demos. The offline library pyttsx3 talks to the speech engine already installed in your operating system (SAPI5 on Windows, NSSpeechSynthesizer on macOS, eSpeak on Linux):

Python
import pyttsx3

engine = pyttsx3.init()
engine.setProperty("rate", 170)          # words per minute
engine.setProperty("volume", 0.9)        # 0.0 to 1.0

voices = engine.getProperty("voices")    # the voices installed on this computer
engine.setProperty("voice", voices[0].id)

engine.say("Your report is ready.")
engine.say("Have a nice day.")
engine.runAndWait()                       # blocks until everything has been spoken

say() only queues text; nothing is heard until runAndWait(). The available voices and their quality differ from computer to computer. Online services (cloud text-to-speech APIs, or the gTTS package) often sound better but need internet access and usually an account.

There is nothing to hear in a browser sandbox, so this stand-in does the part you can test: it splits text into sentences, queues them like say() does, and estimates how long speaking would take at a given rate.

Python Stand-in: a speech queue with a duration estimate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Putting it together: a small desktop helper

A realistic helper combines all three jobs. This outline watches a screen region, takes a screenshot when something happens, and announces it. It is read-only because it needs your real desktop:

Python
import time
import pyautogui
import pyttsx3

engine = pyttsx3.init()
pyautogui.PAUSE = 0.2
region = (0, 0, 400, 300)                  # left, top, width, height

def announce(message):
    engine.say(message)
    engine.runAndWait()

previous = pyautogui.screenshot(region=region)
for _ in range(60):                         # watch for about a minute
    time.sleep(1)
    current = pyautogui.screenshot(region=region)
    if current.tobytes() != previous.tobytes():
        current.save(f"change-{int(time.time())}.png")
        announce("The screen changed.")
        previous = current

The loop is bounded (60 rounds), the region keeps screenshots small, and the saved file names carry a timestamp so nothing is overwritten. Comparing raw bytes is the simplest possible change detector; for noisy screens you would compare a measured difference between the two images against a threshold instead. Those are the same habits you used in the stand-ins above.

Common mistakes

  • Hard-coded coordinates. Another screen size, scaling setting or window position breaks the script. Find things with pyautogui.locateOnScreen("button.png") (an image match) or use the application's own API when it has one.
  • No pauses. Programs need time to react. Without delays, the script types into windows that have not opened yet.
  • Disabling the fail-safe. Leave pyautogui.FAILSAFE on while developing.
  • Unbounded loops. Always count rounds or check a deadline.
  • Forgetting display scaling. On high-DPI screens (for example 150% scaling), screenshot pixels and mouse coordinates may differ. Check pyautogui.size() against screenshot().size.
  • Automating sensitive windows. Never script password entry or payments, and never leave a script running unattended near private data.
  • Expecting it to work on a server. Machines without a display (CI runners, most cloud servers) have no desktop to control.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. In screen coordinates, where is the point (0, 0)?

    Screen coordinates start in the top-left corner. x increases to the right and y increases downward.
  2. 2. What is the purpose of pyautogui's fail-safe?

    The fail-safe is an emergency brake. Slamming the mouse into a corner raises FailSafeException, so you can stop a script that is doing the wrong thing.
  3. 3. Why does the recorder loop add interval to next_due instead of sleeping for interval after each frame?

    Scheduling from the plan means a late frame is followed by a shorter wait, so errors do not accumulate. A fixed sleep after each grab makes the recording drift slower.
  4. 4. In pyttsx3, when is the queued text actually spoken?

    say() only adds text to a queue. runAndWait() processes the queue and blocks until everything has been spoken.

Summary

  • Desktop automation means a script observes the screen, controls mouse and keyboard and can speak; the real libraries are pyautogui, mss and pyttsx3, which need a real desktop and cannot run in the browser.
  • Screen coordinates start at the top-left; screenshots are Pillow images, so the Pillow lesson applies directly.
  • Typing without focus is silently lost, so click first, pause, and verify results.
  • Keep the fail-safe on, use pauses, bound every loop and validate coordinates.
  • A recorder is a timed loop: grab, store, schedule the next frame from the plan, then save as an animated GIF or a video file.
  • pyttsx3 queues text with say() and speaks it with runAndWait(); voices vary by computer.
  • The runnable examples here are stand-ins on a virtual screen, labelled as such; the logic carries over to the real libraries.

Next: Tic-Tac-Toe, another project, this time with game logic you can fully test.