Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 37 ยท Locators and Gestures

W3C Actions and Touch Gestures

Perform taps, swipes, drags and multi-finger gestures with the W3C Actions API: pointer sources, move, down, up and pause, and a builder that shows the JSON.

Real users do more than click. They swipe a carousel, pull to refresh, drag a slider, long press an item and pinch an image. Appium 2 handles all of these through the W3C Actions API, a low-level description of pointer movements over time. The older TouchAction and MultiAction classes were removed in favor of it, so tutorials that use TouchAction are out of date.

This lesson explains how Actions work, shows tap, swipe and drag recipes, and lets you build the JSON payloads in a stand-in so the structure is no longer mysterious.

The model: input sources and ticks

An action sequence is a list of input sources. For touch we use one or more sources of type pointer with pointerType: "touch". Each source has a list of actions that happen in ticks: tick 1 contains the first action of every source, tick 2 the second, and so on. Actions available for a pointer:

Action Meaning
pointerMove Move to x, y (viewport coordinates, or relative to an element or the current position) over duration ms
pointerDown Press (finger touches the screen)
pointerUp Release
pause Wait duration ms
pointerCancel Abort the pointer interaction

A single-finger tap is therefore: move to the point, press down, release. A long press adds a pause between down and up. A swipe is: move to the start, press, move to the end over some duration, release.

The wire format

This is the JSON of a swipe from the lower to the upper part of a 1080 by 2400 screen:

JSON
{
  "actions": [
    {
      "type": "pointer",
      "id": "finger1",
      "parameters": { "pointerType": "touch" },
      "actions": [
        { "type": "pointerMove", "duration": 0, "x": 540, "y": 1800 },
        { "type": "pointerDown", "button": 0 },
        { "type": "pause", "duration": 100 },
        { "type": "pointerMove", "duration": 600, "x": 540, "y": 600 },
        { "type": "pointerUp", "button": 0 }
      ]
    }
  ]
}

It is sent with POST /session/{id}/actions. Because the whole gesture is a single command, it is far more reliable than several separate commands.

The Python client

The Python client wraps this in ActionChains from Selenium with a touch pointer:

Python
from selenium.webdriver.common.actions import interaction
from selenium.webdriver.common.actions.action_builder import ActionBuilder
from selenium.webdriver.common.actions.pointer_input import PointerInput

finger = PointerInput(interaction.POINTER_TOUCH, "finger1")
actions = ActionBuilder(driver, mouse=finger)

actions.pointer_action.move_to_location(540, 1800)
actions.pointer_action.pointer_down()
actions.pointer_action.pause(0.1)
actions.pointer_action.move_to_location(540, 600)
actions.pointer_action.pointer_up()
actions.perform()

(Class and method names vary slightly between Selenium and client versions; check your installed versions.) In Java you use PointerInput and Sequence, in JavaScript clients driver.action('pointer') or performActions.

Coordinates: do not hard code pixels

Different devices have different screen sizes. Compute coordinates from the window size:

Python
size = driver.get_window_size()          # {'width': 1080, 'height': 2400}
start_x = size["width"] // 2
start_y = int(size["height"] * 0.75)
end_y = int(size["height"] * 0.25)

Avoid the very edges: swiping from the extreme left edge can trigger the system's back gesture, and swiping from the bottom edge goes home. Stay inside a safe margin (10 to 15 percent).

Common gestures

  • Tap at a point: move, down, up. Often needed for elements that cannot be located.
  • Long press: move, down, pause 1000 ms, up.
  • Double tap: two tap sequences with a short pause between.
  • Swipe / scroll: move, down, move with duration, up. Longer duration means slower, gentler movement; very fast flicks cause momentum.
  • Drag and drop: move to source, down, pause, move to target over a longer duration, up.
  • Pinch / zoom: two pointer sources moving in opposite directions at the same time.
  • Pull to refresh: swipe down from near the top of a scrollable list.

Multi-touch

Two fingers are two input sources. Both have actions in the same ticks, so they move simultaneously:

JSON
{
  "actions": [
    { "type": "pointer", "id": "f1", "parameters": { "pointerType": "touch" },
      "actions": [
        { "type": "pointerMove", "duration": 0, "x": 500, "y": 1000 },
        { "type": "pointerDown", "button": 0 },
        { "type": "pointerMove", "duration": 500, "x": 300, "y": 800 },
        { "type": "pointerUp", "button": 0 } ] },
    { "type": "pointer", "id": "f2", "parameters": { "pointerType": "touch" },
      "actions": [
        { "type": "pointerMove", "duration": 0, "x": 580, "y": 1000 },
        { "type": "pointerDown", "button": 0 },
        { "type": "pointerMove", "duration": 500, "x": 780, "y": 1200 },
        { "type": "pointerUp", "button": 0 } ] }
  ]
}

Whether an app responds to synthetic pinch gestures depends on the app and driver; the drivers' mobile: pinch commands (next lesson) are often more reliable.

Build action payloads in a stand-in

The stand-in below builds the JSON for tap, long press and swipe, and computes safe swipe coordinates from a screen size. Output is deterministic.

JS Stand-in: build W3C action payloads

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Simulate a swipe and check its distance

A swipe that is too short will not scroll, and one that is too fast may fling. The stand-in estimates a swipe's distance and speed, and warns about edge starts, which are checks you can apply in a helper before sending the actions.

JS Stand-in: swipe sanity checks

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which API replaced TouchAction in Appium 2?

    TouchAction and MultiAction were removed; the W3C Actions API (pointer input sources) is the standard.
  2. 2. What is the action sequence for a long press?

    You press, hold for the pause, then release.
  3. 3. How do you do a two-finger pinch?

    Each finger is its own input source, and their actions run simultaneously per tick.
  4. 4. Why compute swipe coordinates from the window size?

    Proportional coordinates work across phones and tablets; avoid the screen edges too.

Summary

  • W3C Actions describe pointer input sources, each with move, down, up and pause actions in ticks.
  • Tap, long press, swipe and drag are different sequences of the same actions.
  • Multi-touch uses several input sources acting simultaneously.
  • Calculate coordinates from the screen size and keep away from the edges.
  • TouchAction is gone; use Actions or the driver's mobile: gesture commands.

Next: Scroll, Swipe and Long Press with Mobile Commands