Project

Benchmark

Every agent mobile tool we could run on one Mac was given the same 12 steps in the Settings app, 10 runs each, on an iPhone simulator and an Android emulator: looktap (CLI and MCP), mobile-mcp, Maestro MCP, Appium MCP, raw Appium, ios-simulator-mcp, MobileBuildMCP, raw idb and raw adb. A neutral checker decided whether each step worked. These are the totals over the steps every tool can do:

¹ The device's reaction time plus 2 s per tool call for the model's round trip.

On real phones (iPhone 14 Pro and Samsung Galaxy A06 over USB, looktap 0.4.0, all 12 steps), the looktap CLI again had the lowest agent time (87 s and 38 s, against 104–114 s and 45–89 s) and returned the least text (1.7 KB and 1.0 KB, against 8.9–191 KB). Four of the iOS tools can't drive a real iPhone at all: ios-simulator-mcp, MobileBuildMCP, idb and Maestro.

Where looktap loses:

• On raw device time, others are faster in total: ios-simulator-mcp on the iOS simulator (4.5 s against 5.2 s, with 20 calls to looktap's 8), mobile-mcp and raw Appium on the real iPhone (60 s and 62 s against 63 s), and raw Appium on the Samsung (11 s against 12 s).
• The looktap MCP waits for the screen to settle after every action, which makes it slower per action than the CLI. On the real iPhone that takes 6–11 s a tap and puts it last for agent time.
• In the simulator and emulator runs, looktap 0.2.1 couldn't stop an app or press Home on iOS; 0.3.0 added stop and home.

Method, every device's tables and what else they show are in BENCHMARK.md.

Where Appium is still the better choice: it plugs into existing WebDriver test suites and CI grids, and supports many more platforms. looktap's tests are flow files of plain commands: good for smoke tests and agent-written flows, not a replacement for a large suite in code.

#The full benchmark: Benchmark: looktap and every other agent mobile tool

Measured on one Mac (Apple M3 Pro, macOS 26.5.2): on an iPhone simulator and an Android emulator on 2026-10-16 (looktap 0.2.1), and on two real phones over USB, a Samsung Galaxy A06 on 2026-10-17 and an iPhone 14 Pro on 2026-10-18 (looktap 0.4.0). Every tool did the same steps in the same system app, 10 runs each. A neutral checker, not the tool, decided whether each step worked.

The full results for each platform, step by step, with p90s, calls, bytes and notes, are in separate reports:

• iOS: bench/results/ios.md (raw data: ios.json)
• Android: bench/results/android.md (raw data: android.json)
• Android, real phone: bench/results/android-device.md (raw data: android-device.json)
• iOS, real iPhone: bench/results/ios-device.md (raw data: ios-device.json)

#The tools

On a real iPhone, four of them can't run: ios-simulator-mcp, MobileBuildMCP and idb drive simulators only, and Maestro doesn't list the phone (its real-iPhone driver uses bundle IDs registered to Maestro's own team, so no other team can sign it). mobile-mcp reached the iPhone the way its wiki gives for real devices (go-ios and WebDriverAgent), because its own agent needs a paid team's wildcard App ID.

#Results

Devices: iPhone 14 simulator (iOS 26.0) and Pixel 9a emulator (Android 16).

App: the system Settings app (no sign-in, nothing real is changed).

Steps: launch app, read the screen, tap by label, back, tap x,y, swipe, scroll to a label, fill a field, screenshot, home, open a URL, terminate the app.

The tables below total the steps every tool on that platform can do, so the totals compare like with like: 8 on iOS, 11 on Android.

• Reaction time: until the result was on screen (medians, summed over those steps).
• Agent time: the reaction time plus 2 s for each tool call. Each call is one more model round trip, and needing fewer calls matters more to an agent than a few milliseconds per call.

#iOS (8 steps: launch, read, tap by label, back, swipe, scroll to label, fill, screenshot)

#Android (11 steps: all but terminate)

"Passed" counts every step the tool can do, terminate included. "On the device" is the peak RAM of the helpers a tool keeps inside the emulator.

#Real phones

The same steps on real phones over USB, with looktap 0.4.0, whose home and stop do the two steps 0.2.1 couldn't. Every step a tool can do is in these totals.

iPhone 14 Pro (iOS 26.0), 12 steps. The oracle reads the phone's screen over the cable (the capture device QuickTime records from) with Apple's text recognition, so it shares nothing with the tools.

Samsung Galaxy A06 (Android 16, One UI), 12 steps. The oracle is dumpsys, as on the emulator.

#What it shows

• For an agent, the looktap CLI does best on both platforms. It has the lowest agent time and the least text: 747 B for the whole iOS scenario, against 17–102 KB for the MCP servers. It needs the fewest calls on iOS (8); on Android Maestro MCP needs one fewer (11 to 12), because its fill is one call. It passed every step it can do, and it is ready in under a second.
• The raw device time is closer.
  • On iOS, ios-simulator-mcp is faster in total (4.5 s against 5.2 s): its fill takes 0.5 s against looktap's 1.8 s. But it needs 20 calls where looktap needs 8, and its swipes sometimes do nothing (7/10).
  • On Android, the looktap CLI is fastest in total. Reading the screen takes 108 ms, against 1.6–5.6 s for the others.
• The looktap MCP is slower per action than the CLI. Each action waits for the screen to settle and returns it (ls: false turns that off). That saves a call, but it costs 0.7–1.9 s an action, and on Android it makes the MCP's total 13 s against the CLI's 4.4 s. That is the clearest thing for looktap to improve.
• Screenshots:
  • The looktap MCP sends a half-size PNG (74–104 KB, about 0.9k image tokens on Android).
  • mobile-mcp sends a smaller JPEG (39–41 KB).
  • Appium and Maestro send full-size images (186–268 KB).
  • The CLIs save a file and return its path.
• looktap 0.2.1 couldn't do two of the steps in the simulator and emulator runs: it had no way to stop an app (launch restarts it), and on iOS no home button. 0.3.0 added stop and home, which the phone runs use.
• Some tools reported success when the step didn't happen.
  • On Android, Appium's terminateApp answered success, but Settings was still running 8 s later: 0/10 through Appium MCP, 2/10 raw. Maestro's stopApp did the same in 2 of 10 runs.
  • On iOS, idb's swipes often did nothing (2/10 raw, 7/10 through ios-simulator-mcp).
• On the real phones, the looktap CLI still has the lowest agent time and the least text, but its raw device time is no longer ahead on the iPhone.
  • iPhone: mobile-mcp (60 s) and raw Appium (62 s) are slightly faster in total than the looktap CLI (63 s). looktap's scroll to a label (24 s) and fill (15 s) are its slowest steps there.
  • The looktap MCP is last on the iPhone (114 s agent time): every action waits for the screen to settle, which takes 6–11 s a tap on the phone. That makes it the clearest thing for looktap to fix.
  • Two steps failed once each on the iPhone: Appium MCP's swipe left Settings in search mode, and raw Appium's terminateApp answered while Settings kept running.
  • Samsung: every tool passed every step. Raw Appium is fastest in total (11 s against the looktap CLI's 12 s), and Maestro MCP needs one call fewer (12 to 13), because its fill is one call. The looktap CLI's agent time is 38 s, against 51–89 s for the others.
• Setup and footprint:
  • Appium and Maestro need 17–29 s before the first command, and up to 660 MB of RAM on iOS.
  • On Android, mobile-mcp and Maestro peaked at 1.4–1.6 GB.
  • MobileBuildMCP's tool definitions are 224 KB, sent to the model in every session.

#How it was measured, and how fair it is

• One scenario, one checker. Every tool did the same steps. Whether a step worked was decided by a neutral oracle, not by the tool: idb ui describe-all on the iOS simulator; on a real iPhone, its screen over USB read with text recognition; on Android, dumpsys (the resumed activity, the fragments, the view tree, and the search screen's result list).
• Driven like an agent. MCP servers ran over stdio with tools/call, and CLIs ran once per step. When a tool can't act on a label, the step reads the screen first and acts on what it found, and that read counts as a call.
• Two times per step. Answered in is until the tool call returned; reaction time is until the oracle saw the result on screen.
• Same start for every tool.
  • Before each tool, the simulator or emulator was rebooted (with its window open), whatever the previous tool left running was killed, and the Mac was back at its idle load.
  • On Android, before every run, the device was also back below a load of 1.0.
  • Resets between steps used plain simctl/adb (devicectl on the iPhone) and were not timed.
  • The phones weren't rebooted (they would stop at their passcode). Instead, every helper the previous tool left behind was stopped, on the Mac and on the phone.
• CPU and RAM:
  • ps every 250 ms on the Mac, over each tool's whole process tree and the helpers it leaves running.
  • On Android, also the device's own ps once a second, for helpers inside the emulator.
  • The runner's own commands are left out.
• Re-runs only for benchmark mistakes. When a step failed because of the benchmark, not the tool, it was fixed and run again, and each report lists those re-runs with the reason. A step was never re-run because a tool failed it.
• Limits:
  • looktap's author wrote this benchmark and every adapter (in bench/), and looktap was developed on this app. Corrections from the other tools' authors are welcome.
  • It covers one system app (Settings) on a simulator, an emulator and two real phones, not a real app.
  • The Mac had other apps open; every tool waited for the same idle level before it started.

#Reproduce

# iOS: one booted simulator; Appium needs a WebDriverAgent built once (xcodebuild build-for-testing ... -derivedDataPath bench/out/wda)
BENCH_IOS_UDID=<udid> BENCH_APPIUM_HOME=<dir with appium + drivers> BENCH_WDA_DERIVED_DATA=$PWD/bench/out/wda \
  node bench/run.mjs ios 10
# Android: one emulator (BENCH_ANDROID_SERIAL, default emulator-5554)
BENCH_APPIUM_HOME=<dir with appium + drivers> node bench/run.mjs android 10
# A real iPhone over USB, unlocked, Auto-Lock off: WebDriverAgent built once for it and signed with your team
#   (xcodebuild build-for-testing -scheme WebDriverAgentRunner -destination id=<udid> -derivedDataPath bench/out/wda-device
#    -allowProvisioningUpdates DEVELOPMENT_TEAM=<team> PRODUCT_BUNDLE_IDENTIFIER=<id>), go-ios for mobile-mcp
BENCH_IOS_UDID=<udid> BENCH_TAG=device BENCH_APPIUM_HOME=<appium-mcp's install> BENCH_WDA_DERIVED_DATA=$PWD/bench/out/wda-device \
  BENCH_WDA_BUNDLE_ID=<id> BENCH_TEAM_ID=<team> BENCH_GO_IOS=<path to go-ios's ios> node bench/run.mjs ios 10
# A real Android phone over USB, unlocked
BENCH_ANDROID_SERIAL=<serial> BENCH_TAG=device node bench/run.mjs android 10
# Tables
node bench/report.mjs ios && node bench/report.mjs android && node bench/report.mjs ios-device && node bench/report.mjs android-device

The runner takes a third argument, a comma-separated list of tools (for example looktap,mobile-mcp). It also needs idb with idb-companion, Maestro, and Appium with the xcuitest and UiAutomator2 drivers; the MCP servers come through npx. A full run takes about 1.5 hours per platform.

Generated from README.md and BENCHMARK.md. Edit it there.