Jetpack Logo Jetpack / API Reference ← Back to Home

Jetpack Scripting API

Scripts are written in Lua and run on-device. They can simulate touches, read screen pixels, find colors/images, perform OCR, make HTTP requests, and interact with the user via dialogs.

All coordinates are in physical pixels (logical pixels × screen scale). Use getScreenResolution() to get the full physical resolution. Scripts are stored on the device and managed via the built-in web dashboard (port 5555) or the on-device editor and player.

Touch & Input

tap(x, y)

Tap a single point on screen.

tap(500, 800)

sendTouchEvent(touches)

Send raw touch events for multi-touch, swipes, and drags.

-- Swipe from (500,1000) to (500,500)
sendTouchEvent { {TOUCH_PHASE.DOWN, 0, 500, 1000} }
usleep(16000)
for y = 1000, 500, -50 do
    sendTouchEvent { {TOUCH_PHASE.MOVE, 0, 500, y} }
    usleep(16000)
end
sendTouchEvent { {TOUCH_PHASE.UP, 0, 500, 500} }
-- Two-finger pinch
sendTouchEvent {
    {TOUCH_PHASE.DOWN, 0, 400, 600},
    {TOUCH_PHASE.DOWN, 1, 600, 600}
}
usleep(16000)
sendTouchEvent {
    {TOUCH_PHASE.MOVE, 0, 300, 600},
    {TOUCH_PHASE.MOVE, 1, 700, 600}
}
usleep(16000)
sendTouchEvent {
    {TOUCH_PHASE.UP, 0, 300, 600},
    {TOUCH_PHASE.UP, 1, 700, 600}
}

inputText(text)

Set text on the currently focused text field or text view.

inputText("hello world")

Gestures

High-level gesture helpers built on top of sendTouchEvent. These handle the touch event sequences for you.

swipe(x1, y1, x2, y2, duration?)

Perform a swipe gesture from one point to another.

-- Swipe up
local w, h = getScreenResolution()
swipe(w/2, h*0.7, w/2, h*0.3, 300000)

-- Quick swipe right
swipe(100, 500, 900, 500, 200000)

longPress(x, y, duration?)

Perform a long press at a point.

longPress(500, 800, 2000000)  -- 2 second long press

pinch(centerX, centerY, startRadius, endRadius, duration?)

Perform a two-finger pinch gesture. Fingers move along the vertical axis.

Use startRadius < endRadius to pinch out (zoom in), or startRadius > endRadius to pinch in (zoom out).

-- Zoom in (pinch out)
pinch(w/2, h/2, 100, 300, 500000)

-- Zoom out (pinch in)
pinch(w/2, h/2, 300, 100, 500000)

Screen

getScreenResolution()

Get screen size in physical pixels.

local w, h = getScreenResolution()
log("Screen: " .. w .. "x" .. h)
-- e.g. 1170x2532 on iPhone 13

getScreenSize()

Get screen size in logical (point) pixels.

local w, h = getScreenSize()
-- e.g. 390x844 on iPhone 13

getScreenScale()

Get the screen scale factor.

local scale = getScreenScale()
-- 3.0 on iPhone 13

getOrientation()

Get current device orientation.

local orient = getOrientation()
if orient == ORIENTATION_TYPE.PORTRAIT then
    log("Portrait mode")
end

screenshot(savePath, region?)

Save a screenshot to a file.

screenshot("screen.png")
screenshot("cropped.png", {100, 200, 500, 500})

Color Detection

Colors are integers. Use getColor() to sample a color from the screen, then pass that integer to findColor(), findColors(), etc.

getColor(x, y)

Get the color of a single pixel.

local color = getColor(500, 300)
log("Color: " .. color)

getColors(locations)

Get colors at multiple points in one call (more efficient than multiple getColor calls).

local colors = getColors({{100, 200}, {300, 400}, {500, 600}})
for i, c in ipairs(colors) do
    log("Point " .. i .. ": " .. c)
end

findColor(color, count, region?, tolerance?)

Find pixels matching a single color, with optional tolerance.

-- Sample a color from a known location, then find it elsewhere
local targetColor = getColor(500, 300)
local results = findColor(targetColor, 1, nil)
if results then
    log("Found at: " .. results[1][1] .. ", " .. results[1][2])
    tap(results[1][1], results[1][2])
end

-- With tolerance (matches similar colors)
local results = findColor(targetColor, 1, nil, 30)

findColors(colors, count, region?)

Find a multi-pixel color pattern. The first entry is the anchor color (offsets 0,0). Subsequent entries define relative offsets from the anchor.

-- Sample colors from known pixel positions to build a pattern
local c1 = getColor(200, 400)  -- anchor pixel
local c2 = getColor(210, 400)  -- 10px right of anchor
local c3 = getColor(200, 405)  -- 5px below anchor

local pattern = {
    {c1, 0, 0},     -- anchor
    {c2, 10, 0},    -- 10px right of anchor
    {c3, 0, 5}      -- 5px below anchor
}
local results = findColors(pattern, 1, nil)
if results then
    tap(results[1][1], results[1][2])
end

intToRgb(color)

Convert integer color to RGB components.

local color = getColor(500, 300)
local r, g, b = intToRgb(color)
log("R=" .. r .. " G=" .. g .. " B=" .. b)

rgbToInt(r, g, b)

Convert RGB components to integer color.

local color = rgbToInt(255, 136, 0)

hexToColor(hexString)

Convert a hex color string to an integer color.

local red = hexToColor("#FF0000")
local green = hexToColor("00FF00")

colorToHex(intColor)

Convert an integer color to a hex string with # prefix.

local color = getColor(500, 300)
local hex = colorToHex(color)
log(hex)  -- e.g. "#FF0000"

Image Detection

findImage(imagePath, opts?)

Find an image on screen using a unified API with selectable matching engine.

Engine modes:
"template" — OpenCV matchTemplate (fast, exact-ish matching)
"feature" — OpenCV SURF/FLANN/homography (slower, handles rotation/scale)
"auto" — Tries template first, falls back to feature matching if not found

-- Fast template matching (default)
local results = findImage("button.png", {
    engine = "template",
    count = 1,
    threshold = 0.9
})
if results then
    tap(results[1][1], results[1][2])
end
-- Feature matching (handles rotation/scale)
local results = findImage("icon.png", {
    engine = "feature",
    count = 1,
    debug = true
})
if results then
    tap(results[1][1], results[1][2])
end
-- Auto engine with search region
local results = findImage("icon.png", {
    engine = "auto",
    count = 1,
    region = {100, 200, 500, 500},
    threshold = 0.85
})

Migration from findImage2: findImage2() has been removed. Use findImage(path, { engine = "feature", count = N }) instead.

Text / OCR

findText(region?, debug?, language?)

Read text from the screen using Apple Vision OCR.

Note: findText region uses named keys (x, y, width, height), not positional like other region parameters.

-- Read all text on screen
local text = findText(nil, false)
log(text)

-- Read text in a specific area
local text = findText({x=100, y=200, width=800, height=100}, false)
if text and string.find(text, "Continue") then
    log("Found Continue button text")
end

-- Read Japanese text with language hint
local text = findText(nil, false, "ja-JP")

findTextCoords(text, region?, language?)

Find text on screen using OCR and return the coordinates of all matches.

local results = findTextCoords("Settings", nil)
if results then
    log("Found 'Settings' at " .. results[1][1] .. ", " .. results[1][2])
    tap(results[1][1], results[1][2])
end

-- Find Chinese text with language hint
local results = findTextCoords("设置", nil, "zh-Hans")

findTextTap(text, region?)

Find text on screen and tap all matches.

findTextTap("Continue", nil)

findTextRegex(pattern, region?, language?)

Find text on screen matching a regular expression pattern using OCR.

-- Find any price like "$12.99" or "$5.00"
local results = findTextRegex("\\$\\d+\\.\\d{2}", nil)
if results then
    tap(results[1][1], results[1][2])
end

-- Find text matching "Level" followed by any number
local results = findTextRegex("Level\\s*\\d+", nil, "en-US")
if results then
    log("Found level indicator")
end

findTextBounds(text, region?, language?)

Find text on screen and return full bounding box information for each match.

-- Find "Settings" and get its full bounding box
local matches = findTextBounds("Settings", nil)
if matches then
    local m = matches[1]
    log(string.format("Text: '%s' at (%d,%d) size %dx%d center (%d,%d)",
        m.text, m.x, m.y, m.width, m.height, m.centerX, m.centerY))
    tap(m.centerX, m.centerY)
end

-- Use bounding box to define a region relative to found text
local matches = findTextBounds("Username", nil)
if matches then
    local m = matches[1]
    -- Tap to the right of the label (where the input field likely is)
    tap(m.x + m.width + 100, m.centerY)
end

Clipboard

copyText(text)

Copy text to the device clipboard.

copyText("Hello")

clipText()

Read text from the device clipboard.

local text = clipText()
log("Clipboard: " .. text)

Timing

usleep(microseconds)

Pause script execution. Respects playback speed setting.

usleep(500000)   -- 0.5 seconds
usleep(1000000)  -- 1 second
usleep(2000000)  -- 2 seconds

User Interface

alert(message)

Show a blocking alert dialog.

alert("Script finished!")

toast(message, delay?)

Show a non-blocking toast notification.

toast("Starting bot...", 3)

vibrate()

Trigger device vibration.

vibrate()

dialog(controls, enableRemember?, orientationMask?)

Show a dialog with input controls. Blocks until user submits or cancels. If canceled, the script stops.

Control types:

local label = {type=CONTROLLER_TYPE.LABEL, text="Configure your bot:"}
local nameInput = {type=CONTROLLER_TYPE.INPUT, title="Target:", key="Target", value="button"}
local speedPicker = {type=CONTROLLER_TYPE.PICKER, title="Speed:", key="Speed", value="Normal",
    options={"Slow", "Normal", "Fast"}}
local loopSwitch = {type=CONTROLLER_TYPE.SWITCH, title="Loop:", key="Loop", value=1}

dialog({label, nameInput, speedPicker, loopSwitch}, true)

log("Target: " .. nameInput.value)
log("Speed: " .. speedPicker.value)
log("Loop: " .. tostring(loopSwitch.value))

clearDialogValues(scriptPath?)

Clear remembered dialog values.

HTTP Networking

httpGet(url, headers?)

Perform an HTTP GET request.

local response, err = httpGet("https://api.example.com/data")
if err then
    log("Error: " .. err)
else
    log("Response: " .. response)
end

httpPost(url, body, headers?)

Perform an HTTP POST request.

local response, err = httpPost("https://api.example.com/submit", {
    name = "test",
    value = "123"
})

httpUpload(url, filePath, headers?)

Upload a file via HTTP.

httpDownload(url, filePath, headers?)

Download a file from a URL.

local path, err = httpDownload("https://example.com/image.png", rootDir() .. "/downloaded.png")

File I/O

Read, write, and manage files on the device. Paths are relative to the running script's directory unless an absolute path is provided.

readFile(path)

Read the entire contents of a text file.

local content, err = readFile("config.txt")
if content then
    log("File contents: " .. content)
else
    log("Error: " .. err)
end

writeFile(path, content)

Write content to a file, creating it if it doesn't exist and overwriting if it does. Creates parent directories as needed.

writeFile("data/output.txt", "Hello, Jetpack!")

appendFile(path, content)

Append content to the end of a file, creating it if it doesn't exist.

appendFile("log.txt", os.date() .. " - Bot started\n")

fileExists(path)

Check if a file exists.

if fileExists("config.txt") then
    local config = readFile("config.txt")
else
    log("No config file found, using defaults")
end

deleteFile(path)

Delete a file.

deleteFile("temp_screenshot.png")

listFiles(directory)

List files in a directory.

local files = listFiles(".")
if files then
    for i, name in ipairs(files) do
        log(name)
    end
end

JSON

Encode and decode JSON data for configuration files, API communication, and data storage.

jsonEncode(table)

Encode a Lua table as a JSON string.

local data = {name = "bot", version = 2, enabled = true}
local json = jsonEncode(data)
log(json)  -- '{"name":"bot","version":2,"enabled":true}'

-- Save settings to file
writeFile("settings.json", jsonEncode(data))

jsonDecode(jsonString)

Decode a JSON string into a Lua table.

local t = jsonDecode('{"name":"bot","value":42}')
log(t.name)   -- "bot"
log(t.value)  -- 42

-- Load settings from file
local content = readFile("settings.json")
if content then
    local settings = jsonDecode(content)
end

System

rootDir()

Get the scripts directory path.

local dir = rootDir()
log("Scripts at: " .. dir)

getSN()

Get device UUID.

getDeviceModel()

Get device model identifier.

getDeviceBattery()

Get current battery level and charging state.

local battery = getDeviceBattery()
log("Battery: " .. battery.level .. "% (" .. battery.state .. ")")

if battery.level < 20 and battery.state == "unplugged" then
    toast("Low battery! Stopping bot.", 3)
    stop()
end

log(message)

Write a message to the log file (viewable in the dashboard).

log("Bot started at step 3")

stop()

Stop the current script.

if somethingWrong then
    stop()
end

Convenience Functions

These combine basic operations for common tasks.

findColorTap(color, count, region?, tolerance?)

Find a color and tap all matches.

local targetColor = getColor(500, 300)
findColorTap(targetColor, 0, nil)  -- tap every matching pixel found

findColorsTap(colors, count, region?, tolerance?)

Find a multi-color pattern and tap all matches.

findImageTap(imagePath, count, threshold, region?, debug?, method?)

Find an image and tap all matches. Wraps findImage() with the unified options API.

findImageTap("play_button.png", 1, 0.9)

Constants

TOUCH_PHASE = {
    DOWN = 0,
    MOVE = 1,
    UP = 2
}

ORIENTATION_TYPE = {
    UNKNOWN = 0,
    PORTRAIT = 1,
    PORTRAIT_UPSIDE_DOWN = 2,
    LANDSCAPE_LEFT = 3,
    LANDSCAPE_RIGHT = 4
}

CONTROLLER_TYPE = {
    LABEL = 1,
    INPUT = 2,
    PICKER = 3,
    SWITCH = 4
}

Common Patterns

Wait for an element to appear

Write a polling loop to wait for a color, image, or text:

-- Wait for a specific color
local targetColor = getColor(500, 300)  -- sample color first
local timeout = 10000000
local elapsed = 0
while elapsed < timeout do
    local results = findColor(targetColor, 1, nil)
    if results then
        tap(results[1][1], results[1][2])
        break
    end
    usleep(500000)
    elapsed = elapsed + 500000
end

Find and tap text

-- Simple: find and tap text in one call
findTextTap("Continue", nil)

-- With more control: find coordinates first
local results = findTextCoords("Settings", nil)
if results then
    tap(results[1][1], results[1][2])
end

Swipe gesture

local w, h = getScreenResolution()
-- Swipe up
swipe(w/2, h*0.7, w/2, h*0.3, 300000)

Long press

longPress(500, 800, 2000000)  -- 2 second long press

Configurable bot with dialog

local repeatInput = {type=CONTROLLER_TYPE.INPUT, title="Repeats:", key="Repeats", value="10"}
local delayPicker = {type=CONTROLLER_TYPE.PICKER, title="Delay:", key="Delay", value="1s",
    options={"0.5s", "1s", "2s", "5s"}}

dialog({repeatInput, delayPicker}, true)

local repeats = tonumber(repeatInput.value) or 10
local delays = {["0.5s"]=500000, ["1s"]=1000000, ["2s"]=2000000, ["5s"]=5000000}
local delay = delays[delayPicker.value] or 1000000

for i = 1, repeats do
    log("Iteration " .. i .. " of " .. repeats)
    -- bot logic here
    usleep(delay)
end

toast("Done!", 3)

Save screenshots for debugging

local w, h = getScreenResolution()
log("Screen: " .. w .. "x" .. h)

screenshot("debug_before.png")
tap(500, 800)
usleep(1000000)
screenshot("debug_after.png")

Complete Bot Example

A bot that opens an app, waits for it to load, finds a button by image, taps it, and repeats.

-- Config
local targetImage = "claim_button.png"
local maxRuns = 50
local checkInterval = 1000000  -- 1 second

-- Setup
local w, h = getScreenResolution()
log("Starting bot - Screen: " .. w .. "x" .. h)
toast("Bot starting...", 2)

-- Main loop
for run = 1, maxRuns do
    log("Run " .. run .. "/" .. maxRuns)

    -- Look for the target button
    local results = findImage(targetImage, {
        engine = "template",
        count = 1,
        threshold = 0.85
    })

    if results then
        local x, y = results[1][1], results[1][2]
        log(string.format("Found button at %d, %d", x, y))
        tap(x, y)
        usleep(2000000)  -- wait 2s for action to complete
    else
        -- Button not found, scroll down and try again
        log("Button not visible, scrolling...")
        swipe(w/2, h*0.7, w/2, h*0.3, 300000)
        usleep(1000000)
    end

    usleep(checkInterval)
end

-- Save run log
writeFile("bot_log.txt", "Bot complete - " .. maxRuns .. " runs at " .. os.date())

toast("Bot finished!", 3)
log("Bot complete - " .. maxRuns .. " runs")

Important Notes

  1. Coordinates are physical pixels. Multiply logical points by screen scale. A tap at point (100, 200) on a 3x screen means physical pixel (300, 600).
  2. Colors are integers. getColor() returns an integer. Pass it directly to findColor(), findColors(), etc. Use colorToHex() to display as hex.
  3. findColors offsets are relative. The first color entry must have offsets 0, 0 (anchor). Others are pixel offsets from that anchor.
  4. findText region uses named keys ({x=N, y=N, width=N, height=N}), unlike other region parameters that use positional ({x, y, w, h}).
  5. Relative paths for images, file I/O, and other file operations resolve relative to the running script's directory.
  6. usleep is in microseconds. 1 second = 1,000,000 microseconds.
  7. Scripts run in the app's process. They have access to the current app's UI.
  8. dialog() blocks and stops the script if canceled. Always handle this gracefully.
  9. log() writes to a file viewable from the web dashboard's log panel.
  10. findImage threshold: 0.9 is a good default. Lower values (0.7-0.8) for fuzzy matching, higher (0.95+) for exact matching. Use engine = "feature" for rotation/scale-tolerant matching.
  11. Color tolerance: findColor() and findColors() accept an optional tolerance parameter (0-255). Match colors that are close but not identical.
  12. JSON encoding/decoding uses NSJSONSerialization. Sequential integer-keyed tables become JSON arrays; string-keyed tables become JSON objects.