Android Automation Script OCR Tutorial: On-Device Offline Text Recognition with Free PPOCR Models

1. Why Scripts Need to “Read Text”

The essence of an automation script is “watching the screen, making decisions, and tapping buttons on the user’s behalf.” Most of the time, a script can “understand” the UI through the widget tree: the system accessibility service parses the current screen into a tree of nodes, where every button, input field, and text has an id, text, description, and position. Finding a widget by its text and then tapping it is the most stable approach—and the one least affected by screen resolution.

But the widget tree is not omnipotent. There is a whole category of text that the widget tree cannot read at all:

Scenario	Why the widget tree cannot read it	How OCR handles it
Game popups / announcements	The game engine “draws” the scene; there are no standard widget nodes	Recognize the text inside the popup image directly
Custom-drawn text (Canvas)	Text is drawn on the canvas, not present in the widget tree	Recognize the text on the canvas
Text baked into images / promo graphics	The text is part of the image	Run OCR on the image region
CAPTCHA-like content	Designed to resist machine reading; usually no text interface is exposed	Read with OCR (whether it may be used depends on the compliance of the scenario)

For content like this, the only way to read it is to treat the screen as an image—which is exactly what OCR (Optical Character Recognition) does: feed it an image, get back the text and its positions.

There is also an easily overlooked related scenario: in Bluetooth HID / OTG HID running modes, the script does not depend on the accessibility service, so the widget-tree path is unavailable altogether, and OCR is almost the only way to “read text” (those modes support only image matching, OCR, and tap/swipe operations). So “being able to read text” is not a nice-to-have—it is the second pillar of automation capability.

In EasyClick, OCR is a built-in capability: root-free, supports Android 5.0 through the latest system versions, and PPOCR-V4/V5/V6 are all free with local offline recognition (OCR documentation).

2. Cloud OCR vs On-Device OCR: Why Local Recognition Is the Better Choice

When people hear “OCR,” many first think of “calling some cloud OCR API”: upload the image, wait for the server to recognize it, and receive the result. That certainly works, but for automation scripts it loses on almost every dimension:

Dimension	Cloud OCR	On-device OCR (local on the phone)
Latency	Screenshot → upload → cloud queue → recognize → return, affected by network round trips	Computed locally; no network step
Privacy	Screen content is uploaded to a third-party server	Data never leaves the device
Cost	Per-request / usage-based billing	Completely free
Offline availability	Unavailable without a connection	Keeps recognizing while offline
Stability	Depends on server and network quality	Depends on no external service at all

Automation scripts are high-frequency, real-time, long-running tasks—one screenshot, one recognition, thousands or tens of thousands of times a day. The cloud approach is both slower and more expensive, and it can break at any moment due to network jitter or service hiccups. On-device OCR puts the recognition engine and models directly on the phone and completes everything locally—a natural fit.

To be fair, the documentation also offers online recognition types (such as Baidu online recognition baiduOnline and EasyClick’s own PC-side paddleOcrOnline service), as well as a free PC-side PPOCR-ONNX program (runs on Windows and exposes HTTP endpoints for Android / iOS / HarmonyOS Next clients, default port 9022—see the PPOCR notes). Those suit deployments where a computer sits next to you and you need higher throughput. This article focuses on local on-device recognition, which is also how the PPOCR family is used by default in EasyClick.

3. How an OCR Engine Works: Detect + Recognize, Two Stages

OCR is not “an image magically turns into text.” It is a two-stage pipeline:

1. Detection: first find “where the text is” in the image and draw a box around each line or block of text. Think of it as “finding which spots in the room have notes stuck to them.”
2. Recognition: then “translate” the text image inside each box into a character sequence—“reading each note aloud.”

Each stage is its own neural network model (or group of models). These model files ship inside the app (the model path is configurable; if not configured, the built-in models are used), so recognition runs entirely on the phone—no internet, no image upload, no per-request billing.

PaddleOCR is the open-source OCR solution from the Baidu PaddlePaddle ecosystem, and PPOCR-V4 / V5 / V6 are its successive model versions. EasyClick ships the PaddleOCR family: the documentation confirms support for PPOCR-V4, PPOCR-V5, and PPOCR-V6 models, with V6 requiring Android 8.0 / API 26 or higher (OCR documentation).

Understanding the “two stages” also pays off in practice: when tuning parameters you will keep seeing phrases like “detection box” and “the text box did not enclose the text”—they all refer to the detection stage, and many parameters (such as padding, discussed later) exist precisely to fix problems in that stage.

4. Choosing Between PPOCR-V4 / V5 / V6

Here is the bottom line first: the system version is the primary constraint; everything else is decided by testing real results.

initOcr type	Model	Notes	Documented example since
paddleOcrOnnxV4	PPOCR-V4	ONNX implementation, full parameter set (padding, maxSideLen, text-orientation detection, etc.)	11.26+
paddleOcrOnnxV5	PPOCR-V5	ONNX implementation; parameters in the same family as V4	11.26+
paddleOcrNcnnV5	PPOCR-V5	ncnn implementation—ncnn is a neural-network inference framework optimized for mobile devices	11.28+
paddleOcrOnnxV6	PPOCR-V6	Official ppocr-sdk + ORT implementation, default modelTier=small	12.4.0+, requires Android 8.0+

Selection advice:

• Android 8.0 and above: prefer paddleOcrOnnxV6. V6 is the newest model (official ppocr-sdk + ORT), with the default tier modelTier=small; the current EasyClick release 12.4 already supports it.
• Below Android 8.0: use paddleOcrNcnnV5 or paddleOcrOnnxV5. This is the explicit recommendation in the documentation—older systems cannot run V6, so the V5 family covers them.
• The documentation gives no direct accuracy/speed comparison between V4/V5/V6, so do not agonize over “which one is definitely faster”: get it running with default parameters first, then fine-tune based on real results (speed, missed text, misrecognition).

One more thing: since EC 9.17+ OCR switched from a singleton to a multi-instance model, one script can initialize several OCR instances at once and switch between them on demand (for example, one V5 and one V6, each handling its own type of scenario) without interfering with each other.

5. Using OCR in Scripts: Four Steps

The usage pattern is fixed, just four steps: create an instance → initialize → recognize → release resources.

Step	Function	Purpose
Create instance	ocr.newOcr()	Create an OCR instance (multi-instance mode since 9.17+)
Initialize	initOcr({type: ...})	Choose the recognition engine and parameters; returns success or failure
Recognize	ocrImage(img, timeout, extra)	Recognize a screenshot or image; returns an array of results
Release	releaseAll()	Free the resources occupied by OCR

Before recognition you need startEnv() to start the automation environment and image.requestScreenCapture() to request screen-capture permission (image documentation); the example below omits that part to focus on OCR itself. Using PPOCR-V6 as the example (requires Android 8.0+):

// ① Create an OCR instance (once per script is enough)
let ocrEngine = ocr.newOcr();

// ② Initialize: type selects the recognition engine; here paddleOcrOnnxV6 (PPOCR-V6)
let config = {
    "type": "paddleOcrOnnxV6",
    "modelTier": "small",  // model tier, default small (PP-OCRv6_small)
    "numThread": 2,        // number of CPU threads
    "padding": 32,         // white border around the image, default 32
    "maxSideLen": 640      // scale by the long edge, default 640
};
if (!ocrEngine.initOcr(config)) {
    loge("OCR initialization failed: " + ocrEngine.getErrorMsg());
    exit();
}

// ③ Capture the screen and recognize (callable repeatedly with the same instance)
function ocrScreen() {
    let img = image.captureFullScreenEx();
    if (!img) {
        loge("Screenshot failed");
        return;
    }
    // Recognize the whole screen with a 20-second timeout;
    // recognition parameters can also be passed dynamically in extra, e.g. {"padding":32}
    let result = ocrEngine.ocrImage(img, 20 * 1000, {});
    if (result) {
        for (let i = 0; i < result.length; i++) {
            let item = result[i];
            logd("text: " + item.label +
                 " confidence: " + item.confidence +
                 " position: " + item.x + "," + item.y +
                 " size: " + item.width + "x" + item.height);
        }
    } else {
        logw("No text recognized");
    }
    // Recycle the image when done
    image.recycle(img);
}

ocrScreen();
ocrScreen();

// ④ Release OCR resources before exiting (or inside the setStopCallback handler)
ocrEngine.releaseAll();

ocrImage returns a JSON array, and each item looks like this (structure per the official documentation):

[
  {
    "label": "领取奖励",
    "confidence": 0.93,
    "x": 11,
    "y": 25,
    "width": 100,
    "height": 40
  }
]

• label: the recognized text content
• confidence: confidence score; the higher, the more trustworthy
• x / y / width / height: the position and size of the text

With the coordinates in hand, the script can do true “read then tap”—for example, only handle text containing “领取” (claim) with a sufficiently high confidence, and tap its center:

for (let i = 0; i < result.length; i++) {
    let item = result[i];
    if (item.label.indexOf("领取") >= 0 && item.confidence > 0.8) {
        let cx = item.x + Math.floor(item.width / 2);
        let cy = item.y + Math.floor(item.height / 2);
        clickPoint(cx, cy);
        break;
    }
}

6. Combining OCR with Image / Color Matching

OCR reads “meaning,” while image/color matching recognizes “appearance,” and each has blind spots: OCR can misread similar glyphs (especially in low-resolution images or stylized fonts), and image matching can fail due to resolution, theme, or scaling changes. The idea behind combined locating is mutual verification:

Use OCR to locate the text region first, then use image matching inside that region as a second confirmation—only act when both signals hit.

// ① First use OCR to find the text containing "确认" (confirm)
let target = null;
for (let i = 0; i < result.length; i++) {
    if (result[i].label.indexOf("确认") >= 0) {
        target = result[i];
        break;
    }
}

if (target) {
    // ② Confirm with a template image in the region near the text (template in the project res folder)
    let template = readResAutoImage("confirm_btn.png");
    // Search region: expanded outward from the text box; keep it inside the screen
    let sx = Math.max(0, target.x - 50);
    let sy = Math.max(0, target.y - 50);
    let rect = image.findImage(
        img, template,
        sx, sy,
        target.x + target.width + 50, target.y + target.height + 50,
        0.7, 0.9, 1, 5   // weak threshold, similarity, result count, matching method
    );
    if (rect && rect.length > 0) {
        let cx = parseInt((rect[0].left + rect[0].right) / 2);
        let cy = parseInt((rect[0].top + rect[0].bottom) / 2);
        clickPoint(cx, cy);
    }
}

The reverse order is common too: first confirm the button exists with image matching, then run OCR on the button region to read its text for branch decisions (reading a balance, a status, a countdown). One principle covers it all: give the “semantic judgment” to OCR and the “appearance confirmation” to image/color matching—use each where it is strongest.

7. Common Pitfalls and Parameter Tuning

First, the frequently used parameters (behavior per the official documentation):

Parameter	Purpose	Tuning direction
padding	White border added around the image to improve recognition; increase it when the text box does not fully enclose all the text	Increase when text is not fully boxed or characters are missed
maxSideLen	Overall scaling by the long edge: larger is slower but more accurate; smaller is faster but less accurate (set to 0 in V4/V5 to disable scaling)	Lower for speed, raise for accuracy
numThread	Number of CPU threads used (paddleOcrNcnnV5 defaults to 0; -1 means maximum CPU)	More threads, faster recognition, but higher CPU usage
modelTier	Model tier for V6, default small (PP-OCRv6_small)	Choose the tier offered by the documentation

Common pitfalls in practice and how to fix them:

1. Slow recognition: recognizing the whole screen is computationally heavy. Crop the screen to the text region first; lower maxSideLen; raise numThread sensibly.
2. Small text missed / characters dropped: raise maxSideLen for better accuracy (at the cost of time); padding specifically fixes “the text box did not enclose all the text.”
3. Re-initializing on every recognition: instantiation and model loading should happen once (the documentation stresses “once at the start of the script”); reuse the same instance in the recognition loop to avoid large repeated overhead.
4. Forgetting to recycle images: call image.recycle(img) after recognition, or memory keeps climbing.
5. Initialization failure—check the error first: use getErrorMsg() to get the concrete reason. If V6 initialization fails on an old device, it is usually because the system is below Android 8.0—switch to the V5 family.
6. Parameters seem to have no effect: engine-level parameters (type, etc.) are decided at initOcr time; some recognition parameters can be passed dynamically in the extra argument of ocrImage—make sure you are changing the right place.

8. FAQ

Q1: Does on-device OCR need an internet connection? A: No. PPOCR-V4/V5/V6 all run recognition locally on the phone with no dependency on cloud APIs—they keep working even when the device is offline.

Q2: Are PPOCR-V4/V5/V6 free? A: Yes, the whole family is free. Because recognition runs locally, there are also no hidden per-request or usage-based costs.

Q3: Do I need root to use OCR recognition? A: No. EasyClick works root-free for OCR and supports Android 5.0 through the latest system versions.

Q4: Can an old Android 7 phone use PPOCR-V6? A: No. V6 requires Android 8.0 (API 26) or higher. On older phones, use paddleOcrNcnnV5 or paddleOcrOnnxV5 instead.

Q5: What does the OCR result contain besides the text? A: Each result item includes label (the recognized text), confidence, and x/y/width/height (position and size), so you can locate and tap directly using the coordinates.

Q6: What can I do if OCR is slow? A: First try lowering maxSideLen (scaling by the long edge—smaller means faster), and set numThread sensibly. You can also crop the screen to the text region before recognition to reduce computation.

Q7: What if small text is missed or not fully boxed? A: Raising maxSideLen improves accuracy at the cost of more time. padding adds a white border around the image; increase it when the text box does not fully enclose all the text.

Q8: How do OCR and image/color matching relate? A: Image/color matching relies on pixel/template matching, while OCR reads text content. Combine them: use OCR to get text coordinates first, then confirm the target with image matching near those coordinates to reduce false hits.

Q9: Does OCR upload my screen content to a server? A: No. On-device recognition runs entirely on the phone and the data never leaves the device. Cloud OCR, by contrast, uploads images to a server, which is where privacy and traffic concerns come from.

Q10: How do I initialize OCR in a script? A: Create an instance with ocr.newOcr(), initialize it with initOcr({type: …}) (type can be paddleOcrOnnxV4/V5, paddleOcrNcnnV5, paddleOcrOnnxV6, etc.), recognize with ocrImage, and finally release resources with releaseAll(). See the official OCR documentation for complete examples.

---

About EasyClick: A mobile-automation AI-agent platform covering three ecosystems—root-free Android, jailbreak-free iOS, and HarmonyOS Next—offering script development, Apple cluster control, local central-control screen mirroring, and cloud control systems. → Learn about all products