1. Why Automation Needs “Eyes”
Anyone who writes automation scripts soon hits a basic question: before operating an interface, a script first has to “see” where things are.
The usual path is “reading the interface” — Android generates an accessibility node tree for the current screen, and the AccessibilityService can read it. Every button, input box, and image has an id, text, description, class name, and bounds. The script “finds controls by name,” clicks them, and life is elegant.
But “reading the interface” fails in plenty of situations:
- Game screens: content rendered by OpenGL / SurfaceView has no standard control nodes; the whole screen is nearly empty to the accessibility service.
- Custom-drawn controls: many apps draw their UI with Canvas, and the node tree contains only empty shells with no usable text or descriptions.
- Image buttons: a whole image is the button, with no readable text and no usable id.
At that point the script has only one way out: look at the screen like a human — take a screenshot, find the target in it, and operate at the found position. That is where image recognition comes in.
In one sentence: control-based positioning “reads the interface,” while image recognition “looks at the screen.” When reading fails, looking is the fallback — and often the only option for game and custom-drawn interfaces. In EasyClick, this set of “eyes” is the image module (the image series API, with the object prefix image, e.g. image.requestScreenCapture()), covering the whole chain of requesting screenshot permission, capturing the screen, finding images, finding colors, and comparing colors — all built on official system capabilities, no root required.
2. From Coordinates to Images: Why Recognition Is More Stable
The earliest automation scripts were “coordinate scripts”: record an (x, y) point on screen and click it when the time comes. Their fatal weakness is fragility — change the resolution, adjust the layout, pop up a dialog, and the coordinates are all wrong. That is the root of many “recorded scripts that run but are unstable.”
Image recognition takes a different approach: it does not care where the target is; it cares what the target looks like. Image finding locates the target itself, and the coordinates are merely a by-product of the recognition result.
That brings two direct benefits:
- Independent of resolution and layout: move the button, switch device models, change the font — as long as it still looks the same, it can be found. One set of templates often works across many device models.
- Independent of whether controls exist: game screens and custom-drawn interfaces that the node tree cannot read are still recognizable as long as they are visible.
Coordinates were not eliminated; they were demoted to “the fallback of fallbacks.” Only when a target has no recognizable color or shape features at all (for example a blank solid-color area) do you fall back to coordinates — and mature platforms support “random clicks within a coordinate area,” so each click position is not identical, reducing operation fingerprints (feature overview).
3. Image Finding: How OpenCV Template Matching Works
Image finding is the most widely used recognition technique today. Its core is one of the most classic algorithms in computer vision — template matching.
The principle fits in one sentence: prepare a small image of the target (the template), slide it across the big screen image from top to bottom and left to right, compute a similarity score at every position, and declare a hit when the score exceeds a threshold. On a hit, the target position area (Rect) on screen is returned; the center of that area is a usable click coordinate.
Where do templates come from? In the development tool, capture the target area as a small image, put it in the project res folder, and read it in the script with readResAutoImage().
Similarity is controlled by the threshold parameters:
| Parameter | Meaning | Default |
|---|---|---|
threshold |
Image similarity, a float from 0 to 1; the final hit criterion | 0.9 |
weakThreshold |
Weak threshold deciding whether each matching round continues; rounds with similarity below it are abandoned early | 0.7 |
Six matching methods (method parameter) are available, all standard OpenCV template matching methods:
| Value | Method | Description |
|---|---|---|
| 0 | TM_SQDIFF | Squared difference matching |
| 1 | TM_SQDIFF_NORMED | Normalized squared difference matching |
| 2 | TM_CCORR | Cross-correlation matching |
| 3 | TM_CCORR_NORMED | Normalized cross-correlation matching |
| 4 | TM_CCOEFF | Correlation coefficient matching |
| 5 | TM_CCOEFF_NORMED | Normalized correlation coefficient matching |
On the recognition rate, let us first align on one thing: “image recognition rate above 95%” is the officially claimed figure, referring to performance on regular, clear, static interfaces (feature overview). Actual results depend on screenshot clarity, whether the template matches the current screen, how often the interface changes, and whether thresholds are reasonable — Section 5 covers how to tune these.
A minimal working image-finding example (function names and parameter order per the official docs):
image.initOpenCV(); // initialize OpenCV before image finding (slow first time, fast after)
let template = readResAutoImage("sms.png"); // read the template image from the project res folder
let aimage = image.captureFullScreen(); // capture the current screen
// Image finding: weak threshold 0.7, similarity 0.9, find up to 21, method=5 normalized correlation coefficient
let points = image.findImage(aimage, template, 0, 0, 0, 0, 0.7, 0.9, 21, 5);
if (points && points.length > 0) {
// A Rect area is returned; click its center
let x = parseInt((points[0].left + points[0].right) / 2);
let y = parseInt((points[0].top + points[0].bottom) / 2);
clickPoint(x, y);
}
image.recycle(aimage); // always recycle images to avoid memory bloat
image.recycle(template);The image-finding family has several practical variants: findImageEx finds on the current screen with automatic capture (no manual screenshot needed); findImage2 is scaled image finding, more accurate than findImage (useful when the target is rendered scaled up or down; EC 9.41.0+); findImageByColor is transparency-aware image finding that supports transparent templates and does not need OpenCV initialization (EC 7.15.0+).
On performance, one piece of common knowledge is worth knowing: the capture itself has cost — captureScreen converts the screenshot to Bitmap, taking roughly 0–20 ms, and repeated calls within a short time (about 16 ms) return the same screenshot (image API docs). Template matching internally uses an image pyramid for acceleration (the maxLevel parameter of matchTemplate), which usually needs no manual intervention.
4. Color Finding: Single-Point, Multi-Point, and Color Blocks
Color finding is “lighter” than image finding: it does not compare whole images; it finds pixels by color — look for pixels of a given color inside a region and return their coordinates.
Single-point color finding (findColor)
Finds points in a region whose color matches the given value (optionally with a color offset) and returns an array of coordinates. Typical scenarios: solid-color UI, solid-color buttons, status indicator lights. The color format looks like 0xCDD7E9-0x101010 — the color value, then the offset tolerance. threshold is the color similarity from 0 to 1, limit caps the number of results, and orz is the search direction from 1 to 8.
The weakness of single-point color finding is obvious: it verifies only one pixel, so it misreports easily on large same-color backgrounds.
Multi-point color finding (findMultiColor)
Multi-point color finding uses one point as the “anchor” and then verifies the relative positions plus colors of several surrounding points; the hit only counts when all of them match:
// The first pixel color is the anchor; then verify the offset points
// Format: xOffset|yOffset|color[-offset], multiple points separated by commas
let firstColor = "0xDD7A5F-0x101010";
let points = "29|25|0xBB454B-0x101010,58|44|0xA6363A-0x101010";
// findMultiColorEx: multi-point color finding with automatic capture
let result = image.findMultiColorEx(firstColor, points, 0.9, 0, 0, 0, 0, 10, 1);
if (result && result.length > 0) {
clickPoint(result[0].x, result[0].y);
}A multi-point combination is like giving the target an “ID card” — the relative positions and colors of several pixels are almost impossible to coincide by chance elsewhere, so the robustness is an order of magnitude better than single-point. Typical scenarios: icon badge dots, red notifications, textured buttons.
Color comparison (cmpColor) and color block detection
Color comparison answers a different question: “at this position, do a few pixels have the colors I expect?” cmpColor / cmpColorEx verify a group of points one by one and return true only when all match — great for judging button state changes (a “Claim” button turning gray, clickable-state changes). cmpMultiColor / cmpMultiColorEx try multiple color schemes in turn and return the index of the matched group, or -1 when none matches.
Color block detection exploits the “batch return” ability of color finding: the find-color functions return all matching points in a region (limit caps the count), so you can locate contiguous color blocks and find their boundaries. For health bars and progress bars, for instance, a common technique is to find the color boundaries at both ends of the bar and estimate the remaining amount from the coordinate ratio — no full template image needed.
Also, the development tool has built-in color picking, letting you sample points and colors directly on a screenshot and generate config files. The J-series functions (findColorJ, findMultiColorJ, and so on) read parameters directly from JSON files, so you never hand-write long color strings.
5. When Recognition Fails: Tuning, Scoping, Fallbacks, and Retries
Recognition is not magic, and failure is normal. What matters is a systematic way to debug it.
Move 1: Tune the threshold
- Nothing found → lower
threshold: if 0.9 fails, try 0.85 or 0.8. - Wrong position (false positive) → raise
threshold: when similar elements abound, tighten the standard.
The color-finding threshold works the same way. Note that the threshold only controls tolerance; the template itself should be fresh and clear. After an app update changes the interface, replace the template before tuning thresholds.
Move 2: Scope the search area
Full-screen searches are slow and easily disturbed. Both image finding and color finding accept x, y, ex, ey to limit the search region — scoping the search to the small area where the target can appear is both faster and more accurate, and it also avoids false positives from similar elements elsewhere on the same screen.
Move 3: Multiple templates + wait and retry
- Multiple templates: prepare several templates for the same target (different states, different themes) and try them in turn; for multi-group color comparison,
cmpMultiColorhandles it in one call. - Wait for appearance: pages load with delay; the right pattern is “wait for the target to appear, then operate” (loop recognition with a timeout) instead of blind fixed
sleeps.image.setInitParamcan set the maximum time for find-image/find-color actions (action_timeout), returning automatically on timeout to avoid blocking.
A combined fallback example (image finding + color finding)
Recognition methods are often mixed: try image finding first, fall back to color finding, then retry. The example below chains “request screenshot permission → image finding → color finding fallback”:
function main() {
// 1. Request screenshot permission; once per script run (type 0 = auto)
let request = image.requestScreenCapture(10000, 0);
if (!request) {
loge("Failed to request screenshot permission");
exit();
}
// Wait at least 1 second after permission before capturing
sleep(1000);
// 2. Image finding: find the "Start Task" button template (start.png in res folder)
image.initOpenCV();
let template = readResAutoImage("start.png");
// findImageEx: automatic capture; weak threshold 0.7, similarity 0.9, find 1
let points = image.findImageEx(template, 0, 0, 0, 0, 0.7, 0.9, 1, 5);
if (points && points.length > 0) {
// A Rect area is returned; click its center
let x = parseInt((points[0].left + points[0].right) / 2);
let y = parseInt((points[0].top + points[0].bottom) / 2);
clickPoint(x, y);
} else {
// 3. Image finding failed; fall back to finding the red badge (color with offset tolerance)
let redPoints = image.findColorEx("0xFF3B30-0x101010", 0.9, 0, 0, 0, 0, 10, 1);
if (redPoints && redPoints.length > 0) {
clickPoint(redPoints[0].x, redPoints[0].y);
} else {
loge("Neither image finding nor color finding hit; retry after waiting");
}
}
image.recycle(template);
}
main();Troubleshooting quick reference
| Symptom | Common cause | Countermeasure |
|---|---|---|
| Image not found | Target not visible yet / template mismatch / threshold too high | Wait for the target; use a fresh template; lower threshold |
| Wrong position found | Many similar elements / threshold too low | Raise threshold; scope the search area; switch to multi-point color finding |
| Recognition is slow | Full-screen search / repeated manual captures | Limit x,y,ex,ey; use Ex-series auto-capture functions |
| Memory grows in long runs | Image objects not recycled | Call image.recycle(img) immediately after use |
Three screenshot facts to finish this section: in accessibility mode, the first requestScreenCapture call pops up the system authorization dialog — choose “Always allow”; in proxy mode, screenshots need no permission, so proxy mode is recommended for long unattended runs (image API docs); and wait about 1 second after permission is granted before capturing.
6. Lowering Operation Fingerprints: Randomized Clicks Within a Coordinate Area
Automation has to face a practical reality: machine behavior has fingerprints. Clicking the same pixel every time, at fixed intervals, in a fixed order, is easily recognized as non-human operation in batch scenarios.
The most practical way to lower fingerprints is to randomize the click position within the target area — EasyClick supports random clicks within coordinate areas for all click actions (feature overview). This combines naturally with image recognition: image finding already returns an area (Rect), so pick a random point inside it:
// Pick a random point inside the target area so every click position differs
let x = parseInt(points[0].left + Math.random() * (points[0].right - points[0].left));
let y = parseInt(points[0].top + Math.random() * (points[0].bottom - points[0].top));
clickPoint(x, y);One clarification: every recognition and operation technique in this article is built on official system capabilities (the accessibility service, ADB, screenshot APIs, and so on) and is legal in itself; the risk always comes from the use case — always keep your usage compliant, and treat technical means as tools.
7. Image Recognition API Quick Reference
All functions below come from the image API documentation; names and parameters are subject to that documentation:
| Category | Function | Purpose |
|---|---|---|
| Screenshot permission | image.requestScreenCapture(timeout, type) |
Request screen capture permission; type: 0 auto / 1 authorized / 2 no permission needed (requires proxy mode) |
| Capture | image.captureScreen / image.captureFullScreen / image.captureFullScreenEx |
Capture the screen / full screen / full screen extended; returns an image object |
| Capture | image.captureScreenshot / image.captureScreenSurface |
Proxy-mode SurfaceControl capture (EC Android 12.4.0+) |
| Capture | image.captureToFile |
Capture the screen and save as a PNG file |
| Image finding | image.findImage(image, template, …) |
Template matching on an image; returns position areas |
| Image finding | image.findImageEx(template, …) |
Image finding with automatic capture |
| Image finding | image.findImage2(image, template, …) |
Scaled image finding, more accurate (EC 9.41.0+) |
| Image finding | image.findImageByColor(image, template, …) |
Transparency-aware image finding; no OpenCV initialization needed |
| Color finding | image.findColor(image, color, threshold, …) |
Single-point color finding; returns coordinate array |
| Color finding | image.findColorEx(color, …) |
Single-point color finding with automatic capture |
| Color finding | image.findMultiColor(image, firstColor, points, …) |
Multi-point color finding |
| Color finding | image.findMultiColorEx(firstColor, points, …) |
Multi-point color finding with automatic capture |
| Color comparison | image.cmpColor / image.cmpColorEx |
Single-point or multi-point comparison; true only when all match |
| Color comparison | image.cmpMultiColor / image.cmpMultiColorEx |
Multi-group comparison; returns the index of the matched group (-1 if none) |
| Find non-color | image.findNotColor |
Find points whose color does NOT match the given color |
| OpenCV | image.initOpenCV() |
Initialize the OpenCV library (call before image finding) |
| OpenCV | image.matchTemplate / image.matchTemplateEx |
Template matching wrapper (weak/strong dual thresholds) |
| OpenCV | image.useOpencvMat(1) |
Switch to Mat storage, faster with less memory (EC 10.18.0+) |
| Resource management | image.recycle(img) / image.recycleAllImage() |
Recycle images to prevent memory bloat |
Two rules of thumb for the naming: Ex suffix = automatic capture version (no manual screenshot); J suffix = parameters read from a JSON file (findColorJ, findMultiColorJ, and so on — config files generated by the development tool).
8. Scenario Quick Reference Table
| Your scenario | Recommended tool | Why |
|---|---|---|
| Game screen / custom-drawn UI (no controls) | Image finding findImageEx + multi-point color finding |
Template matching locates non-standard controls |
| Solid-color buttons / status points | Single-point color finding findColorEx |
Vivid colors, fastest execution |
| Icon badges / red dots / complex icons | Multi-point color finding findMultiColorEx |
Combined signature resists interference |
| Judge whether a button is clickable | Color comparison cmpColorEx |
True only when the whole group matches |
| Health bars / progress bars | Color finding on boundaries, estimate by ratio | No full template image needed |
| Target may be rendered scaled | findImage2 scaled image finding |
More accurate than findImage |
| Read text on screen | OCR (PPOCR-V4/V5/V6, free and fully offline) | Text content is an OCR job |
| Recognize objects (monsters, items, characters) | YOLO object detection (ncnn, mobile-optimized, docs) | Neural network locates non-text objects |
| Long unattended runs | Proxy mode + randomized clicks in areas | Proxy-mode screenshots need no permission |
For the full recognition stack (image/color finding, OCR, YOLO) and run-mode selection, see the Android automation technology stack overview.
9. FAQ
Q1: What is the difference between image finding and color finding, and when should I use which? A: Image finding is based on OpenCV template matching and recognizes “how the whole image looks,” suiting targets with complex features (image buttons, icons, game screen elements). Color finding recognizes “color features,” suiting targets with vivid colors and simple structure (solid-color buttons, badge dots, health bar ends), and it runs faster. Complex scenarios often combine both.
Q2: What does the 95%+ OpenCV recognition rate mean? A: The 95%+ figure is the officially claimed recognition rate, referring to performance on regular, clear, static interfaces. Actual results depend on screenshot clarity, template quality, how often the interface changes, and threshold settings. Failures can be improved by lowering thresholds, scoping the search area, preparing fallback templates, and retrying.
Q3: How should I tune the image-finding similarity threshold?
A: threshold is a float from 0 to 1, defaulting to 0.9. Lower it when nothing is found (for example 0.85 or 0.8), raise it on false positives. weakThreshold (default 0.7) controls whether each matching round continues, and can generally be left at its default.
Q4: How do I choose between single-point and multi-point color finding? A: Single-point color finding verifies only one pixel color, which misreports easily on large same-color backgrounds. Multi-point color finding uses the first pixel color as the anchor, then verifies the relative positions and colors of surrounding pixels, making the combined signature far more unique and robust. For badge dots, red notifications, and similar targets, multi-point color finding is recommended.
Q5: Does image recognition require root? A: No. Screenshots, recognition, and clicking all rely on official system capabilities and work without root. EasyClick supports Android 5.0 and above.
Q6: How do I click after a target is found? A: Image finding returns a position area, so you can click its center or a random point inside it. Color finding returns an array of coordinate points, so you click the corresponding x/y. To reduce operation fingerprints, it is recommended to randomize the click position within the target area.
Q7: Why does image finding sometimes fail? A: Common causes: the target has not appeared yet (loading delay), the template does not match the actual screen (app updates, theme changes, scaling), the threshold is too high, or the search area is wrong. Countermeasures: wait for the target to appear, prepare multiple templates, lower the threshold, and scope the search area.
Q8: What permissions are needed for screenshots?
A: In accessibility mode, the first requestScreenCapture call pops up a system authorization dialog; choose “Always allow.” In proxy mode, screenshots need no permission, so proxy mode is recommended for long unattended runs. Wait about 1 second after permission is granted before capturing.
Q9: What if image finding or color finding is slow?
A: Scope the search to the small area where the target may appear instead of searching the whole screen. Use the Ex-series auto-screenshot functions to avoid repeated manual captures. Recycle images promptly. Call initOpenCV before image finding. For maximum performance, useOpencvMat switches to the Mat storage format, which is faster and uses less memory.
Q10: Why must screenshot images be recycled?
A: A capture returns an in-memory image object. If it is not recycled, it keeps occupying memory and a long-running script can blow up memory usage. Simply call image.recycle(img) after each use.
About EasyClick: A mobile automation AI agent platform covering Android (no root required), iOS (no jailbreak required), and HarmonyOS Next, offering script development, Apple cluster control, local central control & screen mirroring, and cloud control systems. → Learn about all products
Ready to build it for real?
Every approach in this article can be built on the EasyClick phone automation platform — full documentation, developer tools and cluster/cloud-control products, free to try.