Android Automation Script YOLO Object Detection End-to-End: From Labeling & Training to On-Device ncnn Inference

Traditional image recognition (template matching) has a fatal flaw: if the icon changes size, color, or angle, it simply cannot be found. If you have automated Douyin likes or comment buttons, you know the pain — across different screen resolutions and app versions, template images constantly go stale.

YOLO takes a different route: train a model that “recognizes” the target instead of memorizing one image. This article covers the whole chain from zero: environment setup → data labeling → model training → export → on-device inference.

1. The Confidence Behind On-Device YOLO: The ncnn Inference Framework

First, remove the doubt: running YOLO on a phone requires no high-end hardware and no cloud compute.

EC for Android (10.15.0+) bundles the Tencent/ncnn neural network framework — an open-source framework specifically optimized for mobile devices. ncnn supports the yolov5–yolov8 family; EC uses the yolov8 model for training, inference, and detection.

On-device inference supports two practical features:

• CPU binding: choose ALL / BIG / LITTLE — bind inference to big cores for speed, or little cores to save power;
• Vulkan hardware acceleration: enable GPU compute (use_vulkan_compute = 1) for additional speed.

2. Environment Setup: Anaconda + Three Libraries

Training runs on your computer in a Python environment. Installing Anaconda is recommended (official site anaconda.com; Tsinghua mirror works well in China).

2.1 Create a virtual environment

After installing Anaconda:

1. Click Create to make a new virtual environment (name it whatever you like, e.g. yolotest), and choose Python 3.8.19;
2. Once created, click the green triangle next to yolotest, select Open Terminal — the environment name appearing in the terminal prompt (yolotest) means activation succeeded.

2.2 Install dependencies

Run in the terminal (switch to the Tsinghua mirror first if you are in China):

pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple/
pip config set install.trusted-host pypi.tuna.tsinghua.edu.cn
# the yolo library (version 0.3.1 here)
pip install yolo==0.3.1
# ultralytics (version 8.2.79 here)
pip install ultralytics==8.2.79
pip install ncnn==1.0.20240410
pip install labelimg

Verify the installs: typing yolo in the terminal should show the command help; typing labelimg should open the labeling tool.

3. Data Labeling: Draw Boxes with labelimg

Training starts with letting the model “see” the targets. Using Douyin’s like and comment buttons as an example:

3.1 Prepare the directory structure

Create a training root directory (example: E:/yolotrain):

yolotrain/
├── images/
│   ├── test/     # test images
│   ├── train/    # training images
│   └── val/      # validation images
└── labels/
    ├── train/    # training labels
    └── val/      # validation labels

Put the same batch of images into images/train, images/val, and images/test.

3.2 Label with labelimg

1. Type labelimg in the terminal to open the tool;
2. Open Dir → select the images/train folder;
3. Change Save Dir → select the labels/train folder;
4. Switch the format at the bottom-left to YOLO mode;
5. Right-click the image and select Create RectBox (or the left-side button of the same name), drag a box over the target area, and enter the class name — aixin for the like button and pinglun for the comment button. These names are used later in the script, so remember them exactly;
6. After each image, click Save, then Next Image to move on.

When done, labels/train contains classes.txt (the class-name list) and one txt label file per image. Copy labels/train to labels/val for validation use.

4. Training: One Command with yolov8s

4.1 Write the training config

Create aixin.yaml in the yolotrain directory:

path: E:/yolotrain
train: images/train
val: images/val
test: images/test
nc: 2
names: ["aixin","pinglun"]

Parameters:

Parameter	Description
path	Absolute path to the training root
train / val / test	The three image directories (relative to path)
nc	Number of classes — 2 here
names	The class-name array; the order must match your labeling, or training results suffer

4.2 Start training

In the terminal, enter the yolotrain directory and run:

yolo detect train data=e:/yolotrain/aixin.yaml model=e:/yolotrain/yolov8s.pt imgsz=640

• yolov8s.pt is the pretrained base model; it downloads automatically on first run (if it fails, download it from the ultralytics assets and place it in the folder);
• imgsz=640 is the training input size — keep it consistent when configuring the model later;
• Add epochs=100 to control the number of training epochs.

When training finishes, results are saved under runs/detect/train/, where weights/best.pt is the best trained model.

5. Export: onnx and ncnn

The trained .pt model cannot be used by the phone directly — export it to a mobile-friendly format first.

Export onnx:

yolo export model=e:/yolotrain/runs/detect/train/weights/best.pt format=onnx

Export ncnn (the pnnx component downloads during export):

yolo export model=e:/yolotrain/runs/detect/train/weights/best.pt format=ncnn

After exporting:

Format	Output files	Put on phone
onnx	best.onnx	/sdcard/
ncnn	model.ncnn.param + model.ncnn.bin	/sdcard/

6. On-Device Inference: yolov8Api in Practice

After placing the model files in the phone /sdcard/ root, initialize and detect in the EC script. Use newYolov8 for ncnn and newYolov8Onxx for onnx; the rest of the calls are nearly identical. Here is the full ncnn example:

function main() {
    // Initialize the YOLO instance
    let yolov8s = yolov8Api.newYolov8();

    // Default config: model name, input size 640, confidence 0.25, IoU 0.35,
    // bind all CPUs, no hardware acceleration, class names
    let config = yolov8s.getDefaultConfig("yolov8s-640", 640, 0.25, 0.35, "ALL", 0, [
        "aixin",
        "pinglun"
    ])
    logd("config : " + JSON.stringify(config))

    // Initialize the trained model
    let inted = yolov8s.initYoloModel(config, "/sdcard/model.ncnn.param", "/sdcard/model.ncnn.bin");
    if (inted) {
        logd("yolov8s initialized successfully");
    } else {
        logd("yolov8s init failed: " + yolov8s.getErrorMsg());
        return;
    }

    // Capture the current screen and detect
    let bitmap = image.captureScreenBitmapEx()
    // Or read a local image: image.readBitmap("/sdcard/a.png")
    let result = yolov8s.detectBitmap(bitmap, []);
    // Filter for only the pinglun class: detectBitmap(bitmap, ["pinglun"])

    if (result == null || result == "") {
        logd("yolov8s no result: " + yolov8s.getErrorMsg());
    } else {
        logd("yolov8s result: " + result);
    }

    if (bitmap != null) {
        bitmap.recycle(); // recycle the image
    }
    yolov8s.release(); // release when needed, not after every use
}

main();

Key config items:

Parameter	Description
model_name	Model name, default yolov8s-640
input_size	The imgsz used in training; keep it consistent (640)
conf	Confidence threshold (0.25); raise it on false positives
iou	IoU threshold (0.35), used in non-maximum suppression
bind_cpu	CPU binding: ALL / BIG / LITTLE
use_vulkan_compute	Hardware acceleration: 1 on / 0 off
obj_names	Class-name array, matching training

For onnx, configure with getOnnxConfig, which supports num_thread to control thread count (1–2 recommended, to avoid excessive CPU usage).

7. Common Issues

1. Anaconda cannot pick a Python version when creating an environment: usually a network problem — delete the default channels, add the Tsinghua mirror, and retry;
2. Arial.ttf download error: Ultralytics needs this font; download it and place it in C:\Users\<username>\AppData\Roaming\Ultralytics;
3. GPU training: with a discrete GPU, follow a GPU training guide for a significant speedup;
4. Poor detection accuracy: add more training data first, especially screenshots of the target on different models and resolutions.

8. Summary

Going through the whole chain, YOLO is far less mysterious than it sounds:

• Environment: Anaconda + yolo + ultralytics + labelimg — four components;
• Data: draw boxes with labelimg, remember the class names, and mirror them in the training config;
• Training: one yolo detect train command produces best.pt;
• Export: pick onnx or ncnn, drop the files into the phone /sdcard;
• Inference: yolov8Api initialization + screenshot + detectBitmap — used like any other script function.

Compared to template matching, YOLO’s value is recognizing rather than merely matching — one model detects multiple classes and tolerates size changes and mild deformation. If the targets in your automation scenario often “misbehave”, it is time to give your scripts a pair of AI eyes.