Getting a reliable, real-time screen of an Android device is more challenging than one might think, e.g. for the purpose of training a reinforcement learning agent.
This post describes a basic approach:
adb shell screencap -p
Sadly, adb mangles binary output and has a tendency to convert \n to \r\n, which we need to manually untangle. Newer versions of adb come with an exec-out command which removes this problem:
adb exec-out screencap -p
Though note that exec-out in older versions still returns mangled output.
In any case, using this command as a real-time image feed is unfeasible. Let’s define a small helper script:
import subprocess
ADBBIN = 'adb.exe'
def run_adb(arguments, clean=False, as_str=False, print_out=False, out_file=None):
if type(arguments) == str:
arguments = arguments.split(' ')
result = subprocess.run([ADBBIN] + arguments, stdout=subprocess.PIPE)
stdout = result.stdout
if clean:
stdout = stdout.replace(b'\r\n', b'\n')
if as_str:
stdout = stdout.decode("utf-8")
if print_out:
print(stdout)
if out_file:
mode = 'w' if as_str else 'wb'
with open(out_file, mode) as file:
file.write(stdout)
return stdout
Which we can then use with e.g.
from adb import *
import tkinter as tk
from time import time
from PIL import ImageTk, Image
from io import BytesIO
window = tk.Tk()
window.title("Image")
window.geometry("360x660")
window.configure(background='grey')
panel = tk.Label(window)
panel.pack(side="bottom", fill="both", expand="yes")
previous_time = time()
frames_drawn = 0
while True:
data = run_adb('exec-out screencap -p', clean=False)
im = Image.open(BytesIO(data))
im.thumbnail((im.size[0] * .33, im.size[1] * .33), Image.ANTIALIAS)
img = ImageTk.PhotoImage(im)
panel.configure(image=img)
panel.image = img
window.update_idletasks()
window.update()
frames_drawn += 1
if time() > previous_time + 10:
print('FPS:', frames_drawn / (time() - previous_time))
previous_time = time()
frames_drawn = 0
This works, but leads to a horrible framerate (and a very hot phone):
Some devices also allow to use a screenrecord command:
adb exec-out screenrecord --output-format=h264 -
Though the stream starts to lag after a few minutes and most players will have trouble to handle this raw h264 stream. Heavy applications also cause a lot of lag and dropped frames.
The code over at https://github.com/fhorinek/adbmirror provides a solid solution, though with no support for Python 3. The shell commands to set things up is also a bit messy, but it provides a perfect starting ground to work with.
First, we need to create a local bin folder with minicap, minicap-shared and minitouch placed in there:
Next, we rewrite the scripts a bit to support Python 3. We don’t need the “rotation” apk, as I didn’t need to handle rotation support, made available at https://github.com/Macuyiko/adbmirror.
Running start-mirror.py now shows:
Device info: arm64-v8a 26 8.0.0 1080x1920
Now pushing files
[ 11%] /data/local/tmp/adbmirror/minicap
[ 22%] /data/local/tmp/adbmirror/minicap
[ 33%] /data/local/tmp/adbmirror/minicap
[ 45%] /data/local/tmp/adbmirror/minicap
[ 56%] /data/local/tmp/adbmirror/minicap
[ 67%] /data/local/tmp/adbmirror/minicap
[ 79%] /data/local/tmp/adbmirror/minicap
[ 90%] /data/local/tmp/adbmirror/minicap
[100%] /data/local/tmp/adbmirror/minicap
bin/minicap/arm64-v8a/minicap: 1 file pushed. 17.7 MB/s (580048 bytes in 0.031s)
[100%] /data/local/tmp/adbmirror/minitouch
bin/minitouch/arm64-v8a/minitouch: 1 file pushed.
[100%] /data/local/tmp/adbmirror/minicap.so
bin/minicap-shared/android-26/arm64-v8a/minicap.so: 1 file pushed. 1.4 MB/s (23592 bytes in 0.016s)
Now ready to start GUI, press ENTER when done for cleanup
Example command:
python gui.py 540x960 1080x1920 /data/local/tmp/adbmirror/
Running python gui.py 540x960 1080x1920 /data/local/tmp/adbmirror/ now provides us with a live mirror, no root required:
Touch is simulated through mouse clicks and drags. Adding extension to fetch e.g. a PIL image every 5 seconds or to automate touch is pretty easy.