python - 捕获屏幕并找到引用图像

Question

我正在尝试在 python 中执行以下操作:

1. 截屏
2. 如果屏幕截图包含给定的引用图像(可以是 jpg 或 pgn)，获取该图像在屏幕上的坐标

更多信息:

• 引用图片不会很大(5x5 像素就足够了)
• 它应该尽可能快，因为它应该不断扫描屏幕
• 如果可能:在 Windows 和 Linux 上工作

什么是最好的方法，我可以在 python 中实现它？

编辑: 感谢 Leo Antunes，我使以下解决方案有效:

def bitmap2brg(bmp):
  w = bmp.width;
  h = bmp.height;
  a = np.empty((h, w, 3), dtype=np.uint8);

  for r in xrange(h):
    for c in xrange(w):
      v = bmp.get_color(c, r);
      a[r, c, 2] = (v >> 16) & 0xFF;
      a[r, c, 1] = (v >> 8) & 0xFF;
      a[r, c, 0] = v & 0xFF;
  return a;

def grabScreen():
  THRESHOLD = 1

  # reference image
  needle = cv2.imread('img_top_left.png')
  needle_height, needle_width, needle_channels = needle.shape

  # Grabbing with autopy
  screen = autopy.bitmap.capture_screen()
  haystack = bitmap2brg(screen)

  # work through the frame looking for matches above a certain THRESHOLD
  # and mark them with a green circle
  matches = 0
  for pt in np.transpose(np.where(cv2.matchTemplate(haystack, needle, cv2.TM_CCOEFF_NORMED) >= THRESHOLD)):
      cv2.circle(haystack, (pt[1] + needle_width/2, pt[0] + needle_height/2), 10, (0,255,0))
      matches += 1

  # display the altered frame
  print "Number of matches: {}".format(matches)
  cv2.imshow('matches', haystack)
  if cv2.waitKey(0) & 0xFF == ord('q'):
        cv2.destroyAllWindows()

Answer 1

更新:有了 OP 自己的一个伟大发现 (autopy)，“正确”的解决方案变得非常简单:

import autopy

needle = autopy.bitmap.Bitmap.open('needle.png')

while True:
        haystack = autopy.bitmap.capture_screen()
        found = haystack.find_every_bitmap(needle)
        print(found)

下面的原始次优答案:

---

您可以使用 OpenCV 相对轻松地对视频执行此类操作. 唯一的问题是以便携方式捕获屏幕。我能找到的唯一理论上可移植的库是 this one ，但我无法让它工作。

我没有使用真正可移植的库，而是围绕 ffmpeg 破解了一个适用于 linux 的解决方案，并且 theoretically could be made to work on Windows and OSX as well :

import cv2
import numpy as np
import subprocess

THRESHOLD = 0.7

# your reference image
needle = cv2.imread('/path/to/some/needle.jpg', cv2.CV_LOAD_IMAGE_GRAYSCALE)
needle_height, needle_width = needle.shape

width, height = (800, 600) # you could of course detect this

command = [ '/usr/bin/ffmpeg',
        '-f', 'x11grab',
        '-i', ':0.0+0,0',
        '-r', '3', # lower frame-rate for testing
        '-s', '%dx%d' % (width, height),
        '-f', 'rawvideo',
        '-vcodec', 'rawvideo',
        '-pix_fmt', 'bgr24',
        '-']

pipe = subprocess.Popen(command, stdout = subprocess.PIPE, bufsize=10**8)

while(True):
    # get a frame
    raw = pipe.stdout.read(width*height*3)
    # format it into a matrix which can be worked on by openCV
    original = np.fromstring(raw, dtype='uint8').reshape((height, width, 3))
    pipe.stdout.flush()

    # transform it to grayscale for the matching
    haystack = cv2.cvtColor(original, cv2.COLOR_BGR2GRAY)

    # work through the frame looking for matches above a certain THRESHOLD
    # and mark them with a green circle
    for pt in np.transpose(np.where(cv2.matchTemplate(haystack, needle, cv2.TM_CCOEFF_NORMED) >= THRESHOLD)):
        cv2.circle(original, (pt[1] + needle_width/2, pt[0] + needle_height/2), 10, (0,255,0))

    # display the altered frame
    cv2.imshow('matches', original)

    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

cap.release()
cv2.destroyAllWindows()

请注意，从管道读取期间使用的魔法“3”来自 ffmpeg 的输出像素格式:bgr24。