20

I'm using Python 2.7 and OpenCV 2.4.9.

I need to capture the current frame that is being shown to the user and load it as an cv::Mat object in Python.

Do you guys know a fast way to do it recursively?

I need something like what's done in the example below, that captures Mat frames from a webcam recursively:

import cv2

cap = cv2.VideoCapture(0)
while(cap.isOpened()):
    ret, frame = cap.read()
    cv2.imshow('WindowName', frame)
    if cv2.waitKey(25) & 0xFF == ord('q'):
        cap.release()
        cv2.destroyAllWindows()
        break

In the example it's used the VideoCapture class to work with the captured image from the webcam.

With VideoCapture.read() a new frame is always being readed and stored into a Mat object.

Could I load a "printscreens stream" into a VideoCapture object? Could I create a streaming of my computer's screen with OpenCV in Python, without having to save and delete lots of .bmp files per second?

I need this frames to be Mat objects or NumPy arrays, so I can perform some Computer Vision routines with this frames in real time.

Renan V. Novas
  • 1,070
  • 1
  • 7
  • 18
  • It's hard to tell what your question actually is. Could you edit your question to clarify exactly what it is? For example, what do you mean 'recursively'? `frame` already contains the image. Why don't you just use it directly? – Aurelius Jun 09 '14 at 23:25
  • 1
    @Aurelius From What I understand, he's just looking for a way to inject printscreens as frames instead of images from the webcam. Since the monitor is not included in the device list taken by `cv2.VideoCapture`, You just need to grab the printscreen from elsewhere, such as **PIL `Image.Imagegrab.grab()`**, convert it to a numpy array, and inject it in the code shown above as a frame... – Raoul Jun 10 '14 at 03:42

3 Answers3

33

That's a solution code I've written using @Raoul tips.

I used PIL ImageGrab module to grab the printscreen frames.

import numpy as np
from PIL import ImageGrab
import cv2

while(True):
    printscreen_pil =  ImageGrab.grab()
    printscreen_numpy =   np.array(printscreen_pil.getdata(),dtype='uint8')\
    .reshape((printscreen_pil.size[1],printscreen_pil.size[0],3)) 
    cv2.imshow('window',printscreen_numpy)
    if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break
artburkart
  • 1,590
  • 1
  • 17
  • 26
Renan V. Novas
  • 1,070
  • 1
  • 7
  • 18
  • 1
    When using this solution I was getting the error that uint8 is undefined. Wrapping it in single quotes as 'uint8' all began to work. – jsleuth Aug 30 '15 at 01:18
  • Why did you do .reshape() and how does it work? The documentation is a bit difficult to understand. I get "ValueError: total size of new array must be unchanged" – andli Nov 07 '15 at 23:09
  • @jsleuth Sorry, my mistake, you can use dtype = numpy.uint8 – Renan V. Novas Dec 13 '15 at 18:05
  • 1
    @andii Reshape is a method of objects numpy.ndarray. I've reshaped the array because an image is a matrix (n,m), with n rows and m columns. I am transforming a flattened array into a matrix (n,m) with the method reshape. – Renan V. Novas Dec 13 '15 at 18:08
  • 6
    I get a speed (framerate) increase of almost 100x if I change `np.array(printscreen_pil.getdata(), dtype=uint8)` to `np.array(printscreen_pil, dtype=uint8)`. – Steve Feb 10 '17 at 23:24
  • Attention: `ImportError: ImageGrab is macOS and Windows only` – WuerfelDev Apr 25 '18 at 22:03
  • It seems like my mac (OSX 10.11) does not support cv2.imshow('window',printscreen_numpy) I am using Python 3.5.0 and Opencv 3.1.0 – Zhenye Na May 06 '18 at 17:42
  • >>> cv2.imshow('window', ps_numpy) qt.qpa.plugin: Could not find the Qt platform plugin "cocoa" in "" This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem. Abort trap: 6 – user391339 Feb 07 '20 at 02:04
  • Using this method inverts colors, (pillow uses RGB but OpenCV uses BGR) you might want to add this line: ```im_bgr = cv2.cvtColor(printscreen_numpy, cv2.COLOR_RGB2BGR) cv2.imshow('window',im_bgr)``` – Takelovski Mar 16 '20 at 20:57
24

I had frame rate problems with other solutions, mss solve them.

import numpy as np
import cv2
from mss import mss
from PIL import Image

mon = {'top': 160, 'left': 160, 'width': 200, 'height': 200}

sct = mss()

while 1:
    sct.get_pixels(mon)
    img = Image.frombytes('RGB', (sct.width, sct.height), sct.image)
    cv2.imshow('test', np.array(img))
    if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break
user3666197
  • 1
  • 6
  • 43
  • 77
Neabfi
  • 3,070
  • 1
  • 23
  • 37
4

This is the updated answer for the answer by @Neabfi

import time

import cv2
import numpy as np
from mss import mss

mon = {'top': 160, 'left': 160, 'width': 200, 'height': 200}
with mss() as sct:
    # mon = sct.monitors[0]
    while True:
        last_time = time.time()
        img = sct.grab(mon)
        print('fps: {0}'.format(1 / (time.time()-last_time)))
        cv2.imw('test', np.array(img))
        if cv2.waitKey(25) & 0xFF == ord('q'):
            cv2.destroyAllWindows()
            break

And to save to a mp4 video 

import time

import cv2
import numpy as np
from mss import mss


def record(name):
    with mss() as sct:
        # mon = {'top': 160, 'left': 160, 'width': 200, 'height': 200}
        mon = sct.monitors[0]
        name = name + '.mp4'
        fourcc = cv2.VideoWriter_fourcc(*'mp4v')
        desired_fps = 30.0
        out = cv2.VideoWriter(name, fourcc, desired_fps,
                              (mon['width'], mon['height']))
        last_time = 0
        while True:
            img = sct.grab(mon)
            # cv2.imshow('test', np.array(img))
            if time.time() - last_time > 1./desired_fps:
                last_time = time.time()
                destRGB = cv2.cvtColor(np.array(img), cv2.COLOR_BGRA2BGR)
                out.write(destRGB)
            if cv2.waitKey(25) & 0xFF == ord('q'):
                cv2.destroyAllWindows()
                break


record("Video")
Phani Rithvij
  • 2,303
  • 2
  • 16
  • 37