Capture video data from screen in Python

Question

Is there a way with Python (maybe with OpenCV or PIL) to continuously grab frames of all or a portion of the screen, at least at 15 fps or more? I've seen it done in other languages, so in theory it should be possible.

I do not need to save the image data to a file. I actually just want it to output an array containing the raw RGB data (like in a numpy array or something) since I'm going to just take it and send it to a large LED display (probably after re-sizing it).

http://stackoverflow.com/a/24213099/1832058 – furas Jan 30 '16 at 04:34 — furas, Jan 30 '16 at 04:34

score 24 · Answer 1 · edited Apr 02 '18 at 16:42

24

There is an other solution with mss which provide much better frame rate. (Tested on a Macbook Pro with MacOS Sierra)

import numpy as np
import cv2
from mss import mss
from PIL import Image

mon = {'top': 160, 'left': 160, 'width': 200, 'height': 200}

sct = mss()

while 1:
    sct.get_pixels(mon)
    img = Image.frombytes('RGB', (sct.width, sct.height), sct.image)
    cv2.imshow('test', np.array(img))
    if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break

edited Apr 02 '18 at 16:42

Dartmouth

1,016
2
16
22

answered Apr 22 '17 at 14:02

Neabfi

3,070
1
23
37

1

I get **ValueError: not enough image data** when using Image.frombytes – ypicard May 30 '17 at 14:17
with an i7 2600k, I only get 8fps when recording a 1440p monitor. – NullVoxPopuli Oct 27 '17 at 20:50
4

As of July 2019, this code will trigger this error: "AttributeError: 'MSS' object has no attribute 'get_pixels'". – YakovK Jul 24 '19 at 00:40
1

@YakovK To circumvent this, install an older version of mss: `pip uninstall mss`, `pip install mss==2.0.22` – Nicolas Forstner Mar 23 '20 at 18:12
3

Another workaround is to use `mss().grab(window)` The code would look similar to this: `screenshot = mss.mss().grab(window)` `img = Image.frombytes("RGB", (screenshot.width, screenshot.height), screenshot.rgb)` – qdtroemner Mar 31 '20 at 13:22

ibininja · Answer 2 · 2016-01-30T05:07:05.860

15

You will need to use ImageGrab from Pillow (PIL) Library and convert the capture to numpy array. When you have the array you can do what you please with it using opencv. I converted capture to gray color and used imshow() as a demonstration.

Here is a quick code to get you started:

from PIL import ImageGrab
import numpy as np
import cv2

img = ImageGrab.grab(bbox=(100,10,400,780)) #bbox specifies specific region (bbox= x,y,width,height *starts top-left)
img_np = np.array(img) #this is the array obtained from conversion
frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)
cv2.imshow("test", frame)
cv2.waitKey(0)
cv2.destroyAllWindows()

you can plug an array there with the frequency you please to keep capturing frames. After that you just decode the frames. don't forget to add before the loop:

fourcc = cv2.VideoWriter_fourcc(*'XVID')
vid = cv2.VideoWriter('output.avi', fourcc, 6, (640,480))

and inside the loop you can add:

vid.write(frame) #the edited frame or the original img_np as you please

UPDATE
the end result look something like this (If you want to achieve a stream of frames that is. Storing as video just a demonstration of using opencv on the screen captured):

from PIL import ImageGrab
import numpy as np
import cv2
while(True):
    img = ImageGrab.grab(bbox=(100,10,400,780)) #bbox specifies specific region (bbox= x,y,width,height)
    img_np = np.array(img)
    frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)
    cv2.imshow("test", frame)
    cv2.waitKey(0)
cv2.destroyAllWindows()

Hope that helps

edited Jan 30 '16 at 05:07

answered Jan 30 '16 at 04:40

ibininja

1,094
1
13
28

Not sure what you mean by: "you can plug an array there with the frequency you please to keep capturing frames"... array where? And I don't need to do video output at all... Is there a way of specifying where on the screen to grab? – Adam Haile Jan 30 '16 at 04:51
I meant to say add a while look to have the stream of screencaptures. I will edit the code now. as for specifying specific region use bbox parameter. One min I will make the update – ibininja Jan 30 '16 at 04:54
I made further changes. Hope that helps – ibininja Jan 30 '16 at 05:05
1

This essentially uses `ImageGrab.grab()`, which in my experience is very very slow (about 2 fps on my Macbook Pro 2015). This solution can't capture at more than 15fps as the OP asked. I've also tried the `mss` library, which is much faster, but I didn't manage to guarantee a steady fps or regular intervals between images. – Ray Apr 08 '17 at 18:36
ImageGrab is macOS and Windows only – Sarath Ak May 04 '19 at 02:24
@ibininja I tried above code on my mac machine. It is generating output.avi of size 0 kb. I am not sure what's wrong – SachinB May 18 '19 at 13:37

score 14 · Answer 3 · edited Nov 01 '19 at 16:40

14

With all of the above solutions, I was unable to get a usable frame rate until I modified my code in the following way:

import numpy as np
import cv2
from mss import mss
from PIL import Image

bounding_box = {'top': 100, 'left': 0, 'width': 400, 'height': 300}

sct = mss()

while True:
    sct_img = sct.grab(bounding_box)
    cv2.imshow('screen', np.array(sct_img))

    if (cv2.waitKey(1) & 0xFF) == ord('q'):
        cv2.destroyAllWindows()
        break

With this solution, I easily get 20+ frames/second.

For reference, check this link: OpenCV/Numpy example with mss

edited Nov 01 '19 at 16:40

answered Jan 18 '19 at 00:40

mlz7

1,697
3
20
48

2

65fps on a Mac Mini - awesome! – Wolfgang Fahl Oct 24 '19 at 20:52
1

Saving as a .npy file is a breeze, I am getting a frame every 10-15ms. Turns out to be 100FPS - 67FPS. I guess SSD is also playing an important role here. – Mridul Pandey Jan 22 '20 at 01:54

score 5 · Answer 4 · answered Apr 23 '18 at 12:26

5

You can try this=>

import mss
import numpy

with mss.mss() as sct:
    monitor = {'top': 40, 'left': 0, 'width': 800, 'height': 640}
    img = numpy.array(sct.grab(monitor))
    print(img)

answered Apr 23 '18 at 12:26

Sandzz

51
1
2

szanyi krisztián · Answer 5 · 2019-05-29T13:00:07.843

5

based on this post and others posts, i made something like this . Its taking a screenshot and writing into a video file without saving the img.

import cv2
import numpy as np
import os
import pyautogui

output = "video.avi"
img = pyautogui.screenshot()
img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
#get info from img
height, width, channels = img.shape
# Define the codec and create VideoWriter object
fourcc = cv2.VideoWriter_fourcc(*'mp4v')
out = cv2.VideoWriter(output, fourcc, 20.0, (width, height))

while(True):
 try:
  img = pyautogui.screenshot()
  image = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
  out.write(image)
  StopIteration(0.5)
 except KeyboardInterrupt:
  break

out.release()
cv2.destroyAllWindows()

edited May 29 '19 at 13:00

answered Dec 16 '18 at 18:52

szanyi krisztián

51
1
4

1

Thanks man.Just two typos on line 18 you should pass img not image and on line 19 you should image not img. – Muhammad Younus Feb 25 '19 at 06:22
thanks,didn't noticed just wanted to put it together :D (will fix it soon) – szanyi krisztián May 29 '19 at 12:53

score 3 · Answer 6 · edited Feb 23 '20 at 10:54

3

You can try this code as it is working for me. I've tested it on Linux

import numpy as np
import cv2
from mss import mss
from PIL import Image

sct = mss()

while 1:
    w, h = 800, 640
    monitor = {'top': 0, 'left': 0, 'width': w, 'height': h}
    img = Image.frombytes('RGB', (w,h), sct.grab(monitor).rgb)
    cv2.imshow('test', cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR))
    if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break

Make sure that the following packages are installed:

Pillow, opencv-python, numpy, mss

edited Feb 23 '20 at 10:54

ZF007

3,318
8
26
39

answered May 04 '19 at 02:52

Sarath Ak

5,066
35
37

Great approach, managed to get an average of 90 fps using this method – DankCoder Feb 23 '20 at 10:00

score 2 · Answer 7 · answered Aug 01 '18 at 22:41

2

I tried all of the above but it did not give me the real-time screen update. You can try this. This code is tested and worked successfully and also give you a good fps output. You can also judge this by each loop time it's needed.

import numpy as np
import cv2
from PIL import ImageGrab as ig
import time

last_time = time.time()
while(True):
    screen = ig.grab(bbox=(50,50,800,640))
    print('Loop took {} seconds',format(time.time()-last_time))
    cv2.imshow("test", np.array(screen))
    last_time = time.time()
    if cv2.waitKey(25) & 0xFF == ord('q'):
        cv2.destroyAllWindows()
        break

answered Aug 01 '18 at 22:41

Istiyak

587
5
15

1

Tried this on Windows with Python 3.6 and got around 10 fps with my i7 6700k CPU. – Pablo Canseco Feb 18 '19 at 16:00
Loop took {} seconds. is that my FPS? – Maksim Kniazev Jul 18 '19 at 21:27
No, its the time difference between two frame process... The less difference is very good. You can make another calculation like analyze the difference in every 60 seconds how many frames are processed... – Istiyak Jul 23 '19 at 11:07
1

2.5 fps on a MacMini - while Markoe7's approach renders at > 65 fps – Wolfgang Fahl Oct 24 '19 at 20:52

score 2 · Answer 8 · answered Aug 14 '20 at 12:32

If anyone looking for a much easier and faster way to grab screen with mss library, then try ScreenGear class from my high-performance video-processing vidgear library. Just write these few lines of python code on any machine (Tested on all platforms, including Windows 10, MacOS Serra, Linux Mint) and enjoy threaded screen-casting.

# import required libraries
from vidgear.gears import ScreenGear
import cv2

# define dimensions of screen w.r.t to given monitor to be captured
options = {'top': 40, 'left': 0, 'width': 100, 'height': 100}

# open video stream with defined parameters
stream = ScreenGear(monitor=1, logging=True, **options).start()

# loop over
while True:

    # read frames from stream
    frame = stream.read()

    # check for frame if Nonetype
    if frame is None:
        break


    # {do something with the frame here}


    # Show output window
    cv2.imshow("Output Frame", frame)

    # check for 'q' key if pressed
    key = cv2.waitKey(1) & 0xFF
    if key == ord("q"):
        break

# close output window
cv2.destroyAllWindows()

# safely close video stream
stream.stop()

VidGear library Docs: https://abhitronix.github.io/vidgear

ScreenGear API: https://abhitronix.github.io/vidgear/gears/screengear/overview/

More examples: https://abhitronix.github.io/vidgear/gears/screengear/usage/

score 0 · Answer 9 · answered Apr 11 '20 at 15:29

This task is very simple with opencv, we are just capturing screenshots in loop, and converting it into frames. I created timer for screenrecording, in start you have to enter how many seconds you want to record:) Here is the code.

import cv2
import numpy as np
import pyautogui
from win32api import GetSystemMetrics
import time

#Take resolution from system automatically
w = GetSystemMetrics(0)
h =  GetSystemMetrics(1)
SCREEN_SIZE = (w,h)
fourcc = cv2.VideoWriter_fourcc(*"XVID")
out = cv2.VideoWriter("recording.mp4", fourcc, 20.0, (SCREEN_SIZE))
tim = time.time()
tp = int(input('How many times you want to record screen?->(Define value in Seconds): '))
tp = tp+tp
f = tim+tp
while True:
    img = pyautogui.screenshot()
    frame = np.array(img)
    frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    out.write(frame)
    tu = time.time()
    if tu>f:
        break
cv2.destroyAllWindows()
out.release()

So that's how you can use time in screen recording, you don't need to use imshow() because it shows infinitely our screen recording on-screen so output video looks weird.

This works but it's really not fast, at least on linux where it depends on an external tool `scrot` to actually take the screenshots. It takes almost half a second with my 3 monitors, even if tell it to only screenshot a small region with `pyautogui.screenshot(region=(...))`. — jlh, Jun 01 '20 at 14:43

score -1 · Answer 10 · answered Sep 27 '18 at 16:13

I've tried ImageGrab from PIL and it gave me 20fps which is ok but using win32 libraries gave me +40fps which is amazing!

I used this code by Frannecklp but it didn't work just fine so I needed to modify it:

-Firstly pip install pywin32 in case using the libraries

-import the libraries like this instead:

import cv2
import numpy as np
from win32 import win32gui
from pythonwin import win32ui
from win32.lib import win32con
from win32 import win32api

for geting a simple image screen do:

from grab_screen import grab_screen
import cv2
img = grab_screen()
cv2.imshow('frame',img)

and for getting frames:

while(True):
#frame = grab_screen((0,0,100,100))
frame = grab_screen()
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF == ord('q') or x>150:
    break

Capture video data from screen in Python

10 Answers10

Linked