13

I would like to implement near real-time OCR on the camera feed of my flutter app. To do this I would like to access the camera data in a speedy manner. As far as I can tell I have two options, and have hit roadblocks with both:

  1. Take a screenshot of the CameraPreview by putting a RepaintBoundary around it and creating a RenderRepaintBoundary, and calling boundary.toImage(). The problem with this method is that the .toImage method only seems to capture the painted widgets in the boundary and not the data from the camera preview. Simmilar to the issue described here: https://github.com/flutter/flutter/issues/17687

  2. Capture an image with controller.takePicture(filePath) from Camera 0.2.1, similar to the example docs. The problem here is that it takes super long before the image becomes available (2-3 seconds). I guess that this is because the file is saved to the disc on capture and then needs to be read from the file again.

Is there any way that one can directly access the picture information after capture, to do things like pre-process and OCR?

EduardKieser
  • 361
  • 3
  • 11
  • This is a long shot, but can you grab frames from a video? https://stackoverflow.com/questions/48718551/capture-video-with-flutter – dazza5000 Jun 28 '18 at 15:20
  • 1
    Would you be doing the OCR in OpenCV? – Richard Heap Jun 28 '18 at 15:39
  • My plan was to crop the frames and use the ml-kit to access the text-recogniser, using the mk-kit plugin: https://pub.dartlang.org/packages/mlkit – EduardKieser Jun 29 '18 at 13:26
  • @dazza5000 I think the video recording also has to first save the file to the disk before it can be accessed, which means that I would sit with the same problem.. – EduardKieser Jun 29 '18 at 13:27
  • Did you find a solution? – dazza5000 Jul 08 '18 at 02:33
  • Nope not yet. Holding my breath for news from the devs re: update to the Camera package. – EduardKieser Jul 09 '18 at 05:44
  • I'm trying to find a solution to this problem as well. I'm thinking with access to the camera's stream you could pipe that buffer into 'ml_vision' then use FirebaseVisionDetectors and CustomPainters to paint. There's two problems: 1 ) No access to the Camera stream (output) 2 ) 'FirebaseVisionDetector' lacks the capability to detect from a stream. – Jon Scalet Jul 17 '18 at 18:49
  • So it seems the flutter team has come through, camera 0.2.8 brings with it access to the image stream from Dart using cameraController.startImageStream(listener) to process the images. Unfortunately I cannot yet find any documentation or examples of how to use this, but I'm sure it will follow soon. – EduardKieser Jan 10 '19 at 07:07
  • this post might helpful for you.. https://blog.usejournal.com/real-time-object-detection-in-flutter-b31c7ff9ef96 – Harsha pulikollu Feb 24 '19 at 08:18
  • I posted my answer, I hope you find it useful – TruongSinh Mar 05 '19 at 06:48

1 Answers1

15

For "near real-time OCR", you need CameraController#startImageStream

example code

import 'package:camera/camera.dart';
import 'package:flutter/foundation.dart';
import 'package:flutter/material.dart';

void main() => runApp(MaterialApp(home: _MyHomePage()));

class _MyHomePage extends StatefulWidget {
  @override
  _MyHomePageState createState() => _MyHomePageState();
}

class _MyHomePageState extends State<_MyHomePage> {
  dynamic _scanResults;
  CameraController _camera;

  bool _isDetecting = false;
  CameraLensDirection _direction = CameraLensDirection.back;

  @override
  void initState() {
    super.initState();
    _initializeCamera();
  }

  Future<CameraDescription> _getCamera(CameraLensDirection dir) async {
    return await availableCameras().then(
      (List<CameraDescription> cameras) => cameras.firstWhere(
            (CameraDescription camera) => camera.lensDirection == dir,
          ),
    );
  }

  void _initializeCamera() async {
    _camera = CameraController(
      await _getCamera(_direction),
      defaultTargetPlatform == TargetPlatform.iOS
          ? ResolutionPreset.low
          : ResolutionPreset.medium,
    );
    await _camera.initialize();
    _camera.startImageStream((CameraImage image) {
      if (_isDetecting) return;
      _isDetecting = true;
      try {
        // await doSomethingWith(image)
      } catch (e) {
        // await handleExepction(e)
      } finally {
        _isDetecting = false;
      }
    });
  }
  Widget build(BuildContext context) {
    return null;
  }
}

This functionality was merged to https://github.com/flutter/plugins but it was not well documented.

Ref:

TruongSinh
  • 3,484
  • 23
  • 48
  • Are you able to draw on the image in real time as well and thus manipulate the camera's recording? – mik Jan 21 '21 at 07:19
  • ''Future _getCamera(CameraLensDirection dir) async { return await availableCameras().then( (List cameras) => cameras.firstWhere( (CameraDescription camera) => camera.lensDirection == dir, ), ); }'Unhandled Exception: Bad state: No element ListMixin.firstWhere – Chanaka Weerasinghe Feb 18 '21 at 07:44