What is the best method to capture images from a live video device for use by a Java-based application?

Question

I am looking into an image processing problem for semi-real time detection of certain scenarios. My goal is to have the live video arrive as Motion JPEG frames in my Java code somehow.

I am familiar with the Java Media Framework and, sadly, I think we can consider that an effectively dead API. I am also familiar with Axis boxes and, while I really like their solution, I would appreciate any critical feedback on my specific points of interest.

This is how I define "best" for the purpose of this discussion:

• Latency - if I'm controlling the camera using this video stream, I would like to keep my round-trip latency at less than 100 milliseconds if possible. That's measured as the time between my control input to the time when I see the visible change. EDIT some time later: another thing to keep in mind is that camera control is likely to be a combination of manual and automatic (event triggers). We need to see those pictures right away, even if the high quality feed is archived separately.
• Cost - free / open source is better than not free.
• Adjustable codec parameters - I need to be able to tune the codec for certain situations. Sometimes a high-speed low-resolution stream is actually easier to process.
• "Integration" with Java - how much trouble is it to hook this solution to my code? Am I sending packets over a socket? Hitting URLs? Installing Direct3D / JNI combinations?
• Windows / Linux / both? - I would prefer an operating system agnostic solution because I have to deliver to several flavors of OS but there may be a solution that is optimal for one but not the other.

NOTE: I am aware of other image / video capture codecs and that is not the focus of this question. I am specifically not interested in streaming APIs (e.g., MPEG4) due to the loss of frame accuracy. However, if there is a solution to my question that delivers another frame-accurate data stream, please chime in.

Follow-up to this question: at this point, I am strongly inclined to buy appliances such as the Axis video encoders rather than trying to capture the video in software or on the PC directly. However, if someone has alternatives, I'd love to hear them.

Answer 1

This JavaCV implementation works fine.

CODE:

import com.googlecode.javacv.OpenCVFrameGrabber;

import com.googlecode.javacv.cpp.opencv_core.IplImage;
import static com.googlecode.javacv.cpp.opencv_highgui.*;

public class CaptureImage {
    private static void captureFrame() {
        // 0-default camera, 1 - next...so on
        final OpenCVFrameGrabber grabber = new OpenCVFrameGrabber(0);
        try {
            grabber.start();
            IplImage img = grabber.grab();
            if (img != null) {
                cvSaveImage("capture.jpg", img);
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }
    public static void main(String[] args) {
        captureFrame();
    }
}

There is also post on viewing live video from Camera .And configuration for JavaCV :

I think this will meet your requirements.

Answer 2

FMJ can definitely capture video and turn it into MJPEG frames.

Answer 3

Regarding the dead-ness of JMF, are you aware of the FMJ implementation? I don't know whether it qualifies as the "best" solution, but it's probably worth adding to the discussion.

Answer 4

Below is shown a very simple implementation using Marvin Framework. Using Marvin you can add real time video processing easily.

import javax.swing.JFrame;
import marvin.gui.MarvinImagePanel;
import marvin.image.MarvinImage;
import marvin.video.MarvinJavaCVAdapter;
import marvin.video.MarvinVideoInterface;

public class SimpleVideoTest extends JFrame implements Runnable{

    private MarvinVideoInterface    videoAdapter;
    private MarvinImage             image;
    private MarvinImagePanel        videoPanel;

    public SimpleVideoTest(){
        super("Simple Video Test");

        // Create the VideoAdapter and connect to the camera
        videoAdapter = new MarvinJavaCVAdapter();
        videoAdapter.connect(0);

        // Create VideoPanel
        videoPanel = new MarvinImagePanel();
        add(videoPanel);

        // Start the thread for requesting the video frames 
        new Thread(this).start();

        setSize(800,600);
        setVisible(true);
    }

    public static void main(String[] args) {
        SimpleVideoTest t = new SimpleVideoTest();
        t.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
    }

    @Override
    public void run() {
        while(true){
            // Request a video frame and set into the VideoPanel
            image = videoAdapter.getFrame();
            videoPanel.setImage(image);
        }
    }
}

Another example applying multiple algorithms for real time video processing.

Answer 5

This is my JavaCV implementation with high resolution video output and no noticeable drop in the frame-rate than other solutions (only when my webcam refocuses do I notice a slight drop, only for a moment though).

import java.awt.image.BufferedImage;
import java.io.File;

import javax.swing.JFrame;

import com.googlecode.javacv.CanvasFrame;
import com.googlecode.javacv.OpenCVFrameGrabber;
import com.googlecode.javacv.OpenCVFrameRecorder;
import com.googlecode.javacv.cpp.opencv_core.IplImage;

public class Webcam implements Runnable {

    IplImage image;
    static CanvasFrame frame = new CanvasFrame("Web Cam");
    public static boolean running = false;

    public Webcam()
    {
        frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
    }

    @Override
    public void run()
    {
        try
        {
            grabber.setImageWidth(800);
            grabber.setImageHeight(600);
            grabber.start();
            while (running)
            {
                IplImage cvimg = grabber.grab();
                BufferedImage image;
                if (cvimg != null)
                {
                    // opencv_core.cvFlip(cvimg, cvimg, 1); // mirror
                    // show image on window
                    image = cvimg.getBufferedImage();
                    frame.showImage(image);
                }
            }
            grabber.stop();
            frame.dispose();
        } catch (Exception e)
        {
            e.printStackTrace();
        }
    }

    public static void main(String... args)
    {
        Webcam webcam = new Webcam();
        webcam.start();
    }

    public void start()
    {
        new Thread(this).start();
        running = true;
    }

    public void stop()
    {
        running = false;
    }
}

Answer 6

Have you ever looked at Processing.org? It's basically a simplified application framework for developing "artsy" applications and physical computing platforms, but it's based on Java and you can dig down to the "real" Java underneath.

The reason it came to mind is that there are several video libraries for Processing which are basically Java components (at least I think they are - the site has all the technical information you might need). There is a tutorial on using the Processing libraries and tools in the Eclipse IDE. There are also numerous examples on video capture and processing.

Even if you can't use the libraries directly, Processing is a great language/environment for working out algorithms. There are several great examples of image and video capture and real-time processing there.