6

I am making an swift video app.

In my app, I need to crop and horizontally flip CVPixelBuffer and return result which type is also CVPixelBuffer.

I tried few things.

First, I used 'CVPixelBufferCreateWithBytes'

func resizePixelBuffer(_ pixelBuffer: CVPixelBuffer, destSize: CGSize) 
-> CVPixelBuffer? 
{

  CVPixelBufferLockAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: O))

  let baseAddress = CVPixelBufferGetBaseAddress(pixelBuffer)
  let bytesPerRow = CVPixelBufferGetBytesPerRow(pixelBuffer)
  let pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer)
  let width = CVPixelBufferGetWidth(pixelBuffer)
  let height = CVPixelBufferGetHeight(pixelBuffer)

  var destPixelBuffer: CVPixelBuffer?

  let topMargin = (height - destsize.height) / 2
  let leftMargin = (width - destsize.width) / 2 * 4   // bytesPerPixel
  let offset = topMargin * bytesPerRow + leftMargin

  CVPixelBufferCreateWithBytes(kCFAllocatorDefault, 
                               destSize.width, 
                               destSize.height, 
                               pixelFormat, 
                               baseAddress.advanced(by: offset),
                               bytesPerRow, 
                               nil, nil, nil, 
                               &destPixelBuffer)

  CVPixelBufferUnlockBaseAddress(pixelBuffer, CVPixelBufferLockFlags(rawValue: O))

  return destPixelBuffer
)

With this code, I can crop CVPixelBuffer directly and return CVPixelBuffer. However, I couldn't figure out how to flip CVPlxelBuffer horizontally.

So I tried other solutions.

Seconds, I converted CVPixelBuffer to CIImage and then, return to CVPixelBuffer

func resizePixelBuffer(_ pixelBuffer, destSize: CGSize) 
-> CVPixelBuffer?
{
  let bufferWidth = CVPixelBufferGetWidth(pixelBuffer)
  let bufferHeight = CVPixelBufferGetHeight(pixelBuffer)

  let ciImage = CIImage(cvPixelBuffer: pixelBuffer)
  let rect = CGRect(x: (bufferWidth - destSize.width)/2, y: (bufferHeight - destSize.height)/2, width: destSize.width, height: destSize.height)
  let croppedImage = ciImage.cropped(to: rect)

  croppedImage.transformed(by: CGAffineTransform(translateX: -1, y: 0))

  var destPixelBuffer: CVPixelBuffer?
  CVPixelBufferCreate(kCFAllocatorDefault, destSize.width, destSize.height,
                      CVPixelBufferGetPixelFormatType(pixelBuffer), nil, 
                      &destPixelBuffer)

  CIContext().render(croppedImage, to: destPixelBuffer!, bounds: croppedImage.extent, croppedImage.colorSpace)

  return destPixelBuffer
}

But the result is not that I expected. some part of image is black, and I think CGAffineTransform doesn't work.

Finally, I tried to convert to CGImage

func resizePixelBuffer(_ pixelBuffer: CVPixelBuffer, destSize: CGSize)
-> CVPixelBuffer? 
{
  let ciImage = CIImage(cvPixelBuffer: pixelBuffer)
  let cgImage = CIContext().createCGImage(ciImage, from: ciImage.extent)
  let rect = CGRect(x: (bufferWidth - destSize.width)/2, y: (bufferHeight - destSize.height)/2, width: destSize.width, height: destSize.height)

  let croppedImage = cgImage.cropping(to: rect)

  let width = croppedImage.width
  let height = croppedImage.height
  let pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer)

  var destPixelBuffer: CVPixelBuffer?
  CVPixelBufferCreate(kCFAllocatorDefault, width, height, pixelFormat, &destPixelBuffer)

  CVPixelBufferLockBaseAddress(destPixelBuffer, CVPixelBufferLockFlags(rawValue: 0))

  let destBaseAddress = CVPixelBufferGetBaseAddress(destPixelBuffer)
  let destBytesPerRow = CVPixelBufferGetBytesPerRow(destPixelBuffer)

  let context = CGContext(data: destBaseAddress, 
                          width: width, 
                          height: height, 
                          bitsPerComponent: 8, 
                          bytesPerRow: destBytesPerRow, 
                          space: croppedImage.colorSpace, 
                          bitmapInfo: CGImageAlphaInfo.premultipliedFirst.rawValue | CGBitmapInfo.byteOrder32Little.rawValue)

  context?.concatenate(__CGAffineTransformMake( 1, 0, 0, -1, 0, CGFloat(height)))

  context?.draw(croppedCgImage, in: CGRect(x: 0, y: 0, width: CGFloat(width), height: CGFloat(height)))

  CVPixelBufferUnlockBaseAddress(srcPixelBuffer, CVPixelBufferLockFlags(rawValue: 0))

  return destPixelBuffer
}

In this time, the output pixelbuffer is totally black.

I can't figure out how to crop and flip CVPixelBuffer and return CVPixelBuffer.

I think converting CIImage or CGImage is better way because there are lots of things I can do with those format.

But I don't know how to covert those formats back to CVPixelBuffer.

Please let me know how to do this.

HB.K
  • 157
  • 1
  • 10
  • I am not an expert. (Similar to IANAL.) Having said that, here's my thoughts - and just that, thoughts. (1) Unless you absolutely have to stick with CoreImage. Why introduce complexity? (2) It sounds like you have half of this figured out - cropping. And unless you have *any* performance concerns, stick with it. (3) That leaves "flipping", which if I understand you have working but with "partly black" image. So working with that? Start with a square image. Do it remove the black? If so, then work with a non-square image. I think you;'re close... and really think you are overthinking something. – dfd Mar 22 '19 at 16:59
  • Check out https://stackoverflow.com/a/12315642/806326 and https://stackoverflow.com/a/10063006/806326 – Sten May 15 '19 at 06:12
  • Were you able to solve this? – Deepak Sharma Nov 28 '20 at 12:44
  • did you find a solution? – Martin Mlostek Mar 12 '21 at 14:12

0 Answers0