15

I am working on an iOS app that uses AVAudioEngine for various things, including recording audio to a file, applying effects to that audio using audio units, and playing back the audio with the effect applied. I use a tap to also write the output to a file. When this is done it writes to the file in real time as the audio is playing back.

Is it possible to set up an AVAudioEngine graph that reads from a file, processes the sound with an audio unit, and outputs to a file, but faster than real time (ie., as fast as the hardware can process it)? The use case for this would be to output a few minutes of audio with effects applied, and I certainly wouldn't want to wait for a few minutes for it to be processed.

Edit: here's the code that I'm using to set up the AVAudioEngine's graph, and play a sound file:

AVAudioEngine* engine = [[AVAudioEngine alloc] init];

AVAudioPlayerNode* player = [[AVAudioPlayerNode alloc] init];
[engine attachNode:player];

self.player = player;
self.engine = engine;

if (!self.distortionEffect) {
    self.distortionEffect = [[AVAudioUnitDistortion alloc] init];
    [self.engine attachNode:self.distortionEffect];
    [self.engine connect:self.player to:self.distortionEffect format:[self.distortionEffect outputFormatForBus:0]];
    AVAudioMixerNode* mixer = [self.engine mainMixerNode];
    [self.engine connect:self.distortionEffect to:mixer format:[mixer outputFormatForBus:0]];
}

[self.distortionEffect loadFactoryPreset:AVAudioUnitDistortionPresetDrumsBitBrush];

NSError* error;
if (![self.engine startAndReturnError:&error]) {
    NSLog(@"error: %@", error);
} else {
    NSURL* fileURL = [[NSBundle mainBundle] URLForResource:@"test2" withExtension:@"mp3"];
    AVAudioFile* file = [[AVAudioFile alloc] initForReading:fileURL error:&error];

    if (error) {
        NSLog(@"error: %@", error);
    } else {
        [self.player scheduleFile:file atTime:nil completionHandler:nil];
        [self.player play];
    }
}

The above code plays the sound in the test2.mp3 file, with the AVAudioUnitDistortionPresetDrumsBitBrush distortion preset applied, in real time.

I then modified the above code by adding these lines after [self.player play]:

        [self.engine stop];
        [self renderAudioAndWriteToFile];

I modified the renderAudioAndWriteToFile method that Vladimir provided so that instead of allocating a new AVAudioEngine in the first line, it simply uses self.engine that has already been set up.

However, in renderAudioAndWriteToFile, it's logging "Can not render audio unit" because AudioUnitRender is returning a status of kAudioUnitErr_Uninitialized.

Edit 2: I should mention that I'm perfectly happy to convert the AVAudioEngine code I posted to use the C apis if that would make things easier. However, I would want the code to produce the same output as the AVAudioEngine code (including the use of the factory preset shown above).

Greg
  • 32,510
  • 15
  • 88
  • 99

1 Answers1

14
  1. Configure your engine and player node.
  2. Call play method for your player node.
  3. Pause your engine.
  4. Get an audio unit from your AVAudioOutputNode (audioEngine.outputNode) with this method.
  5. Render from audio unit with AudioUnitRender in cycle and write audio buffer list to file with Extended Audio File Services.

Example:

Audio engine configuration

- (void)configureAudioEngine {
    self.engine = [[AVAudioEngine alloc] init];
    self.playerNode = [[AVAudioPlayerNode alloc] init];
    [self.engine attachNode:self.playerNode];
    AVAudioUnitDistortion *distortionEffect = [[AVAudioUnitDistortion alloc] init];
    [self.engine attachNode:distortionEffect];
    [self.engine connect:self.playerNode to:distortionEffect format:[distortionEffect outputFormatForBus:0]];
    self.mixer = [self.engine mainMixerNode];
    [self.engine connect:distortionEffect to:self.mixer format:[self.mixer outputFormatForBus:0]];
    [distortionEffect loadFactoryPreset:AVAudioUnitDistortionPresetDrumsBitBrush];
    NSError* error;
    if (![self.engine startAndReturnError:&error])
        NSLog(@"Can't start engine: %@", error);
    else
        [self scheduleFileToPlay];
}

- (void)scheduleFileToPlay {
    NSError* error;
    NSURL *fileURL = [[NSBundle mainBundle] URLForResource:@"filename" withExtension:@"m4a"];
    self.file = [[AVAudioFile alloc] initForReading:fileURL error:&error];
    if (self.file)
        [self.playerNode scheduleFile:self.file atTime:nil completionHandler:nil];
    else
        NSLog(@"Can't read file: %@", error);
}

Rendering methods

- (void)renderAudioAndWriteToFile {
    [self.playerNode play];
    [self.engine pause];
    AVAudioOutputNode *outputNode = self.engine.outputNode;
    AudioStreamBasicDescription const *audioDescription = [outputNode outputFormatForBus:0].streamDescription;
    NSString *path = [self filePath];
    ExtAudioFileRef audioFile = [self createAndSetupExtAudioFileWithASBD:audioDescription andFilePath:path];
    if (!audioFile)
        return;
    AVURLAsset *asset = [AVURLAsset assetWithURL:self.file.url];
    NSTimeInterval duration = CMTimeGetSeconds(asset.duration);
    NSUInteger lengthInFrames = duration * audioDescription->mSampleRate;
    const NSUInteger kBufferLength = 4096;
    AudioBufferList *bufferList = AEAllocateAndInitAudioBufferList(*audioDescription, kBufferLength);
    AudioTimeStamp timeStamp;
    memset (&timeStamp, 0, sizeof(timeStamp));
    timeStamp.mFlags = kAudioTimeStampSampleTimeValid;
    OSStatus status = noErr;
    for (NSUInteger i = kBufferLength; i < lengthInFrames; i += kBufferLength) {
        status = [self renderToBufferList:bufferList writeToFile:audioFile bufferLength:kBufferLength timeStamp:&timeStamp];
        if (status != noErr)
            break;
    }
    if (status == noErr && timeStamp.mSampleTime < lengthInFrames) {
        NSUInteger restBufferLength = (NSUInteger) (lengthInFrames - timeStamp.mSampleTime);
        AudioBufferList *restBufferList = AEAllocateAndInitAudioBufferList(*audioDescription, restBufferLength);
        status = [self renderToBufferList:restBufferList writeToFile:audioFile bufferLength:restBufferLength timeStamp:&timeStamp];
        AEFreeAudioBufferList(restBufferList);
    }
    AEFreeAudioBufferList(bufferList);
    ExtAudioFileDispose(audioFile);
    if (status != noErr)
        NSLog(@"An error has occurred");
    else
        NSLog(@"Finished writing to file at path: %@", path);
}

- (NSString *)filePath {
    NSArray *documentsFolders =
            NSSearchPathForDirectoriesInDomains(NSDocumentDirectory, NSUserDomainMask, YES);
    NSString *fileName = [NSString stringWithFormat:@"%@.m4a", [[NSUUID UUID] UUIDString]];
    NSString *path = [documentsFolders[0] stringByAppendingPathComponent:fileName];
    return path;
}

- (ExtAudioFileRef)createAndSetupExtAudioFileWithASBD:(AudioStreamBasicDescription const *)audioDescription
                                          andFilePath:(NSString *)path {
    AudioStreamBasicDescription destinationFormat;
    memset(&destinationFormat, 0, sizeof(destinationFormat));
    destinationFormat.mChannelsPerFrame = audioDescription->mChannelsPerFrame;
    destinationFormat.mSampleRate = audioDescription->mSampleRate;
    destinationFormat.mFormatID = kAudioFormatMPEG4AAC;
    ExtAudioFileRef audioFile;
    OSStatus status = ExtAudioFileCreateWithURL(
            (__bridge CFURLRef) [NSURL fileURLWithPath:path],
            kAudioFileM4AType,
            &destinationFormat,
            NULL,
            kAudioFileFlags_EraseFile,
            &audioFile
    );
    if (status != noErr) {
        NSLog(@"Can not create ext audio file");
        return nil;
    }
    UInt32 codecManufacturer = kAppleSoftwareAudioCodecManufacturer;
    status = ExtAudioFileSetProperty(
            audioFile, kExtAudioFileProperty_CodecManufacturer, sizeof(UInt32), &codecManufacturer
    );
    status = ExtAudioFileSetProperty(
            audioFile, kExtAudioFileProperty_ClientDataFormat, sizeof(AudioStreamBasicDescription), audioDescription
    );
    status = ExtAudioFileWriteAsync(audioFile, 0, NULL);
    if (status != noErr) {
        NSLog(@"Can not setup ext audio file");
        return nil;
    }
    return audioFile;
}

- (OSStatus)renderToBufferList:(AudioBufferList *)bufferList
                   writeToFile:(ExtAudioFileRef)audioFile
                  bufferLength:(NSUInteger)bufferLength
                     timeStamp:(AudioTimeStamp *)timeStamp {
    [self clearBufferList:bufferList];
    AudioUnit outputUnit = self.engine.outputNode.audioUnit;
    OSStatus status = AudioUnitRender(outputUnit, 0, timeStamp, 0, bufferLength, bufferList);
    if (status != noErr) {
        NSLog(@"Can not render audio unit");
        return status;
    }
    timeStamp->mSampleTime += bufferLength;
    status = ExtAudioFileWrite(audioFile, bufferLength, bufferList);
    if (status != noErr)
        NSLog(@"Can not write audio to file");
    return status;
}

- (void)clearBufferList:(AudioBufferList *)bufferList {
    for (int bufferIndex = 0; bufferIndex < bufferList->mNumberBuffers; bufferIndex++) {
        memset(bufferList->mBuffers[bufferIndex].mData, 0, bufferList->mBuffers[bufferIndex].mDataByteSize);
    }
}

I used some functions from this cool framework:

AudioBufferList *AEAllocateAndInitAudioBufferList(AudioStreamBasicDescription audioFormat, int frameCount) {
    int numberOfBuffers = audioFormat.mFormatFlags & kAudioFormatFlagIsNonInterleaved ? audioFormat.mChannelsPerFrame : 1;
    int channelsPerBuffer = audioFormat.mFormatFlags & kAudioFormatFlagIsNonInterleaved ? 1 : audioFormat.mChannelsPerFrame;
    int bytesPerBuffer = audioFormat.mBytesPerFrame * frameCount;
    AudioBufferList *audio = malloc(sizeof(AudioBufferList) + (numberOfBuffers-1)*sizeof(AudioBuffer));
    if ( !audio ) {
        return NULL;
    }
    audio->mNumberBuffers = numberOfBuffers;
    for ( int i=0; i<numberOfBuffers; i++ ) {
        if ( bytesPerBuffer > 0 ) {
            audio->mBuffers[i].mData = calloc(bytesPerBuffer, 1);
            if ( !audio->mBuffers[i].mData ) {
                for ( int j=0; j<i; j++ ) free(audio->mBuffers[j].mData);
                free(audio);
                return NULL;
            }
        } else {
            audio->mBuffers[i].mData = NULL;
        }
        audio->mBuffers[i].mDataByteSize = bytesPerBuffer;
        audio->mBuffers[i].mNumberChannels = channelsPerBuffer;
    }
    return audio;
}

void AEFreeAudioBufferList(AudioBufferList *bufferList ) {
    for ( int i=0; i<bufferList->mNumberBuffers; i++ ) {
        if ( bufferList->mBuffers[i].mData ) free(bufferList->mBuffers[i].mData);
    }
    free(bufferList);
}
Vlad
  • 6,861
  • 2
  • 22
  • 30
  • Thanks for the advice Vladimir. Unfortunately this code is outputting the message "Can not render audio unit", as the status returned by AudioUnitRender is kAudioUnitErr_Uninitialized. I'll add some code to my question to show how I'm setting up the AVAudioEngine's graph before executing the code you provided. – Greg Jun 09 '15 at 20:08
  • This code just shows you my idea. Unfortunately I can't test it with your configuration. I used a big part of this code in my project exactly for things you are asking. But I worked with Core Audio and AUGraph, not with AVAudioEngine. I sure it will work if you configure your engine right. – Vlad Jun 09 '15 at 20:19
  • OK, thanks again Vladimir. I'm happy to switch to using Core Audio and AUGraph, but I'm less familiar with these APIs. I've added a bounty to the question and will happily award it if you can show me how to adapt my existing code to use Core Audio and AUGraph instead of AVAudioEngine. – Greg Jun 09 '15 at 20:30
  • 2
    I tried your code and fix an issue. Take a look https://github.com/VladimirKravchenko/AVAudioEngineOfflineRender – Vlad Jun 09 '15 at 21:26
  • This is great - thanks very much Vladimir! Just one problem I've noticed - for at least one of my input files, the resulting output file ends about 2 seconds early. Any idea why that would be? I can post a link to the input file if you like. – Greg Jun 10 '15 at 03:39
  • what is the sample rate of this file? – Vlad Jun 10 '15 at 05:17
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/80197/discussion-between-greg-and-vladimir-kravchenko). – Greg Jun 10 '15 at 14:53
  • I dowloaded, built and ran the demo. The resulting file was 40 seconds of silence. I'll try to figure out why, but wanted to give you a heads-up. – mahboudz Jun 11 '15 at 18:00
  • mahboudz, what device or simulator did you use? I tested it with my iphone 6 and simulators on 2 different machines and this code worked. – Vlad Jun 11 '15 at 19:49
  • I tested on iPhone 6, iOS 8.3. Simulator iPad Air 8.3 iPhone 5: iOS 8.3 iPod Touch: iOS 8.3 Doesn't seem to have a pattern. The silence can be the entire output file, or just the first ~5 seconds, or after ~5 seconds of audio then silence. I can send you the sound files if you want. – mahboudz Jun 17 '15 at 07:00
  • 1
    Thanks for your comment, I'll try to reproduce and fix this problem. Did you use my sample project https://github.com/VladimirKravchenko/AVAudioEngineOfflineRender without any changes? – Vlad Jun 17 '15 at 07:46
  • Yes. (Sorry for delay. Traveling. I can try to debug once I am home) – mahboudz Jun 21 '15 at 04:46
  • I can confirm the same issue @mahboudz mentioned with the the silence at the beginning of the output file on iPhone 6, iOS8.3 and 8.2 simulators. – Justin Levi Winter Jun 24 '15 at 00:28
  • I have the same issue with @mahboudz. It's almost 6 second silence in the output file. I tested the project in github with Simulator 9.1 – Vincent Nov 11 '15 at 03:45
  • I've posted a related question about how this approach gives me a silent output if headphones are connected http://stackoverflow.com/questions/34144456/avaudioengine-offline-render-silent-output-only-when-headphones-connected – skattyadz Dec 07 '15 at 22:30
  • This is great! Any ideas how one might make this work with a MIDI file using AVAudioSequencer (or MusicPlayer)? I made a quick prototype that successfully produces an audio file when rendered offline, but it sounds as if all the MIDI events are triggered at once: https://github.com/archagon/aumidisynth-offline-render-test – Archagon Apr 21 '16 at 02:19
  • I tried recording an audio (via `AVRecorder`) and tried loading it, but after rendering, the output file is silent and 0 seconds long. Any clue guys? – hyd00 Aug 11 '16 at 14:57
  • When my duration is not a whole number for example 15.000090702947846 because I am recording the audio, I get a blank in between. Raised an issue with the cause of having blanks in between at https://github.com/VladimirKravchenko/AVAudioEngineOfflineRender/issues/5 – Jugal Desai Feb 08 '17 at 07:18
  • HI . Ive tried this , it doesn't work on iphones 7 , 7+ , it gives me a silenced file – user3703910 Jun 23 '17 at 10:30
  • How is this different from AVAudioEngine's manual rendering mode? – NeoWang Nov 15 '19 at 04:58