Wcf System.InsufficientMemoryException. Failed to allocate a managed memory buffer of 536870912 bytes

Question

I have a WCF service method that returns a big byte[] array around 2 mb in size. However when i run that method from the WcfTestClient, I get the following exception.

System.InsufficientMemoryException
Failed to allocate a managed memory buffer of 536870912 bytes. The amount of available memory may be low.

I have searched on google, and people seem to have this issue but I am struggling to find any solution and the actual cause of why is it trying to create such a huge buffer in the first place?

Below are my config settings. I have set the maxItemsInObjectGraph="2147483647" otherwise it was throwing exception about it. The rest is simple i guess.

Server

  <system.serviceModel>
    <services>
      <service behaviorConfiguration="Wcf.ServiceBehavior"
        name="Wcf.WcfService">
        <endpoint address="" binding="wsHttpBinding" contract="Wcf.IWcfService">
          <identity>
            <dns value="localhost" />
          </identity>
        </endpoint>
        <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
      </service>
    </services>
    <behaviors>
      <serviceBehaviors>
        <behavior name="Wcf.ServiceBehavior">
          <serviceMetadata httpGetEnabled="true" />
          <serviceDebug includeExceptionDetailInFaults="true" />
          <dataContractSerializer maxItemsInObjectGraph="2147483647" />
        </behavior>
      </serviceBehaviors>
    </behaviors>

    <diagnostics>    
      <messageLogging maxMessagesToLog="30000"    
              logEntireMessage="true"    
              logMessagesAtServiceLevel="true"    
              logMalformedMessages="true"    
              logMessagesAtTransportLevel="true">    
      </messageLogging>    
    </diagnostics>
  </system.serviceModel>

WcfTestClient Config This is the default config file that is loaded with WcfTestClient

<?xml version="1.0" encoding="utf-8"?>
<configuration>
    <system.serviceModel>
        <bindings>
            <wsHttpBinding>
                <binding name="WSHttpBinding_IWcfService" closeTimeout="00:01:00"
                    openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
                    bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard"
                    maxBufferPoolSize="524288" maxReceivedMessageSize="65536"
                    messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true"
                    allowCookies="false">
                    <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
                        maxBytesPerRead="4096" maxNameTableCharCount="16384" />
                    <reliableSession ordered="true" inactivityTimeout="00:10:00"
                        enabled="false" />
                    <security mode="Message">
                        <transport clientCredentialType="Windows" proxyCredentialType="None"
                            realm="" />
                        <message clientCredentialType="Windows" negotiateServiceCredential="true"
                            algorithmSuite="Default" />
                    </security>
                </binding>
            </wsHttpBinding>
        </bindings>
        <client>
            <endpoint address="http://localhost:4151/Service1.svc" binding="wsHttpBinding"
                bindingConfiguration="WSHttpBinding_IWcfService" contract="IWcfService"
                name="WSHttpBinding_IWcfService">
                <identity>
                    <dns value="localhost" />
                </identity>
            </endpoint>
        </client>
    </system.serviceModel>
</configuration>

Here is the exception Stack Trace

    <E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent">

  <System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system">

    <EventID>131075</EventID>
    <Type>3</Type>
    <SubType Name="Error">0</SubType>
    <Level>2</Level>
    <TimeCreated SystemTime="2010-06-29T09:18:23.9616589Z" />
    <Source Name="System.ServiceModel" />
    <Correlation ActivityID="{7870ff09-e268-4e9d-a692-389fd03db1aa}" />
    <Execution ProcessName="WebDev.WebServer20" ProcessID="4812"
    ThreadID="17" />
    <Channel />
    <Computer>PC-008915</Computer>
  </System>
  <ApplicationData>
    <TraceData>
      <DataItem>
        <TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord"
        Severity="Error">
          <TraceIdentifier>
          http://msdn.microsoft.com/en-GB/library/System.ServiceModel.Diagnostics.ThrowingException.aspx</TraceIdentifier>
          <Description>Throwing an exception.</Description>
          <AppDomain>5ab4f443-1-129222741421051645</AppDomain>
          <Exception>
            <ExceptionType>System.InsufficientMemoryException,
            mscorlib, Version=2.0.0.0, Culture=neutral,
            PublicKeyToken=b77a5c561934e089</ExceptionType>
            <Message>Failed to allocate a managed memory buffer of
            536870912 bytes. The amount of available memory may be
            low.</Message>
            <StackTrace>at
            System.ServiceModel.Diagnostics.Utility.AllocateByteArray(Int32
            size) at
            System.ServiceModel.Channels.BufferManager.GCBufferManager.TakeBuffer(Int32
            bufferSize) at
            System.ServiceModel.Channels.BufferedOutputStream.AllocNextChunk(Int32
            minimumChunkSize) at
            System.ServiceModel.Channels.BufferedOutputStream.WriteCore(Byte[]
            buffer, Int32 offset, Int32 size) at
            System.ServiceModel.Channels.BufferedOutputStream.Write(Byte[]
            buffer, Int32 offset, Int32 size) at
            System.Xml.XmlStreamNodeWriter.FlushBuffer() at
            System.Xml.XmlStreamNodeWriter.GetBuffer(Int32 count,
            Int32&amp; offset) at
            System.Xml.XmlStreamNodeWriter.UnsafeWriteUTF8Chars(Char*
            chars, Int32 charCount) at
            System.Xml.XmlStreamNodeWriter.WriteUTF8Chars(String
            value) at
            System.Xml.XmlUTF8NodeWriter.WriteEndElement(String
            prefix, String localName) at
            System.Xml.XmlSigningNodeWriter.WriteEndElement(String
            prefix, String localName) at
            System.Xml.XmlBaseWriter.WriteEndElement() at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteString(XmlWriterDelegator
            xmlWriter, String value, XmlDictionaryString name,
            XmlDictionaryString ns) at
            WritePropertyDtoToXml(XmlWriterDelegator , Object ,
            XmlObjectSerializerWriteContext , ClassDataContract )
            at
            System.Runtime.Serialization.ClassDataContract.WriteXmlValue(XmlWriterDelegator
            xmlWriter, Object obj, XmlObjectSerializerWriteContext
            context) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator
            xmlWriter, Object obj, Boolean isDeclaredType, Boolean
            writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle
            declaredTypeHandle) at
            WriteArrayOfPropertyDtoToXml(XmlWriterDelegator ,
            Object , XmlObjectSerializerWriteContext ,
            CollectionDataContract ) at
            System.Runtime.Serialization.CollectionDataContract.WriteXmlValue(XmlWriterDelegator
            xmlWriter, Object obj, XmlObjectSerializerWriteContext
            context) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator
            xmlWriter, Object obj, Boolean isDeclaredType, Boolean
            writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle
            declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerializeReference(XmlWriterDelegator
            xmlWriter, Object obj, Boolean isDeclaredType, Boolean
            writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle
            declaredTypeHandle) at
            WriteEntityTypeDtoToXml(XmlWriterDelegator , Object ,
            XmlObjectSerializerWriteContext , ClassDataContract )
            at
            System.Runtime.Serialization.ClassDataContract.WriteXmlValue(XmlWriterDelegator
            xmlWriter, Object obj, XmlObjectSerializerWriteContext
            context) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator
            xmlWriter, Object obj, Boolean isDeclaredType, Boolean
            writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle
            declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerializeReference(XmlWriterDelegator
            xmlWriter, Object obj, Boolean isDeclaredType, Boolean
            writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle
            declaredTypeHandle) at
            WriteNodeTypeDtoToXml(XmlWriterDelegator , Object ,
            XmlObjectSerializerWriteContext , ClassDataContract )
            at
            System.Runtime.Serialization.ClassDataContract.WriteXmlValue(XmlWriterDelegator
            xmlWriter, Object obj, XmlObjectSerializerWriteContext
            context) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.InternalSerialize(XmlWriterDelegator
            xmlWriter, Object obj, Boolean isDeclaredType, Boolean
            writeXsiType, Int32 declaredTypeID, RuntimeTypeHandle
            declaredTypeHandle) at
            WriteArrayOfNodeTypeDtoToXml(XmlWriterDelegator ,
            Object , XmlObjectSerializerWriteContext ,
            CollectionDataContract ) at
            System.Runtime.Serialization.CollectionDataContract.WriteXmlValue(XmlWriterDelegator
            xmlWriter, Object obj, XmlObjectSerializerWriteContext
            context) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.WriteDataContractValue(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.XmlObjectSerializerWriteContext.SerializeWithoutXsiType(DataContract
            dataContract, XmlWriterDelegator xmlWriter, Object obj,
            RuntimeTypeHandle declaredTypeHandle) at
            System.Runtime.Serialization.DataContractSerializer.InternalWriteObjectContent(XmlWriterDelegator
            writer, Object graph) at
            System.Runtime.Serialization.DataContractSerializer.InternalWriteObject(XmlWriterDelegator
            writer, Object graph) at
            System.Runtime.Serialization.XmlObjectSerializer.WriteObjectHandleExceptions(XmlWriterDelegator
            writer, Object graph) at
            System.Runtime.Serialization.XmlObjectSerializer.WriteObject(XmlDictionaryWriter
            writer, Object graph) at
            System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.SerializeParameterPart(XmlDictionaryWriter
            writer, PartInfo part, Object graph) at
            System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.SerializeParameter(XmlDictionaryWriter
            writer, PartInfo part, Object graph) at
            System.ServiceModel.Dispatcher.DataContractSerializerOperationFormatter.SerializeBody(XmlDictionaryWriter
            writer, MessageVersion version, String action,
            MessageDescription messageDescription, Object
            returnValue, Object[] parameters, Boolean isRequest) at
            System.ServiceModel.Dispatcher.OperationFormatter.SerializeBodyContents(XmlDictionaryWriter
            writer, MessageVersion version, Object[] parameters,
            Object returnValue, Boolean isRequest) at
            System.ServiceModel.Dispatcher.OperationFormatter.OperationFormatterMessage.OperationFormatterBodyWriter.OnWriteBodyContents(XmlDictionaryWriter
            writer) at
            System.ServiceModel.Channels.BodyWriter.WriteBodyContents(XmlDictionaryWriter
            writer) at
            System.ServiceModel.Channels.BodyWriterMessage.OnWriteBodyContents(XmlDictionaryWriter
            writer) at
            System.ServiceModel.Channels.Message.WriteBodyContents(XmlDictionaryWriter
            writer) at
            System.ServiceModel.Security.SecurityAppliedMessage.WriteBodyToSignThenEncryptWithFragments(Stream
            stream, Boolean includeComments, String[]
            inclusivePrefixes, EncryptedData encryptedData,
            SymmetricAlgorithm algorithm, XmlDictionaryWriter
            writer) at
            System.ServiceModel.Security.WSSecurityOneDotZeroSendSecurityHeader.ApplyBodySecurity(XmlDictionaryWriter
            writer, IPrefixGenerator prefixGenerator) at
            System.ServiceModel.Security.SecurityAppliedMessage.OnWriteMessage(XmlDictionaryWriter
            writer) at
            System.ServiceModel.Channels.Message.WriteMessage(XmlDictionaryWriter
            writer) at
            System.ServiceModel.Channels.BufferedMessageWriter.WriteMessage(Message
            message, BufferManager bufferManager, Int32
            initialOffset, Int32 maxSizeQuota) at
            System.ServiceModel.Channels.TextMessageEncoderFactory.TextMessageEncoder.WriteMessage(Message
            message, Int32 maxMessageSize, BufferManager
            bufferManager, Int32 messageOffset) at
            System.ServiceModel.Channels.HttpOutput.SerializeBufferedMessage(Message
            message) at
            System.ServiceModel.Channels.HttpOutput.Send(TimeSpan
            timeout) at
            System.ServiceModel.Channels.HttpRequestContext.OnReply(Message
            message, TimeSpan timeout) at
            System.ServiceModel.Activation.HostedHttpContext.OnReply(Message
            message, TimeSpan timeout) at
            System.ServiceModel.Channels.RequestContextBase.Reply(Message
            message, TimeSpan timeout) at
            System.ServiceModel.Security.SecuritySessionServerSettings.SecuritySessionRequestContext.OnReply(Message
            message, TimeSpan timeout) at
            System.ServiceModel.Channels.RequestContextBase.Reply(Message
            message, TimeSpan timeout) at
            System.ServiceModel.Channels.RequestContextBase.Reply(Message
            message) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Reply(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessageCleanup(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc&amp;
            rpc) at
            System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean
            isOperationContextSet) at
            System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Dispatch(MessageRpc&amp;
            rpc, Boolean isOperationContextSet) at
            System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext
            request, Boolean cleanThread, OperationContext
            currentOperationContext) at
            System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext
            request, OperationContext currentOperationContext) at
            System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult
            result) at
            System.ServiceModel.Dispatcher.ChannelHandler.OnContinueAsyncReceive(Object
            state) at
            System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke2()
            at
            System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.OnSecurityContextCallback(Object
            o) at
            System.Security.SecurityContext.Run(SecurityContext
            securityContext, ContextCallback callback, Object
            state) at
            System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.WorkItem.Invoke()
            at
            System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ProcessCallbacks()
            at
            System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.CompletionCallback(Object
            state) at
            System.ServiceModel.Channels.IOThreadScheduler.CriticalHelper.ScheduledOverlapped.IOCallback(UInt32
            errorCode, UInt32 numBytes, NativeOverlapped*
            nativeOverlapped) at
            System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32
            error, UInt32 bytesRead, NativeOverlapped*
            nativeOverlapped) at
            System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32
            errorCode, UInt32 numBytes, NativeOverlapped*
            pOVERLAP)</StackTrace>
            <ExceptionString>System.InsufficientMemoryException:
            Failed to allocate a managed memory buffer of 536870912
            bytes. The amount of available memory may be low.
            ---&gt; System.OutOfMemoryException: Exception of type
            'System.OutOfMemoryException' was thrown. at
            System.ServiceModel.Diagnostics.Utility.AllocateByteArray(Int32
            size) --- End of inner exception stack trace
            ---</ExceptionString>
            <InnerException>
              <ExceptionType>System.OutOfMemoryException, mscorlib,
              Version=2.0.0.0, Culture=neutral,
              PublicKeyToken=b77a5c561934e089</ExceptionType>
              <Message>Exception of type
              'System.OutOfMemoryException' was thrown.</Message>
              <StackTrace>at
              System.ServiceModel.Diagnostics.Utility.AllocateByteArray(Int32
              size)</StackTrace>
              <ExceptionString>System.OutOfMemoryException:
              Exception of type 'System.OutOfMemoryException' was
              thrown. at
              System.ServiceModel.Diagnostics.Utility.AllocateByteArray(Int32
              size)</ExceptionString>
            </InnerException>
          </Exception>
        </TraceRecord>
      </DataItem>
    </TraceData>
  </ApplicationData>
</E2ETraceEvent>

Awaiting Nabeel

I'm dealing with the same error. Could you post your stacktrace so that I can see if there are any similarities between our cases? http://stackoverflow.com/questions/3016446/how-to-best-transfer-large-payloads-of-data-using-wshttp-with-wcf-with-message-se — jpierson, Jun 29 '10 at 17:20
possible duplicate of [Failed to allocate a managed memory buffer of 134217728 bytes. The amount of available memory may be low](http://stackoverflow.com/questions/20209641/failed-to-allocate-a-managed-memory-buffer-of-134217728-bytes-the-amount-of-ava) — Liam, Jun 11 '14 at 14:30

score 5 · Accepted Answer · edited May 23 '17 at 12:16

5

Are you sure your byte array is only around 2MB? From the exception message that you posted it seems like the message in the buffer is approaching 0.5GB (536870912 bytes). Is this error happening consistently? Are you running multiple service calls at the time this occurs?

Also understand that when using WSHttpBinding with Message Security that the whole message is buffered. My understanding is that this is done to encrypt the whole message in one block so that the corresponding details can be supplied in the header. In other words the header needs the length of the message which is unknown until the encryption is done.

See my related Question where I ask How to best transfer large payloads of data using wsHttp with WCF with message security.

edited May 23 '17 at 12:16

Community

1
1

answered Jun 29 '10 at 17:26

jpierson

13,736
10
94
137

1

Hi JPierson, The byte array is actually 1 MB. Before returning the array from the service, I am binary serializing it to a file on the disk and the file size is 1MB. And I don't think its very large, 1 MB is nothing infact unless wcf itself is serializing it as 0.5 GB like you said, but i don't understand why would it do that. Nabeel – nabeelfarid Jun 30 '10 at 08:44
Well, by default WCF uses the DataContractSerializer which you can test yourself by instantiating one and serializing the same byte array and saving the output to disk as well. This way you can see the serialized content and see checks its size as well. Another option would be to use the WCF tracing utilities to take a look at the messages. – jpierson Jun 30 '10 at 16:22
You may also want to look at the MaxBytesPerRead property as well as the other constraints. There is a nice list of these at the bottom of the post in the following link. http://nirajrules.wordpress.com/2009/08/03/mtom-vs-streaming-vs-compression-%E2%80%93-large-attachments-over-wcf/ – jpierson Jun 30 '10 at 16:25
1

Thanks Pierson. I will have a look. – nabeelfarid Jul 05 '10 at 08:38

score 1 · Answer 2 · answered Jun 29 '15 at 07:16

Our Application Pool was set to Recycle every 0 minutes (not recycle). After changing it back to 1740, and manually recycling the pool, the error went away. I think the Application Pool runs out of available / committed memory and returns that message if all your objects haven't been disposed properly in .NET

score 0 · Answer 3 · answered Oct 21 '16 at 07:01

We also had exactly the same problem. Tried to download a document of only 15mb and the memory usage sky rocketed to about 1gb. This resulted in an outofmemory exception on the server. The solution for us was very simple. The data contract implemented an ICollection for the ByteArray. When we changed this in to a Byte[] the problem was gone and the memory usage was normal again. Apparently the serialization process can not handle the ICollection type in an efficient way.

Wcf System.InsufficientMemoryException. Failed to allocate a managed memory buffer of 536870912 bytes

3 Answers3