4

I have some C++ code that communicates with Matlab via the Engine C API. My code creates temporary variables in the Matlab workspace, which it diligently cleans up via clear calls as soon as possible. However, at some point, my application fails, telling me that it is unable to create the next Matlab temporary variable (usually after ~65530 such operations).

After some experimentation on the Matlab command line, I discovered I could recreate this problem in pure Matlab (that is, independent of my C++ code and its use of the Engine API). Consider the following code:

for i = 1 : 100000
  eval(sprintf('x_%d = %d', i, i));
  whos
  eval(sprintf('clear x_%d', i));
  whos
end

Executing this code on my 32-bit windows laptop with Matlab R2008B (ancient, I know), the loop eventually aborts with the error message:

The current workspace already has too many variables; there is no room for "x_65532".

So, it seems that at least this obsolete version of Matlab has a 64K symbol table limit. Perhaps the limit is bigger on newer (64-bit) versions of Matlab--I'd be interested to hear what results others get.

However, the more interesting question is what effect the clear call is having and how to work around its odd behavior. Here is the output from an iteration a little prior to the abort:

x_65530 =

       65530

  Name         Size            Bytes  Class     Attributes

  i            1x1                 8  double              
  x_65530      1x1                 8  double              

  Name      Size            Bytes  Class     Attributes

  i         1x1                 8  double

As you can see, the whos output clearly shows that the temporaries from the prior iterations have been removed from the workspace, and the clear seemingly works as expected. Nonetheless, the symbol table has apparently reached capacity.

So, two questions for the SO faithful:

  1. How can I work around this somewhat arbitrary limit? That is, what should my clear calls be replaced with?
  2. Has this behavior changed with newer and/or 64-bit versions of Matlab?
Drew Hall
  • 26,700
  • 10
  • 58
  • 79
  • Can you tell more about the use cases which require so many temporary variables? P.S: you don't need `eval` for the first `sprintf` inside the loop... – Eitan T Dec 02 '13 at 12:32
  • 1
    Replicable in 2011b (32 bit) - exact same error point. The interesting bit is that if I then define additional variables at the command line or re-run the loop I don't get an error (until I hit that point in the loop again). – nkjt Dec 02 '13 at 12:36
  • @EitanT: Use case: my C++ code animates a matlab plot, sending it lots of new data in each frame. Probably there's a better way to do it, but it works (at least until it hits the limit). As for the eval, are you sure? I'm explicitly trying to create a variable with an algorithmically chosen, unique, name. Skipping the eval would give me an unnamed temporary string, no? – Drew Hall Dec 02 '13 at 12:37
  • @nkjt: Interesting. Perhaps the loop context has its own symbol table, subordinate to the global table? My C++ code that fails doesn't do this within a matlab for loop, BTW--so I'm likely polluting the global table. All speculation, of course... – Drew Hall Dec 02 '13 at 12:47
  • Also reproducible on 64bit 2011b. The question however is, why oh why, are you doing such a nasty thing? Why not take a single variable and update its value? If you additionally need the `i`, extracted from the name, than you should simply add a second variable - or use one struct-variable e.g. with fields `x` and `i`, whos values you can update. – sebastian Dec 02 '13 at 12:50
  • @sebastian: Thanks for the data point. As for the reason for doing so, I've got a TempVar class in C++ that holds data that I'm shooting over to Matlab. It maintains a static count of the number of TempVars created so that it can keep them from interfering with each other and clear the appropriate Matlab variable in the TempVar destructor. I'm sure there's a more elegant way to do it, but my way doesn't run afoul of any matlab/mex/engine API documentation I can find so I would expect it to work as well. I will, however, revisit my code to see if a more elegant approach can be taken. – Drew Hall Dec 02 '13 at 12:57
  • 1
    Ok, so not all TempVar instances necessarily hold similar information? I was assuming that all the variables are read by the same "client", in which case updating a single variable seemed much better. If you have to stick to the variable naming, you could try to re-use variable names, once they were cleared. – sebastian Dec 02 '13 at 13:09
  • @sebastian: Your name re-use idea might just do the trick--I'd maintain a "free list" of IDs and pull from that unless I was forced to use a new ID. Will report back once I try this approach. – Drew Hall Dec 02 '13 at 22:24
  • @sebastian: I implemented your "temp name recycling" scheme and managed to avoid triggering this failure mode--thanks for the advice. If you turn that into an answer I'll accept it. – Drew Hall Dec 10 '13 at 02:00

4 Answers4

2

Repeating my comment in form of an answer:

If you have to stick to the variable naming, you could try to re-use variable names, once they were cleared, avoiding the creation of 65xxx different variable names.

sebastian
  • 8,870
  • 20
  • 49
2

The answer to 2) is that this restriction applies to older Matlab versions. While the limit is still present in R2014b, it is not present anymore in R2015b.

jpfeifer
  • 51
  • 2
1

The point is that in matlab you shouldn't need to have that many variables. Use bigger variables rather than many variables.

If each x is a scalar: use a vector

If each x has the same size: use a matrix

If each x is different, use a cell array!

With x{65530} = magic(3) you will not be anywhere near the limit.

Dennis Jaheruddin
  • 19,745
  • 7
  • 58
  • 100
0

I just stumbled on the very same thing. From Python I am creating temporal variables in a Matlab instance, that are later cleared. To make sure I was not overwriting another temporal variable, I made new names form them using the uuid() Python module. This was supposed to be a great idea. However, after a while I got an exception, with the message 'The current workspace already has too many variables'. Executing whos I can see they are just 8 variables, so I must conclude that clear does not actually remove the variable from the workspace. This is really hard to understand, and really inconvenient. It is true I can use names temp001, temp002, temp003, but then when I need a new temporal variable I must make sure I am not currently using that name already. Uuids were invented to avoid this kind of problems, but it seems Matlab awkward behavior is preventing me from using them. Has anyone found a good solution to REALLY delete variables from Matlab's workspace?

zeycus
  • 720
  • 1
  • 5
  • 16