If you want to preserve the genericity, you will have to reimplement something similar to a list and I don't think it is worth it.
You said:
The first is that you can add and remove elements to and from a list.
Well you can create an unmodifiable list:
List<E> list = Collections.unmodifiableList(Arrays.asList(a, b, c, d));
The second is that I'm creating a quadtree data structure and using a list wouldn't be too good for performance. Quadtrees have a lot of quadrants and using lists would decrease performance significantly.
First you can initialise the list to the right size:
List<E> list = new ArrayList<>(4);
Once you have done that, the list will only use a little bit more memory than an array (probably 8 bytes: 4 byte for the backing array reference and another 4 byte for the size).
And in terms of performance an ArrayList performs almost as good as an array.
Bottom line: I would start by using a list and measure the performance. If it is not good enough AND it is due to using a list instead of an array, then you will have to adapt your design - but I doubt that this will be your main issue.