Not seeing performance increase running django test /w postgresql on ramdrive

Question

For a django project, the tests are running slower and slower as the models grow and the number of migrations as well. A single unittest i have timed now at 200s.

I followed this post: http://tech.marksblogg.com/test-django-on-ram-drive.html to work on a ramdrive only for the testing but strangly enough i'm not seeing any improvement.... So i expect something is not going as it should...

I have debugged some and did see the TABLESPACE statements being generated towards postgres, for instance:

... CREATE TABLE ""django_content_type"" (""id"" serial NOT NULL PRIMARY KEY USING INDEX TABLESPACE ""ram_disk"", ""name"" varchar(100) NOT NULL, ""app_label"" varchar(100) NOT NULL, ""model"" varchar(100) NOT NULL) TABLESPACE ""ram_disk""",,,,,,,,,"

Could it be that postgresql rejects it? And how can i test/see if the ram drive is actually being used?

Paul

The advice given in that blog is pretty dire. Don't do that. See https://stackoverflow.com/q/9407442/398670 — Craig Ringer, Jul 26 '17 at 15:00

score 0 · Answer 1 · answered Jul 26 '17 at 15:02

The advice given in that blog is pretty poor. Don't do that. Not only is it unsafe if you have anything else on your database that you care about, but it's also not going to address WAL and fsync costs, so it won't even be all that much faster.

A ramdisk offers little to no benefit on a modern virtual memory system. You're better off just using postgres as normal, with durability protections turned off for testing purposes.

See Optimise PostgreSQL for fast testing for tips on that.

If you must use a ramdisk / tempfs for some reason, initdb a whole new postgres instance there when you bring up your system. You'll get better results and it's safer.

score 0 · Answer 2 · answered Jul 27 '17 at 11:02

Thanks for that link.

I don't really need the ramdrive, i just eager for performance increase on running my tests.

Turning off fsync=off and full_page_writes does not affect performance much:

python manage.py test -v3 --noinput 192,49s user 0,69s system 92% cpu 3:28,44 total

vs

python manage.py test -v3 --noinput 200,73s user 0,71s system 94% cpu 3:32,38 total

I think actually it's my ssd driver that seems to get slower and slower? In the post about ramdrive there is a simple performance check, when i run on against my ssd:

dd if=/dev/zero \
  of=/tmp/benchmark \
  conv=fdatasync \
  bs=4k \
  count=100000 \
  && rm -f /tmp/benchmark

409600000 bytes (410 MB) copied, 11,6737 s, 35,1 MB/s

versus the same test against the ram driver:

409600000 bytes (410 MB) copied, 0,16099 s, 2,5 GB/s

Actually i expect more than 35MB/s....

Could also be ubuntu/driver issue maybe?

Paul

Not seeing performance increase running django test /w postgresql on ramdrive

2 Answers2