Assuming I have a ASP.NET MVC 3 application that runs in a web farm where each web server belongs to a workgroup (as appose to a domain with shared accounts). The web farm is also auto scalable, meaning that the number of instances are dependent on the load. Sensitive data is encrypted and decrypted when stored/retrieved from the database. The symmetric and asymmetric keys are stored on each machine and protected with ACL and encrypted using DAPI (using the machine key).
For compliance and security reasons it is required that keys be rotated on a regular interval. How would you design/modify the system to automatically rotate keys at a regular interval without bringing the system offline? Assume that there are an arbitrary number of tables each with an arbitrary number of columns that are encrypted using the keys.
Many Q&A are related to which algorithms to use and how to secure the keys, however few actually address how to design and implement an application that would allow those keys were to be rotated, especially in a dynamic environment (autoscaling environment) sharing a database.