I need to deploy Lustre filesystem on AWS. I cannot use the Amazon FSx for Lustre but rather I need to create all the nodes running on EC2 instances.
Also I need a docker configuration for this to simulate it locally. I imagine I need to use some CentOS image for this.
I am kind of lost - is the solution I need even possible? Can somebody please provide me with steps how to achieve this?
I'm working on this myself right now. As far as I understand it:
You need your host kernel to support the lustre module, and you need to install the lustre client at the place where you intend to mount the lustre filesystem. Then you would load the module at some point before you have to mount the filesystem.
If you intend to call mount -t lustre lustre_endpoint /mnt/my_lustre from inside a container, your container will need to be privileged (i.e. docker run --privileged ...) or will need to be granded special linux mount capabilities. Another option is to mount the lustre filesystem on the docker host (e.g. /mnt/my_lustre), and then use a volume mount or a bind mount to access a subportion of that into the container.
There are specific instructions if you're trying to use Lustre FSx (which is the AWS version of the Lustre service). To install the FSx modules and client libraries, you would need to follow the instructions here: https://docs.aws.amazon.com/fsx/latest/LustreGuide/install-lustre-client.html Take special note of the kernel requirements. You may skip steps 1-6 if the kernel version you have is above the one listed.
For a vanilla Lustre filesystem (or a Lustre filesystem not managed by AWS), you should, probably, follow the instructions from the lustre project. Because then, there's nothing specific about AWS.
