Using Vagrant to Virtualize Multiple Hard Drives

on ansible, vagrant, virtual machine, devops

Recently I have been working on upgrading my home server. I keep a home server where me and the wife can back up our files and store our media. The server is really just a spare desktop with some hard drives, not speciailized server hardware, but server our purpose well.

I built the machine in 2012 so by now it is running an older version of Ubuntu (Ubuntu 12.04). I got some new hard drives for the server to expand the storage and felt it was a good time to also upgrade the software.

My main goal was to make the backup system more robust. After reading around on other blogs and reddit I settled on a solution that uses mergerfs to mount the drives to a central point and SnapRAID to create a parity disk for backups. This was a good solution for me because I don't really care about real-time backups of the data, so SnapRAID can run weekly and update the parity disk (if I store something critical, I can always manually invoke SnapRAID). For more information on SnapRAID/mergerfs I recommend this article done by Linux Server IO.

My next task was to model all this using Vagrant + Ansible. I essientially wanted a Vagrant environment I could use to test the new OS, new disk configuration and my Ansible playbook to provision the machine. This will be helpful in the future as well, when I want to install something new on my server I can run all the changes on the VM and test before pushing the update to my server.

To accurately write and test my Ansible playbook and test my configuration for MergerFS and SnapRaid I needed to virtualize multiple hard drives using Vagrant and VirtualBox. My final server consists of 5 total hard drives and wanted to make sure the migration was going to be smooth.

Below is the complete Vagrantfile I ended up with to attach additional hard drives to a Vagrant VM:

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|  
  config.vm.box = "geerlingguy/ubuntu1604"

  # Create a private network, which allows host-only access to the machine
  # using a specific IP.
  config.vm.network "private_network", ip: "192.168.30.200"

  parityDisk = './parityDisk.vdi'
  dataDisk1 = './dataDisk1.vdi'
  dataDisk2 = './dataDisk2.vdi'

  config.vm.provider "virtualbox" do |vb|
    vb.memory = "2048"

    # Building disk files if they don't exist
    if not File.exists?(parityDisk)
      vb.customize ['createhd', '--filename', parityDisk, '--variant', 'Fixed', '--size', 10 * 1024]
    end
    if not File.exists?(dataDisk1)
      vb.customize ['createhd', '--filename', dataDisk1, '--variant', 'Fixed', '--size', 10 * 1024]
    end
    if not File.exists?(dataDisk2)
      vb.customize ['createhd', '--filename', dataDisk2, '--variant', 'Fixed', '--size', 10 * 1024]

      # Adding a SATA controller that allows 4 hard drives
      vb.customize ['storagectl', :id, '--name', 'SATA Controller', '--add', 'sata', '--portcount', 4]
      # Attaching the disks using the SATA controller
      vb.customize ['storageattach', :id,  '--storagectl', 'SATA Controller', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', parityDisk]
      vb.customize ['storageattach', :id,  '--storagectl', 'SATA Controller', '--port', 2, '--device', 0, '--type', 'hdd', '--medium', dataDisk1]
      vb.customize ['storageattach', :id,  '--storagectl', 'SATA Controller', '--port', 3, '--device', 0, '--type', 'hdd', '--medium', dataDisk2]
    end
  end

  config.vm.provision "shell", inline: <<-SHELL
    sudo mkfs.ext4 /dev/sdb
    sudo mkfs.ext4 /dev/sdc
    sudo mkfs.ext4 /dev/sdd
  SHELL

end

The first thing in the Vagrantfile of interest is declaring the disk names that we will use. These files will appear in your working directory (I recommend adding them to your .gitignore).

parityDisk = './parityDisk.vdi'  
dataDisk1 = './dataDisk1.vdi'  
dataDisk2 = './dataDisk2.vdi'

Now we need the specifiy the actual VirtualBox command to create these hard drive files. The below snippet checks if the hard drive file exists and if not executes the "createhd" command of the VirtualBox command line to create the hard drive. In my case I cam creating a fixed hard drive of 10GB.

if not File.exists?(parityDisk)  
      vb.customize ['createhd', '--filename', parityDisk, '--variant', 'Fixed', '--size', 10 * 1024]
end

After creating the last hard drive we need to attach them to Virtual Machine. To do that, we create a SATA storage controller and use that to attach the hard drive files to the VM. Important to note here is that the SATA controller supports 4 disks and when attaching the storage I assign a different port to each drive.

if not File.exists?(dataDisk2)  
      vb.customize ['createhd', '--filename', dataDisk2, '--variant', 'Fixed', '--size', 10 * 1024]

      # Adding a SATA controller that allows 4 hard drives
      vb.customize ['storagectl', :id, '--name', 'SATA Controller', '--add', 'sata', '--portcount', 4]
      # Attaching the disks using the SATA controller
      vb.customize ['storageattach', :id,  '--storagectl', 'SATA Controller', '--port', 1, '--device', 0, '--type', 'hdd', '--medium', parityDisk]
      vb.customize ['storageattach', :id,  '--storagectl', 'SATA Controller', '--port', 2, '--device', 0, '--type', 'hdd', '--medium', dataDisk1]
      vb.customize ['storageattach', :id,  '--storagectl', 'SATA Controller', '--port', 3, '--device', 0, '--type', 'hdd', '--medium', dataDisk2]
end

Now that the drives are attached to the VM we want to format them so they are ready to use. The shell provisioner can be used to execute the correct commands to format the drives as EXT4.

config.vm.provision "shell", inline: <<-SHELL  
    sudo mkfs.ext4 /dev/sdb
    sudo mkfs.ext4 /dev/sdc
    sudo mkfs.ext4 /dev/sdd
SHELL

Overall, this method worked well for me and I was able to write and test my larger Ansible playbook on a VM closer to my targeted production hardware. Using the vb.customize API of Vagrant allows you to do some complex setups for virtual machines.

comments powered by Disqus