Resizing Out-of-Space Linux OS Disk on Microsoft Azure

Some weeks ago I awoke to one of my websites dead. A week before, I had upgraded the Virtual Machine’s MySQL from 5.6 to 8. More on that later.

When I SSH’d into the Virtual Machine, the disk space was surprisingly all used up to the point where the find command had not enough space to write to stdout. From here, I resized the OS disk on Azure by going to:

Select the Virtual Machine from Azure
Stop the VM from running
Disks -> OS Disk
Size + Performance
Moved up the disk one tier (which was 32->64GiB for me), keeping the performance tier as default
Start the VM

Generally, this is where it stops, Azure should have automatically resized the disk and its partition to match, and once you log back in your disk space should match the newly resized amount. However, in my case, it seems that the VM entered a bad state where the space was provisioned but the partition was not resized, essentially having unallocated space.

What Happens if OS Disk Partition Does Not Automatically Resize

If that happens to you, we now have to allocate the space to the partition. Starting the VM might take significantly longer (15 minutes for me) as it struggles to come alive.

Hopefully, it does come back up and we can run sudo resize2fs /dev/sda1 to resize the disk. Verify this worked with df -h.

Aftermath

Finally, with the VM running well again, I was able to use find / -xdev -type f -size +100M -exec du -sh {} ';' | sort -rh | head -n50 to find what was taking up all the space. Turns out, when MySQL updated to 8.x, “binary logging is enabled by default, whether or not you specify the –log-bin option” according to the documentation. This resulted in a lot of large *-bin.* files appearing in /var/lib/mysql/, causing the system to run out of space.

I later found a solution from Twitter user @samsureshx: https://twitter.com/samsureshx/status/1361825580937662464

Which is to essentially write