In a previous blog, I presented the topology of Openstack Swift Object Storage. That is, how Swift uses XFS to store Accounts, Containers, and Objects.
A separate Account Database is created for each Account and an Account Database is a SQLite instantiation in the file system. An Account Database consists of a Account Stat Table and a Container Table. The Account Stat Table contains meta-data for that specific account. The Container Table contains an entry for each container in that account.
Similarly, a separate Container Database is created for each Containerand an Container Database is an SQLite instantiation in the file system. A Container Database consists of a Container Stat Table and an Object Table. The Container Stat Table contains meta-data for that specific container. The Object Table contains an entry for each object in that container.
And finally, object data is stored in XFS files.
So what happens when a Swift Storage Device becomes full? The storage device is managed by XFS, so the file system will run out of space when this happens. Thus XFS will NOT be able to perform subsequent writes, but reads and deletes will still work.
What happens with the Swift REST API? Reads (GETs) continue to work, this is good.
Writes (PUTs and POSTs) obviously do not workand this is as expected. But what about deletes (DELETEs)? If a Swift Account is “full”, then users should be able to free up space in the account.
The sad, fact of the matter is that deletes do NOT work at this point. A DELETE Object operation fails with a 503 Internal Error. This error is the result of a Proxy Server exception. The exception occures because a database update fails. To understand this exception, we must understand how objects are deleted inSwift.
As previously stated, a Container Database has an Object Table, which contains an entry for each object in it. The Object Data itself is stored in an XFS file. So to delete an object, the appropriate Object Table entry has to be updatedand marked as “deleted”. The Object Data is left for the Object Reaper to clean up at a later time.
The Problem
The problem is with the Object Table update. This is a write, and since the file system is full, this update will fail.
So how do we get around this problem? The Swift Account is full and users CANNOT free up account space. Currently, OpenStack recommends monitoring the Storage Devices so that an alarmis generated when a device becomes 80% full. The Swift Administrator then, has to add new Storage Devices to the Swift Rings. This involves modifying the rings and restarting the Proxy Servers. The Account will no longer be full, since it now has additional space, but users will NEVER be able to delete objects and containers from the fullstorage device.
The Solution?
This solution is less than adequate and is clearly lacking in Enterprise quality. I would now like to propose more robust solutions, two in fact.
Solution 1: create a utility that removes objects from the full Storage Device. This utility would remove the Object Data file and then update the Object Table. The customer would obviously have to identify which objects should be removed. Once there is enough free space, the customer could then use the normal DELETE operations to free up further
space. This solution is good in that it allows an account to be quickly repaired, but it is bad in that a cluster administrator has to perform the action.
Soultion 2: create a configurable Swift Storage Device High Water Mark (HWM). The HWM would be expressed as a storage percentage, that is how full can any Storage Device get before PUT operations are turned off? So when the HWM is hit for any Storage Device, all PUT operations would fail for all accounts. But users could still perform GETs, POSTs, and DELETEs. They could thus still manage their accounts. The cluster admin would then have time to add storage devices, without users clammering about “full” accounts.