buildkitd: Run in privileged mode
We've run into issues with running buildkitd in rootless mode, namely that its OCI worker spawns all build containers in the same process, user, and network namespace. Disruption of other build processes is possible as are port collisions.
In general, it seems there's a giant tradeoff between isolation of buildkitd itself from the host system and isolation between container processes. Given that the latter are untrusted processes of our system, and not buildkitd itself, we should solve for the best isolation possible between them.
Unfortunately there are not simply a minimal set of capabilities that
can satisfy buildkitd as it needs to rw to both /sys
/sys/fs/cgroup
which comes only from privileged mode.
Bug: T329213