Infrastructure & Automation
- I am developing
Strata, a declarative cloud provisioning tool with Cluster API pivot automation functionality. I was inspired by the question, "what if a computer worm could provision a cloud?" I likeFlatcar Container Linux(down the road, with moresDDFdrivers for multiplexed paravirtualization, I'm also interested in using something likeLionsOSandlibvmmbeneath it as an seL4 hypervisor). I'm experimenting with usingPiandOverstoryto help me write code andStrictDocfor requirements management. Cloudflareis great for web domain registration, denial of service attack protection, and tunneling private networks to the public. This website showcases its free web hosting capability.- Building a distributed private cloud with full disk encryption and remote attestation to protect from BootKit malware (using
Kubernetes[k0sis awesome],Flatcar Container Linux,Clevis [TPM & Tang], andKeylime). I am interested in using hardware security modules (HSMs) andPKCS#11to wrap data encryption keys (DEKs) with key encryption keys (KEKs) and store them in a remote key management system (KMIP). I've thought about usingSonoff S31smart plugs (industrial control systems are expensive and unncessarily complicated for small office server racks. I currently use two smallCyberPower CP1500PFCLCDuninterruptible power supplies (UPS) that attempt to protect my computers from power surges and brownouts by regulating the voltage of the electricity supplied to them. I was careful to choose a UPS model that can be programmed withNetwork UPS Toolsand theTripp Lite SRCOOL12KEair conditioner is on my wishlist for when I build a rack for my servers. Smart plugs are modular IoT edge infrastructure that can be secured on an air gapped network and isolated in a VLAN [I likeSONiCfor switches andVyOSis okay for routers (I'm thankful for their charitability in open sourcing their rolling release and understand the monetization of their work with their commercial long-term support (LTS) release. I interpret their product to be mostly a very convenient configuration layer that sews together underlying FOSS tools such asFRR,VPP,strongSwan,HAProxy, etc.)]. The smart plugs supportRedfishand can be used withMetal3andMediK8sfor automatic node remediation [like if the computers' operating systems gets stuck and can't shutdown, the smart plug can be programmed to turn off and on the power to fix it]) with Tasmota firmware (the screws always strip. Being careful with a drill jig and a small bit worked for me. I didn't bother to solder I just used test leads like this video) to monitor energy consumption andKeplerto monitor compute cycles (what % of the power bill is used by what % of the smart plug is used by what % of the compute cycles?) like miles on a car to itemize tax deductions. - I've been looking into
Ceph(Rook) for its multi-site, geographically redundant, high availablility (HA), learning about RADOS gateway S3 data bucket lifecycle policies for automating data migration between hot/warm/cold tiers based on usage. - Planning a monolithic repository for applications with static code analysis tooling (using
Git[Scalar,Large File Storage (LFS),ForgejoandCodeberg],Opengrep). - Architecting CI/CD pipelines capable of monorepo scale (using
Nix,Bazel, andArgo). - Developing containers, scanning them for vulnerabilities, signing them, and hosting them in a private repository (using
Podman,Wolfi,Trivy,Sigstore Cosign, andArtifact Hub). - Securing my cloud (using
Wazuh SIEM/XDR,Suricata, andClamAVwith additionalYARArules for ring 3/userland protection. StudyingTetragonpolicies for ring 0/kernel. Fun fact: the ring paradigm comes fromMultics). I want a micro-cut shredder to lawfully shred documents before burning them. It's important to know what documents are required to be archived and which to dispose of. Similarly, for ephemeral media theProton 1100 degaussing wandseems like an affordable entrypoint to the realm of degaussing. I would still useShredOSbefore waving the wand, though. - The privacy and security risk of leaking trade secrets and sensitive information to public AI models is concerning. I'm amazed at what consumer hardware is capable of. eBay is a great resource for cheap enterprise hardware. Not everybody needs the latest and greatest.
3D integrated circuits and neuromorphic processing that avoids systolic arrays (this video explains it well), are the future, and AI is needed to design them. Proprietary software is too expensive for me. The closest open source equivalents require development to catch up. ASML lithography machines are still the best, and there’s only a couple fabs in the world that can afford them. The licenses for proprietary instruction set architectures (ISA) are also too expensive for me. I love RISC-V’s free and open source nature. It’s backed by the Linux Foundation. It now has mainline GPU support, but it’s still not as featureful as x86-64/ARM. I wish for Sylkan to remove the Rusticl-on-Zink translation and go straight to Vulkan.
Concerning the environmental footprint of AI, I like this article funded by the Tarbell Center for AI Journalism. Most data centers are inefficient. They consume massive amounts of energy and water, straining our resources. I’ve found some developing solutions we may see soon:
As a preface, energy powers data centers, generating that energy often consumes water, and water is also used to cool the data center and dissipate the heat created in the process of computation.
Compute & Storage: Use efficient processors and storage. Similarly, hard drives for SSDs—especially EDSFF form factors, which offer higher storage density in a smaller footprint, cutting energy use and cooling needs.
Cooling: Build data centers in colder climates, and adopt efficient cooling technologies. Two-phase immersion cooling fascinates me. It’s wild to see expensive computers submerged in what looks like water, but it’s actually a non-conductive (dielectric) refrigerant. Submerged in a reservoir, the computers heat up, the refrigerant boils and evaporates, the gas cools on a condenser back into a liquid, then returns to the reservoir. Some older refrigerants were per- and polyfluoroalkyl substances (PFAS) which are terrible for the environment, but regulations spurred safer and more sustainable alternatives like hydrofluoroolefins (HFOs). These still contain fluorine, but fluorine is found in most modern refrigerants due to having chemical properties that are generally well suited for transferring heat. The European Union’s push (Regulation (EU) 2024/573) to phase out hydrofluorocarbons (HFCs) by 2050 worried me. Though, HFOs are not included in the EU’s phase-out initiative, nor the EU PFAS Restriction Proposal (REACH). Instead, controls are introduced based on global warming potential (GWP) thresholds, and thankfully HFOs have a low GWP.
Energy: Hot take: some molten salt reactors (MSRs) don’t need water for cooling and can deliver the energy exascale data centers demand. It’s nuclear, but when done right, it is safe and pretty clean. If you’re anti-uranium, thorium is an excellent alternative that thrives in MSRs. Either way, the fuel can be recycled to reduce nuclear waste buildup: about 96% of uranium fuel is recyclable, and thorium fuel offers even greater recyclability, with nearly all of it being reusable.
Beyond this, software should be efficient, and waste heat from data centers should be repurposed. While heat pipes efficiently transfer heat over short to moderate distances, they are not ideal for long-range transfer. Given this, a viable option for repurposing waste heat is to supply warmth to nearby buildings including ancillary facilities on the data center campus and local communities. Other options exist, but using heat for warmth makes sense to me.