Difference between revisions of "Infrastructure:Bastille"

From the Linux and Unix Users Group at Virginia Teck Wiki
Jump to: navigation, search
m
 
(3 intermediate revisions by the same user not shown)
Line 28: Line 28:
  
 
They are at the moment netbooted from [[Infrastructure:Vesuvius]] as NixOS k3s nodes.
 
They are at the moment netbooted from [[Infrastructure:Vesuvius]] as NixOS k3s nodes.
−
 
 
{| class="wikitable sortable"
 
{| class="wikitable sortable"
−
! Hostname !! Frontend IP4 !! Backend IP4 !! Notes
+
! Hostname !! Frontend IP4 !! Backend IP4 !! Global IP6 !! Notes
 
|-
 
|-
−
| [[Infrastructure:Vesuvius|vesuvius]] || 10.98.0.23 || 10.98.3.1 || k3s control plane; PXE boot image server
+
| [[Infrastructure:Vesuvius|vesuvius]] || 10.98.0.23 || 10.98.3.1 || 2607:b400:6:cc80:21e:67ff:feca:9c12 || k3s control plane; PXE boot image server
 
|-
 
|-
−
| backbiter || 10.98.0.25 || 10.98.3.2 ||
+
| backbiter || 10.98.0.25 || 10.98.3.2 || 2607:b400:6:cc80:42f2:e9ff:fec6:655f ||
 
|-
 
|-
−
| damocles || 10.98.0.26 || 10.98.3.3 ||
+
| damocles || 10.98.0.26 || 10.98.3.3 || 2607:b400:6:cc80:42f2:e9ff:fec6:67db ||
 
|-
 
|-
−
| durendal || 10.98.0.27 || 10.98.3.4 ||
+
| durendal || 10.98.0.27 || 10.98.3.4 || 2607:b400:6:cc80:42f2:e9ff:fec6:6895 ||
 
|-
 
|-
−
| excalibur || 10.98.0.29 || 10.98.3.5 ||
+
| excalibur || 10.98.0.29 || 10.98.3.5 || 2607:b400:6:cc80:42f2:e9ff:fec6:6967 ||
 
|-
 
|-
−
| eyelander || 10.98.0.28 || 10.98.3.6 ||
+
| eyelander || 10.98.0.28 || 10.98.3.6 || 2607:b400:6:cc80:42f2:e9ff:fec6:6943 ||
 
|-
 
|-
−
| gram || 10.98.0.30 || 10.98.3.7 ||
+
| gram || 10.98.0.30 || 10.98.3.7 || 2607:b400:6:cc80:42f2:e9ff:fec6:7459 ||
 
|-
 
|-
−
| [[Infrastructure:Prospit|prospit]] || - || - || [[User:Enzoisotton]] took this out of the server room to test pxe booting, yell at him until it's back
+
| [[Infrastructure:Prospit|prospit]] || - || - || - || [[User:Enzoisotton]] took this out of the server room to test pxe booting, yell at him until it's back
 
|-
 
|-
−
| gryffindor || 10.98.0.31 || 10.98.3.8 ||
+
| gryffindor || 10.98.0.31 || 10.98.3.8 || 2607:b400:6:cc80:42f2:e9ff:fec6:74d7 ||
 
|-
 
|-
−
| kusanagi || 10.98.0.32 || - || the SFP+ NIC appears broken. may work if re-seated.
+
| kusanagi || 10.98.0.32 || - || 2607:b400:6:cc80:42f2:e9ff:fec6:7597 || the SFP+ NIC appears broken. may work if re-seated.
 
|-
 
|-
−
| narsil || 10.98.0.33 || 10.98.3.10 ||
+
| narsil || 10.98.0.33 || 10.98.3.10 || 2607:b400:6:cc80:42f2:e9ff:fec6:759d ||
 
|-
 
|-
−
| oathbringer || 10.98.0.34 || 10.98.3.11 ||
+
| oathbringer || 10.98.0.34 || 10.98.3.11 || 2607:b400:6:cc80:42f2:e9ff:fec6:75f1 ||
 
|-
 
|-
−
| riptide || 10.98.0.35 || 10.98.3.13 ||
+
| riptide || 10.98.0.35 || 10.98.3.13 || 2607:b400:6:cc80:42f2:e9ff:fec6:7621 ||
 
|-
 
|-
−
| sting || 10.98.0.36 || 10.98.3.14 ||
+
| sting || 10.98.0.36 || 10.98.3.14 || 2607:b400:6:cc80:42f2:e9ff:fec6:7723 ||
 
|}
 
|}
 +
 +
Each machine can be remotely accessed on port 2222 from it's global IP6 address, or by DNS at <name>.vtluug.org.
  
 
Each machine has a management interface on 10.98.2.0/24, with the same last octet as its backend IP -- i.e. riptide can be managed from 10.98.2.13.
 
Each machine has a management interface on 10.98.2.0/24, with the same last octet as its backend IP -- i.e. riptide can be managed from 10.98.2.13.
Line 70: Line 71:
  
 
The frontend networks, uses DHCP with [[Infrastructure:Shellshock]], while the backend network is statically routed.
 
The frontend networks, uses DHCP with [[Infrastructure:Shellshock]], while the backend network is statically routed.
 +
 +
The blades have completely ephemeral disks -- the k3s node passwords and join tokens are shipped via agenix, targeting an ssh host key that is sealed by TPM1.2 and kept in the netboot image.
  
 
You can spy their config here [https://github.com/vtluug/construct/blob/main/hosts/bastille/blade.nix]  
 
You can spy their config here [https://github.com/vtluug/construct/blob/main/hosts/bastille/blade.nix]  
Line 75: Line 78:
 
=== Operational ===
 
=== Operational ===
  
−
All users with a VTLUUG SSO account (read: FreeIPA/acidburn) can SSH to [[Infrastructure:Vesuvius]] on the VTLUUG LAN.
+
All users with a VTLUUG SSO account (read: FreeIPA/acidburn) can SSH to [[Infrastructure:Vesuvius]] from the VTLUUG LAN or the public IPv6 address.
 +
 
 +
From there, all users with sudo access can use `kubectl` to manage the cluster, or proxy/jump to other nodes through the backend fabric.
  
−
From there, all users with sudo access can use `kubectl` to manage the cluster, or proxy/jump to other nodes.
+
Non-admin tenant namespaces coming Soon^{tm}
  
 
== Things To Host On All This (Post Suggestions!!!) ==
 
== Things To Host On All This (Post Suggestions!!!) ==

Latest revision as of 04:42, 14 September 2026

Bastille is our Lenovo Nextscale chassis donated by jpo from his VT Surplus spoils.

It contains twelve like 'blades', with:

  • 2x Intel(R) Xeon(R) E5-2680 v3 (48) @ 3.30 GHz, 12 cores 24 threads each
  • 256 GiB DDR4
  • a fully-featured remote management interface, with a webUI kvm
  • two gigabit eth NICs, and one Intel X520 2x10G SFP+
  • notably, no peripheral ports other than a multiplexed one that uses a proprietary adapter to VGA, Serial, 2x USB

These blades are appropriately named: 

[
  "backbiter"
  "damocles"
  "durendal"
  "eyelander"
  "excalibur"
  "gram"
  "gryffindor"
  "kusanagi"
  "narsil"
  "oathbringer"
  "riptide"
  "sting"
]

They are at the moment netbooted from Infrastructure:Vesuvius as NixOS k3s nodes.

Hostname Frontend IP4 Backend IP4 Global IP6 Notes
vesuvius 10.98.0.23 10.98.3.1 2607:b400:6:cc80:21e:67ff:feca:9c12 k3s control plane; PXE boot image server
backbiter 10.98.0.25 10.98.3.2 2607:b400:6:cc80:42f2:e9ff:fec6:655f
damocles 10.98.0.26 10.98.3.3 2607:b400:6:cc80:42f2:e9ff:fec6:67db
durendal 10.98.0.27 10.98.3.4 2607:b400:6:cc80:42f2:e9ff:fec6:6895
excalibur 10.98.0.29 10.98.3.5 2607:b400:6:cc80:42f2:e9ff:fec6:6967
eyelander 10.98.0.28 10.98.3.6 2607:b400:6:cc80:42f2:e9ff:fec6:6943
gram 10.98.0.30 10.98.3.7 2607:b400:6:cc80:42f2:e9ff:fec6:7459
prospit - - - User:Enzoisotton took this out of the server room to test pxe booting, yell at him until it's back
gryffindor 10.98.0.31 10.98.3.8 2607:b400:6:cc80:42f2:e9ff:fec6:74d7
kusanagi 10.98.0.32 - 2607:b400:6:cc80:42f2:e9ff:fec6:7597 the SFP+ NIC appears broken. may work if re-seated.
narsil 10.98.0.33 10.98.3.10 2607:b400:6:cc80:42f2:e9ff:fec6:759d
oathbringer 10.98.0.34 10.98.3.11 2607:b400:6:cc80:42f2:e9ff:fec6:75f1
riptide 10.98.0.35 10.98.3.13 2607:b400:6:cc80:42f2:e9ff:fec6:7621
sting 10.98.0.36 10.98.3.14 2607:b400:6:cc80:42f2:e9ff:fec6:7723

Each machine can be remotely accessed on port 2222 from it's global IP6 address, or by DNS at <name>.vtluug.org.

Each machine has a management interface on 10.98.2.0/24, with the same last octet as its backend IP -- i.e. riptide can be managed from 10.98.2.13.

Guide

Technical

Each listed blade has four connections: two 8p8c NICs attached to Infrastructure:Magma, one for the frontend network, and one for its management interface, then two SFP+ NICs bonded together and connected to Infrastructure:Pahoehoe via QSFP+ breakout cable. The 20G bonded interface is used for the cluster's backend interconnect.

The frontend networks, uses DHCP with Infrastructure:Shellshock, while the backend network is statically routed.

The blades have completely ephemeral disks -- the k3s node passwords and join tokens are shipped via agenix, targeting an ssh host key that is sealed by TPM1.2 and kept in the netboot image.

You can spy their config here [1]

Operational

All users with a VTLUUG SSO account (read: FreeIPA/acidburn) can SSH to Infrastructure:Vesuvius from the VTLUUG LAN or the public IPv6 address.

From there, all users with sudo access can use `kubectl` to manage the cluster, or proxy/jump to other nodes through the backend fabric.

Non-admin tenant namespaces coming Soon^{tm}

Things To Host On All This (Post Suggestions!!!)

  • hentai at home (rsk)
  • tor relay (rsk)
    • both of the above possibly better as a slim nixos VM on something like spectre?
  • nix build server (clj)
  • build/parameter optimization rust compiler (clj)
  • folding@home (ap, eri)
  • port sczi over (mkl)
    • slskd/slskd:latest
    • quay.io/oauth2-proxy/oauth2-proxy:latest
    • luuul/4get:latest ported
    • linuxserver/qbittorrent:latest
    • qmcgaw/gluetun
    • quay.io/oauth2-proxy/oauth2-proxy:latest
    • mariadb:10.4.32
    • vtluug/mediawiki-vtluug:1.1 ported
    • linuxserver/swag
    • vectorim/element-web:latest
    • quay.io/dexidp/dex:latest
    • vtluug/pyqdb ported
    • thelounge/thelounge:latest
    • vtluug-site-vtluug-site ported
    • vaultwarden/server:latest
    • gitea/gitea:nightly
    • t4skforce/syncthing-relay-discovery:latest
  • Tigerbeetle (clj, meme suggestion)
  • Something like Sagemath but in a way that doesn't give people free ACE (clj)
  • map stuff (mhazinsk)
    • OSM tile server
    • async LIDAR processing - laz2ept/sinkholes, etc
  • coder ephemeral dev environments for members (mhazinsk)
  • OCCRP Aleph for VT FOIA requests as a hokieprivacy project (mhazinsk)
  • OONI probe (mhazinsk)
  • browsertrix (pew)
  • grafana (ARC and many others have done this and it looks cool) (clj)
Retrieved from "https://vtluug.org/w/index.php?title=Infrastructure:Bastille&oldid=8326"