Infrastructure

This page documents VTLUUG's infrastructure post-CVL eviction. It is intended as a scratch pad of sorts for organizing infrastructure changes, since Infrastructure:Log is incredibly out of date, and there are a lot of details which must be sorted out before major infrastructure changes can be made.

Physical Infrastructure

Cyberdelia is the only connected physical machine right now, besides our router. It contains 48 GB of memory with 4 ethernet interfaces; only eth4 is connected and is attached to br0. All hosts are on our own 1-gbit port with temp88191, a gateway featuring ARP proxying, and partial NDP proxying. For global DNS changes under ece.vt.edu, server room access, and getting hosts added to the ARP proxy, contact John Harris or Roddy Dehart.

Virginia Tech has started blocking inbound ports 22 and 3389 and outbound port 25 at the campus border, in a move echarlie is calling Cyber Auschwitz. Contact itso-g@vt.edu for firewall exceptions. When deploying new hosts, make ssh listen on both 2222 and 22. Acidburn currently listens on 443 as well.

Most other VTLUUG-owned servers are currently stored in pew's apartment, with the exception of sunway and a SuperMicro server, which were all transferred to WUVT. The list of these machines includes Wood, and a variety of other servers, including our old LXC hosts and the old mirror.

Some goals for once we get rackspace:

• Acquire new servers
  • Phase out older infrastructure
  • Rebuild most existing servers, to clear out old accounts, old passwords, etc.
    • Assume that machines with installs which predate the current sysadmin are pwned
• Expand Tank on cyberdelia, as it is nearing full
  • Current array is seven 1TB drives in a zraid array, equivalent to RAID 6.
  • Looking at backblaze reliability statistics and prices, we want to upgrade to HGST 7k2000 2TB drives. ($55 each for $385 total)
• Provide an LXC host (preferably on physical hardware)
  • joey is still functional (echarlie recently reinstalled). phantomphreak still needs a new HDD.
• Acquire an IPv6 prefix delegation, and either an IPv6 tunnel or something else
  • Currently have this with HE
• Build a darknet for fail-over from the main network; perhaps run two OpenBSD boxes in CARP.
• Relaunch and expand mirror.cc.vt.edu, ideally with redundancy.

On the Network Architecture

Further Information: Infrastructure:Networking

A simple network diagram by mjh

Because VTLUUG infra no longer has IPv6 access, we are limited by the IP addresses assigned to us, and what we can get through DHCP from ECE. As a consequence, all services which do not need to have a global IP should move to the internal network on the 10.99.x.x/16 subnet. In the future, it may be wise to bridge this network to an ethernet port on cyberdelia, so that a switch can be used to network our other services. When we have rackspace to do so, VPN/Iodine access to the internal network will be necessary.

In theory, services can be provided via a hidden service (read: Tor), but that introduces other complications. Alternately, services can require use of the VPN on acidburn, Iodine, or ssh tunnelling, although all of these limit usage to members with a shell account. All of these have the benefit of NAT transversal, however.

Services

Critical

• Acidburn
  • Shell server
  • Email
  • Wadsworth
• Razor
  • LDAP
• Sczi
  • Website
    • Wiki
    • Main Page
    • snapfeed
    • linx
    • public file shares
  • git
• Cyberdelia
  • All the above services
  • VMs for projects
  • other services
  • Tank
    • NFS
    • milton_srv (website files)

Acidburn and the website must be accessible through real services, and it is preferable that Cyberdelia is too. This means we must use our public IPv4 Addresses allotted by ECE.

Other (not all functional)

• Tahoe-LAFS
  • Should be on hidden service.
• Jitsi Meet
  • Should be public
• User websites
  • Hosted on Milton/Sczi, so will be public
  • Of course, domain owners have to set up their DNS correctly.
• Diaspora*
  • use of public or hidden service doesn't matter, assuming it can be run as a hidden service.
• Iodine
  • On acidburn. Relies on Acidburn being listed as NS for iod.vtluug.org
• IPsec Tunnel
  • On acidburn

Website details

Main article: Infrastructure:Sczi

Current Setup

All vtluug websites are run on nginx on sczi. These sites are currently functional:

• gobblerpedia.org
• hokieprivacy.org
• linx.vtluug.org
• vtluug.org
• wiki.vtluug.org (this redirects to vtluug.org/wiki/)

All of these sites exclusively allow for the use of TLS connections. All services use automatically updated letsencrypt certs.

Main Page

The main VTLUUG page was scraped from the Internet Archive and added created as a github page at vtluug.github.io, and is reverse-proxied through sczi.

Wiki

The wiki currently runs fully on MediaWiki hosted on nginx. Postgres is used for the database and authentication is through LDAP. MediaWiki is running the current stable version as of March 2017, courtesy hard work by pew.

Gobblerpedia

Runs on the same Mediawiki instance as this wiki, with its schema in the same Postgres database. Currently, you can auth with LDAP, pre-existing accounts, or a Google account in the VT hosted domain.

Linx

Uses aam's linx, and is an identical implementation to his site linx.li. One of the few sites with a complete startup script. The maintenance is currently the responsibility of aam, however this is subject to change, pending his retaining root access on sczi.

Historically

Beyond these, we have data and nginx configurations for these (deprecated) sites, which were previously hosted on milton:

Needs Restoration

These are VTLUUG services that haven't yet been properly restored:

• foodfor.vtluug.org - For Wadsworth's .pickfood and .foodvote commands. This is jpo's fault.
• tahoe.vtluug.org - mhazinsk's Tahoe-LAFS grid
• git.vtluug.org - git hosting; previously a redirect to gitweb with gitosis as the SSH backend
• gitweb.vtluug.org - git hosting; previously gitweb
• map.vtluug.org - OSM (gobblerpedia still relies on this)
• vtbash.org - VT QDB (recently a static version was functional)
• users.vtluug.org - member hosting of acidburn home directories

Deprecated

These are no longer needed but we should preserve the data/config:

• ccdc.vtluug.org - website used for the Collegiate Cyber Defense Competition in the spring of 2011.
• nagios.vtluug.org - was used for monitoring for a period but not kept up-to-date; the primary issue was that infrastructure, being hosted on the same server, tended to fail all at once
• uniluug.org - deprecated project uniluug
• security.ece.vt.edu - REDACTED
• wargame.vtluug.org - vtluug's wargame pre-vtcsec
• munin.vtluug.org - former attempt at monitoring
• webchat.vtluug.org - channel webchat; version of the service still runs on the vtluug website as an embeded frame from freenode's website. This should be disabled.
• cdn.vtluug.org - member website
• randynance.info - member website
• jessicaandchristopher.net - member website
• vtcybersecurity.org - member website from pre-csec days

Prior to the current site, which was written in Python using Django starting in 2010 and deployed in 2011, the VTLUUG website was a WordPress instance. The files for this still exist. Much of the original django deployment was highly dynamic, with an identi.ca feed, automatic calendar updates, and other features.

Additional files and configuration data exist for multiple versions of dokuwiki and mediawiki. There are also files for a NewsBlur installation which was never functional, and "bitcoin," the purpose of which is unknown.

A static version of vtbash.org is hosted at https://vtluug.org/bash/index.html.

Until recently, VTLUUG ran a Gopher server on Milton, with a small welcome page and access to /files. Git hosting was also provided; authenticated access was over SSH with gitosis and public access was via Gitweb and the insecure Git protocol.

vtluug.org/public/$user/ was previously a source directory of files placed in a user's $HOME/public directory on NFS.

VTLUUG members also ran mirror.ece.vt.edu, although it was never an official VTLUUG project. The IPv4 allocation for this domain is still available.

PostgreSQL tables exist on milton for foodforus, jandc, mediawiki, mewsblu, sharedwiki, uniluug, vtluug_wiki, and wargame_bbs. More research is necessary to determine what these are for. There are also a collection of mySQL databases on milton which may be desirable to archive and store.

Considerations for the future

It would be preferable to keep all existing sites functional. Aam suggested the use of caddy to replace nginx, especially on static sites, because it provides automatic letsencrypt. Most likely, this will not be used, though, in favour of manually configuring letsencrypt (because caddy is not in the repositories).

It is also important to ensure there are startup scripts for sczi, so that a reboot of it doesn't result in a 10-hour website outage.

TLS Encryption

All sites should move to automatic letsencrypt certificates within the next few months. letsencrypt is still in beta, but it provides fully functional and low-hassle 3-month certificates. They will have integration for nginx soon, which will decrease the complexity of getting certs.

The version of OpenSSL on milton does not support TLSv1.1 or TLSv1.2

It may also be of interest to the club to revive monkeysphere signing on the servers.

Project Hosting

Most vtluug internal projects (e.g. HokiePrivacy) are currently hosted on git repositories on sczi. How to move forwards with these is still indeterminate.

For public hosting, there has been some conversation about using Go Git Service (gogs), for public, www-viewable git repositories.

Gobblerpedia

Gobblerpedia is effectively non-functional because account creation was disabled to prevent spam. This needs to be addressed by adding captchas which are Blacksburg-specific, limiting account creation access to local IP addresses, or enabling some form of LUUG-managed account creation. One suggestion was using CAS/Login services to authenticate, however that makes it difficult for Blacksburg residents to add content, and has other disadvantages.

VTLUUG Main Page

This needs to be kept up-to-date, and have some way of adding calendar information. echarlie would like a simple static site with some clean CSS, to avoid the clusterf*ck that is re-enabling uwsgi at the current time.

VTLUUG Wiki

Meeting creation script needs to be re-implemented.

Addressing

Main article: Infrastructure:Network

VTLUUG currently has DNS with namecheap, however all domains have been re-registered with gandi under vtluug's own account, to replace the mixture of mutantmonkey-holdings and vtluug-held domains on multiple different registrars.

We currently hold the IPv6 prefix 2001:470:8:6d9::/64 from HE's allocation due to our tunnel. When CNS finally sets up prefix-delegation, we might be able to get something from 2607:b400::/32, which is VT's largest allocation.

An important limitation of the HE tunnel is that they block port 25 by default.

https://linx.vtluug.org/ips1.txt

globally addressable	ECE DNS	Name	fucntion	internal address
128.173.88.78	luug.ece.vt.edu.	sczi	New webserver	10.99.0.150 (Currently DHCP)
128.173.89.244	luug0.ece.vt.edu	cyberdelia	VM Host	10.99.0.1
128.173.89.245	luug1.ece.vt.edu.	acidburn	shell server. Also has MX record for vtluug.org and NS for iod.vtluug.org	10.99.0.4
128.173.89.246	luug2.ece.vt.edu.	milton	Old webserver (dead)	10.99.0.2
128.173.89.247	luug3.ece.vt.edu.	razor	LDAP; external networking is fucky	10.99.0.22
128.173.88.104	luug4.ece.vt.edu	pfsense	HE tunnel runs through here (turned off)	10.99.0.104
128.173.88.36	mirror.ece.vt.edu.	mirror	not in use
128.173.88.191	luug5.ece.vt.edu.	router	IPv6 is broken, but it still does RAs
None	172.16.10.13	archiveteam-warrior	Archive Team Warrior	off?

Deprecated IPs

IPs we were using at some point (by reclaiming CVL IPs, etc), or are mentioned in /etc/exports on Cyberdelia

• 128.173.88.161 (security.ece.vt.edu) - was once milton's IP until luug IPs got routed behind router.ece.vt.edu.
• 128.173.88.145 (cvl05.ece.vt.edu) - was once snapfeed's IP, reclaimed by bmckagen.
• 128.173.88.131 (dog.ece.vt.edu) - we have a Sun box labeled "dog.ece.vt.edu" but the IP appears to be in use now for VMware ESXi. It can be arped from cyberdelia.