Difference between revisions of "Project Bringing Sexy Back"

From the Linux and Unix Users Group at Virginia Teck Wiki
Jump to: navigation, search
imported>Pew
 
(132 intermediate revisions by 3 users not shown)
Line 1: Line 1:
−
This is an effort toward the end of Fall 2017 to get new hardware, determining if we can use old hardware for anything, bring many vtluug services back to life plus add new services, fix major problems relating to the current infrastructure, make managing the current infrastructure easier, and to provide enough documentation so that anybody can have a very good understanding of how everything works without having to ask the officers about every little everything. This goes beyond documentating what we have (aka the wiki), instead provide a sort of guide that provides an overview on how to run everything. Yes, we realize this is a massive task. Timeline TBD.
+
'''Note''': This page does '''not''' reflect current infrastructure. It was for planning purposes. As of 2019 everything has been migrated to new infra, as explained in [[Infrastructure]].
  
−
One major change we are considering is running an oVirt cluster for managing everything, along with offering a VPS service in addition to shell accounts on [[Infrastructure:Acidburn|Acidburn]]. This is not possible with the current hardware.
+
If you're looking for the older project [[User:telnorattti|telnoratti]] proposed, details about it are at [[Summer_2013_New_Machines]].
  
−
Oh, and tshirts.
+
This is an effort toward the end of Fall 2017 and early Spring 2018 to get new hardware, determining if we can use old hardware for anything, bring many vtluug services back to life plus add new services, fix major problems relating to the current infrastructure, make managing the current infrastructure easier, and to provide enough documentation so that anybody can have a very good understanding of how everything works without having to ask the officers about every little thing. This will be summarized at https://vtluug.org/rtfm.txt
 +
 
 +
One major change offering a VPS service in addition to shell accounts on [[Infrastructure:Acidburn|Acidburn]]. This is not possible with the current hardware.
  
 
<!--TODO talk about doing stuff from the project list-->
 
<!--TODO talk about doing stuff from the project list-->
−
 
−
 
−
[[T-shirts]] <!--TODO update with designs-->
 
  
  
 
== Getting new hardware ==
 
== Getting new hardware ==
−
[[Infrastructure:Cyberdelia|Cyberdelia]] is the only connected physical machine right now, besides [[Infrastructure:Temp88191|our router]]. See [[Infrastructure 2017]]. We are planning to request funding from the SBB for new infrastructure as early as possible Spring 2018.
+
[[Infrastructure: Cyberdelia|Cyberdelia]] is the only connected physical machine right now, besides [[Infrastructure:Temp88191|our router]]. See [[Infrastructure 2017]]. We are planning to request funding from the SBB for new infrastructure as early as possible Spring 2018.
  
 
We are still working on exactly what to ask for, but here are some desirable things:
 
We are still working on exactly what to ask for, but here are some desirable things:
−
//TODO link items
+
 
 
{| class='wikitable' id='sortMe'
 
{| class='wikitable' id='sortMe'
 
!Priority
 
!Priority
Line 27: Line 26:
 
|}
 
|}
  
−
Ideally we will get 2.
+
Ideally we will get 2. A much more detailed price analysis will be done for the SBB.
 +
 
 +
\o/ We got $1300 to get 2 R620s!!
  
 
== Hardware Configuration ==
 
== Hardware Configuration ==
−
Hardware we will definitely have:
+
==== Hardware ====
 
* temp88191
 
* temp88191
 
* [[Infrastructure:Cyberdelia|Cyberdelia]]
 
* [[Infrastructure:Cyberdelia|Cyberdelia]]
 
* [[Infrastructure:Wood|Wood]]
 
* [[Infrastructure:Wood|Wood]]
 
* 2x SGI (Phantomphreak/Joey)
 
* 2x SGI (Phantomphreak/Joey)
 +
* 2x R620s (Meltdown/Spectre)
  
−
=== Assuming we do get SBB funding ===
+
==== Configuration ====
−
Additional hardware: 2x R620
+
* Router/DNS Server: joey (sry temp88191 is too old)
−
 
 
−
* Router: temp88191
 
 
* NAS: [[Infrastructure:Cyberdelia|Cyberdelia]] due to its HDD capacity
 
* NAS: [[Infrastructure:Cyberdelia|Cyberdelia]] due to its HDD capacity
−
* oVirt Controller: R620
+
* Libvirt hosts: Meltdown & Spectre
−
* oVirt Workers (via PXE): R620 Controller, the other R620, Phantomphreak, Joey
 
 
* HPC: [[Infrastructure:Wood|Wood]] due to its 4U height that can easily fit gpus
 
* HPC: [[Infrastructure:Wood|Wood]] due to its 4U height that can easily fit gpus
  
−
=== If we don't get SBB funding ===
+
Diagrams in progress
−
We probably need to get some additional RAM & CPUs for Wood in this case in order to do the cluster.
 
  
−
* Router: temp88191
+
== Networking ==
−
* NAS: [[Infrastructure:Cyberdelia|Cyberdelia]]
+
See [[Infrastructure:Network]] for a mostly up to date networking layout.
−
* oVirt Controller: [[Infrastructure:Wood|Wood]]
 
−
* oVirt Workers (via PXE): [[Infrastructure:Wood|Wood]], Phantomphreak, Joey
 
  
 +
Public IPv4s are the same except for a few, the local IPv4 network is different, and IPv6 is completely different because we have a prefix.
  
−
We may also get some additional hardware early Spring 2017. This will replace Phantomphreak/Joey, become the oVirt Controller, or the NAS depending on its specs. TBD
+
==== Public IPv4 addresses ====
 +
{| class='wikitable' id='sortMe'
 +
!Function
 +
!DNS
 +
!IP
 +
|-
 +
|Router/DNS
 +
|joey.vtluug.org
 +
|128.173.88.191
 +
|-
 +
|One Libvirt Node (out of 2)
 +
|meltdown.vtluug.org
 +
|128.173.89.246
 +
|-
 +
|Web/Proxy server
 +
|sczi.vtluug.org
 +
|128.173.88.78
 +
|-
 +
|Shell Server
 +
|acidburn.vtluug.org
 +
|128.173.89.245
 +
|-
 +
|FreeIPA
 +
|chimera.vtluug.org
 +
|128.173.89.247
 +
|-
 +
|Primary NFS server
 +
|dirtycow.vtluug.org
 +
|128.173.89.244
 +
|-
 +
|Matt's server
 +
|mjh.ece.vt.edu
 +
|128.173.88.104
 +
|-
 +
|}
  
−
Diagrams in progress
+
As per [[User:echarlie|echarlie]], "check rdns. if there isn't an entry, and it's in ece's allocated block, we can use it."
 +
 
 +
== Service Importance ==
 +
'''Note''': This section's PoV does '''not''' taking into account the current state of service. Instead, it assumes services are being installed to a new infrastructure.
  
−
== Infrastructure Organization ==
+
Syntax:
−
=== Planned Infrastructure Organization ===
+
* Overall service (hostname)
−
* Use [[w:Debian|Debian]] instead of [[w:Centos|Centos]] due to better compataiblity with [[w:ZFS|ZFS]] and upgradability. Regarding the dead [[EL Migration Project]], I'd like to see some specific examples of this if it's an issue because both [[w:Debian|Debian]] and [[w:Centos|Centos]] are both very popular for servers.
 
−
* Continue using [[Infrastructure:temp88191|temp88191]] as a router. OS TBD. Currently running ubuntu 14.04 (upgrade to 16.04 possible) Also see [[Infrastructure:Networking|our networking setup]]
 
−
* Get a new compute server, and use [[Infrastructure:Cyberdelia|Cyberdelia]] as a NAS (possibly using [[w:FreeNAS|FreeNAS]] or [[w:FreeBSD|FreeBSD]]).
 
−
* Possibly use some old hardware for part of an oVirt or Proxmox cluster.
 
  
 +
Services running on each host are covered in the following sections
  
−
== Service Importance ==
+
=== Essential ===
−
'''Note''': This section assumes we get new hardware that enables us to do an oVirt or Proxmox cluster, which would require downtime and reorganization of the current services in addition to future planned services.
+
==== Bare Metal ====
 +
* Routing (joey)
 +
* DNS (joey)
 +
* Firewall (joey)
 +
* NAS ("new" dell 2900 - dirtycow, dell 2900 - cyberdelia, r310 - shellshock)
 +
* Libvirt hosts (meltdown, spectre)
  
−
=== REQUIRED ===
+
==== VMs ====
−
* [https://github.com/mutantmonkey/foodforus foodforus]
+
* FreeIPA (chimera)
−
* Improved homepage using bootstrap and some webshit.
+
* Web services (sczi)
 +
* Shell (acidburn)
  
−
=== VERY IMPORTANT ===
+
=== Very Important ===
 +
==== Bare Metal ====
 +
* VPS system via Libvirt
 +
** Somewhat automated system where a user is given a VM
 +
** Give each user an ipv6 address
 +
* NAS Backup
  
−
=== IMPORTANT ===
+
==== VMs ====
 +
* elk (theplague)
 +
** Need to research this a bunch
 +
* jitsi.vtluug.org (jitsi)
 +
** Proxy through sczi
 +
* emby.vtluug.org (emby)
 +
** Proxy through sczi
 +
* mirror.vtluug.org (mirror)
 +
* syncthing discovery server (sync)
  
−
* IPv6 support for [http://vtluug.org:70 gopher://vtluug.org]
+
=== Important ===
 +
==== VMs ====
 +
* Tahoe-LAFS (crashoverride)
 +
* telnet BBS
 +
** How tf do these things work
 +
* wargame.vtluug.org (wargame)
 +
* Archive Team Warrior
 +
* map.vtluug.org (map)
  
 
=== Other things to consider ===
 
=== Other things to consider ===
 
* diaspora*
 
* diaspora*
−
* finger (kerberos)
+
* finger (kerberos-based)
 
* talkd
 
* talkd
−
* member web pages in vtluug.org/~username/
 
  
 +
== Migration ==
 +
=== VTLUUG Site ===
 +
* Setup on sczi via the web ansible role
 +
 +
=== VTLUUG & Gobblerpedia wiki ===
 +
* Backup content using dumpBackup.xml script
 +
* Setup on sczi via the web ansible role
 +
* Restore content to new wiki
  
−
== Issues with current services ==
+
=== Wadsworth/phenny ===
−
=== VTLUUG Site ===
+
* Use irc3 module instead
−
* THIS IS SHIT
+
* Setup on acidburn via the small-services ansible role
−
* We're literally proxying a static github page (vtluug.github.io)
 
−
* we killed off all the dynamic pump.io and identi.ca features that used to exist; we should bring these back or at least make the site not shit
 
  
−
=== gopherspace ===
+
=== VTBash ===
−
* server doesn't support IPv6
+
* Setup on sczi via the web ansible role
−
* need to actually use /files on it
 
  
−
=== hokieprivacy ===
+
=== Hokieprivacy ===
−
out of date
+
* Setup on sczi via the web ansible role
  
−
=== linx ===
+
=== Linx ===
−
someone needs to update binary
+
* Setup on sczi via the web ansible role
  
−
=== TOR services ===
+
=== Acidburn ===
−
linx, hokieprivacy, and the main site (hence vtluug.org) can be accessed over tor. advertise these urls or make them more memorable (or both)
+
* Migrate homedirs to dirtycow NAS
 +
* Implement background tmux irc systemd service (or just tell people to use crontabs @reboot (sounds like u got a problem w/ systemd))
  
−
=== vtluug.org/files ===
+
=== User Accounts ===
−
* need easier way for non-root to add stuff here???
+
* Setup on chimera via freeipa-server ansible role
 +
* Import LDAP accounts to freeipa
  
−
=== ldap ===
+
=== /tank ===
−
ldap sucks and needs to die. this is also a cent6 box
+
* Move main stuff to dirtycow
 +
* Clean up a bit
  
−
=== mail ===
+
== Issues with current services ==
−
* train spamassassin against a corpus of spam so it filters better
+
When (re)implementing services listed above do the following changes.
−
* sieve doesn't seem to work at all; fix that
 
  
−
=== Most services (LDAP, others) ===
+
=== VTLUUG Site ===
−
* Need standard place to put script for organization on physical server (aside from a git repo)
+
* We're literally proxying a static github page (vtluug.github.io)
−
** [[Wadsworth]] contains debian SysV init scripts in the ircbot homedir
+
* Probably should be redesigned to be slightly more dynamic using jekyll or flask or something
−
** VTBash contains scripts in [[User:pew|pew]]'s homedir (which echarlie moved to a sane place)
 
−
** router configs are on github, as are old ldap configs and triggered in rc.local
 
  
 
=== VTLUUG wiki ===
 
=== VTLUUG wiki ===
 
* Need meeting script
 
* Need meeting script
−
* Some meetings weren't added to category (fixed??)
+
* Need to clean up a lot of old information
−
* Lots of out of date information---should purge or at least clean up
 
−
* can't delete pages
 
−
* Templates?
 
  
 
=== Gobblerpedia ===
 
=== Gobblerpedia ===
 
* Lots of out of date information
 
* Lots of out of date information
−
* Need templates for buildings---currently using bloated wikipedia scripts that don't reflect our needs
+
* Building templates (populate automatically from database)
−
* automagically add data scraped from webpages to gobblerpedia
+
 
−
* can't delete pages
+
=== Wadsworth/phenny ===
 +
* Things constantly break, but there's nothing we can really do about this
 +
* Migrate to irc3 pythong library for a stable core
 +
* Migrate SysV scripts to systemd unit files
  
 
=== VTBash ===
 
=== VTBash ===
−
* [https://bash.vtluug.org bash.vtluug.org] is working as of August 2017
+
* Old shitty interface converted from python2 to python3
−
* An issue with the code resulted in 2 different databases, ea=ch of which is very large and good
+
** Nothing good seems to already exist, so we probably need to write one from scratch
−
** We need to convert these sqlite dbs
+
* Need to merge 2 old sqlite databases
−
* Needs overhaul
+
** Causes bug when only 1 quote author is entered, this is why they can't be merged
−
** Shittly ported to python 3 from python 2
+
 
−
** word wrap doesn't happen (feature or bug???)
+
=== Hokieprivacy ===
 +
* Mobile formatting is fucked -- someone should look into the hugo templates and fix
 +
* Out of date
 +
 
 +
=== Linx ===
 +
* Offload storage to NAS
 +
* Binaries need updating
 +
* Need to do reddit stress test
 +
 
 +
=== vtluug.org/files ===
 +
* Need easy way fo non-root to add stuff here
 +
 
 +
=== vtluug.org/users/~$USERNAME ===
 +
* should document for users
 +
* maybe support dynamic server-side content (probably not; use js)
 +
 
 +
=== mail ===
 +
* Train spamassassin (use corpus)
 +
* Sieve doesn't seem to work
 +
 
 +
=== gopherspace ===
 +
* who tf uses gopher (fwiw, gopher was 90% of the goal of project bringing sexy back)
 +
* legacy IP only, b/c pygopherd is shite
 +
 
 +
=== TOR services ===
 +
linx, hokieprivacy, and the main site (hence vtluug.org) can be accessed over tor. advertise these urls or make them more memorable (or both)
 +
 
  
 
[[Category:VTLUUG:Projects]]
 
[[Category:VTLUUG:Projects]]
 
[[Category:Infrastructure:Misc]]
 
[[Category:Infrastructure:Misc]]

Latest revision as of 02:34, 10 January 2019

Note: This page does not reflect current infrastructure. It was for planning purposes. As of 2019 everything has been migrated to new infra, as explained in Infrastructure.

If you're looking for the older project telnoratti proposed, details about it are at Summer_2013_New_Machines.

This is an effort toward the end of Fall 2017 and early Spring 2018 to get new hardware, determining if we can use old hardware for anything, bring many vtluug services back to life plus add new services, fix major problems relating to the current infrastructure, make managing the current infrastructure easier, and to provide enough documentation so that anybody can have a very good understanding of how everything works without having to ask the officers about every little thing. This will be summarized at https://vtluug.org/rtfm.txt

One major change offering a VPS service in addition to shell accounts on Acidburn. This is not possible with the current hardware.


Getting new hardware

Cyberdelia is the only connected physical machine right now, besides our router. See Infrastructure 2017. We are planning to request funding from the SBB for new infrastructure as early as possible Spring 2018.

We are still working on exactly what to ask for, but here are some desirable things:

Priority Item Estimated Cost
10 Dell Poweredge R620s $600 ish

Ideally we will get 2. A much more detailed price analysis will be done for the SBB.

\o/ We got $1300 to get 2 R620s!!

Hardware Configuration

Hardware

  • temp88191
  • Cyberdelia
  • Wood
  • 2x SGI (Phantomphreak/Joey)
  • 2x R620s (Meltdown/Spectre)

Configuration

  • Router/DNS Server: joey (sry temp88191 is too old)
  • NAS: Cyberdelia due to its HDD capacity
  • Libvirt hosts: Meltdown & Spectre
  • HPC: Wood due to its 4U height that can easily fit gpus

Diagrams in progress

Networking

See Infrastructure:Network for a mostly up to date networking layout.

Public IPv4s are the same except for a few, the local IPv4 network is different, and IPv6 is completely different because we have a prefix.

Public IPv4 addresses

Function DNS IP
Router/DNS joey.vtluug.org 128.173.88.191
One Libvirt Node (out of 2) meltdown.vtluug.org 128.173.89.246
Web/Proxy server sczi.vtluug.org 128.173.88.78
Shell Server acidburn.vtluug.org 128.173.89.245
FreeIPA chimera.vtluug.org 128.173.89.247
Primary NFS server dirtycow.vtluug.org 128.173.89.244
Matt's server mjh.ece.vt.edu 128.173.88.104

As per echarlie, "check rdns. if there isn't an entry, and it's in ece's allocated block, we can use it."

Service Importance

Note: This section's PoV does not taking into account the current state of service. Instead, it assumes services are being installed to a new infrastructure.

Syntax:

  • Overall service (hostname)

Services running on each host are covered in the following sections

Essential

Bare Metal

  • Routing (joey)
  • DNS (joey)
  • Firewall (joey)
  • NAS ("new" dell 2900 - dirtycow, dell 2900 - cyberdelia, r310 - shellshock)
  • Libvirt hosts (meltdown, spectre)

VMs

  • FreeIPA (chimera)
  • Web services (sczi)
  • Shell (acidburn)

Very Important

Bare Metal

  • VPS system via Libvirt
    • Somewhat automated system where a user is given a VM
    • Give each user an ipv6 address
  • NAS Backup

VMs

  • elk (theplague)
    • Need to research this a bunch
  • jitsi.vtluug.org (jitsi)
    • Proxy through sczi
  • emby.vtluug.org (emby)
    • Proxy through sczi
  • mirror.vtluug.org (mirror)
  • syncthing discovery server (sync)

Important

VMs

  • Tahoe-LAFS (crashoverride)
  • telnet BBS
    • How tf do these things work
  • wargame.vtluug.org (wargame)
  • Archive Team Warrior
  • map.vtluug.org (map)

Other things to consider

  • diaspora*
  • finger (kerberos-based)
  • talkd

Migration

VTLUUG Site

  • Setup on sczi via the web ansible role

VTLUUG & Gobblerpedia wiki

  • Backup content using dumpBackup.xml script
  • Setup on sczi via the web ansible role
  • Restore content to new wiki

Wadsworth/phenny

  • Use irc3 module instead
  • Setup on acidburn via the small-services ansible role

VTBash

  • Setup on sczi via the web ansible role

Hokieprivacy

  • Setup on sczi via the web ansible role

Linx

  • Setup on sczi via the web ansible role

Acidburn

  • Migrate homedirs to dirtycow NAS
  • Implement background tmux irc systemd service (or just tell people to use crontabs @reboot (sounds like u got a problem w/ systemd))

User Accounts

  • Setup on chimera via freeipa-server ansible role
  • Import LDAP accounts to freeipa

/tank

  • Move main stuff to dirtycow
  • Clean up a bit

Issues with current services

When (re)implementing services listed above do the following changes.

VTLUUG Site

  • We're literally proxying a static github page (vtluug.github.io)
  • Probably should be redesigned to be slightly more dynamic using jekyll or flask or something

VTLUUG wiki

  • Need meeting script
  • Need to clean up a lot of old information

Gobblerpedia

  • Lots of out of date information
  • Building templates (populate automatically from database)

Wadsworth/phenny

  • Things constantly break, but there's nothing we can really do about this
  • Migrate to irc3 pythong library for a stable core
  • Migrate SysV scripts to systemd unit files

VTBash

  • Old shitty interface converted from python2 to python3
    • Nothing good seems to already exist, so we probably need to write one from scratch
  • Need to merge 2 old sqlite databases
    • Causes bug when only 1 quote author is entered, this is why they can't be merged

Hokieprivacy

  • Mobile formatting is fucked -- someone should look into the hugo templates and fix
  • Out of date

Linx

  • Offload storage to NAS
  • Binaries need updating
  • Need to do reddit stress test

vtluug.org/files

  • Need easy way fo non-root to add stuff here

vtluug.org/users/~$USERNAME

  • should document for users
  • maybe support dynamic server-side content (probably not; use js)

mail

  • Train spamassassin (use corpus)
  • Sieve doesn't seem to work

gopherspace

  • who tf uses gopher (fwiw, gopher was 90% of the goal of project bringing sexy back)
  • legacy IP only, b/c pygopherd is shite

TOR services

linx, hokieprivacy, and the main site (hence vtluug.org) can be accessed over tor. advertise these urls or make them more memorable (or both)

Retrieved from "https://vtluug.org/w/index.php?title=Project_Bringing_Sexy_Back&oldid=7622"