- scale-infra - lists.linuxfests.org

Fwd: Cron <mailman@scale-lists-centos9> /usr/local/bin/backup-mailman.sh
by Ilan Rabinovitch 17 Mar '26

17 Mar '26

This job looks to still be archiving /var/lib/mailman instead of /var/lib/mailman3/ Not sure why it only started failing recently, or if its still needed after the mailman3 upgrade. ---------- Forwarded message ---------- From: "(Cron Daemon)" <mailman(a)lists.linuxfests.org> To: mailman(a)lists.linuxfests.org Cc: Bcc: Date: Mon, 16 Mar 2026 04:00:02 +0000 (UTC) Subject: Cron <mailman@scale-lists-centos9> /usr/local/bin/backup-mailman.sh Starting backup.... tar: /var/lib/mailman/: Cannot open: No such file or directory tar: Error is not recoverable: exiting now tar: /var/lib/mailman/: Cannot open: No such file or directory tar: Error is not recoverable: exiting now Uploading backup.... /usr/local/bin/backup-mailman.sh: line 19: aws: command not found /usr/local/bin/backup-mailman.sh: line 19: aws: command not found rmdir: failed to remove '/tmp/tmp.mZNllZXzcU': Directory not empty

3 5

Details of changes
by Phil Dibowitz 13 Feb '26

13 Feb '26

Just a record of changes for those of you who care about the infra: * Updated postfix on lists to strip DKIM/ARC headers for incoming messages. These get re-signed on the way out, and even when we have DKIM aligment to the new sig, some things don't like the old sig not verifying. * Updated mailman to always do DMARC From mitigation This will ensure we have DKIM alignment This also updated several lists that were doing Wrap based mitigation to do From mitigation * Updated SPF records To remove unneeded rackspace IPs * Updated our DMARC records Added some missing fields, removed rua/ruf addresses we don't need, added rua/ruf address for mailgun. * Updated DNS for lists.linuxfests.org Added an MX, rather than falling back to A record There's more to come, we need to move to enforcing our SPF/DKIM/DMARC policies, but I want to wait and make sure we're actually getting DMARC alignment first. -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

test
by Phil Dibowitz 13 Jan '26

13 Jan '26

test ignore -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

READ: deprecated systems to be deleted!
by Phil Dibowitz 01 May '25

01 May '25

Folks, ** scale-web-old and scale-lists-old deletion ** I plan to DESTROY scale-web-old (not used since November) and scale-lists-old (not used since February) next week on 4/30. If you think you might need something from those VMs, please power them up and remove the data you need, and power them back off. If you need help, let me know. If you need more time, no problem, let me know. But I figure they've been powered off for months, so this should be a pretty reasonable time frame. NOTE: Personal home directories were *NOT* migrated, only service data. ** scale-reg-old deletion ** I plan to DESTROY scale-reg-old in _two_ weeks, on 5/7. If you have data on this host you would like to keep, please power up the VM and remove it, and power it back off. If you need help, let me know. If you need more time, no problem, let me know. Once this is deleted we can merge the massive cleanup PR that removes all support for mailman2, centos6-8, etc. and that will be very helpful. ** testing-drupal-db-cluster RDS deletion ** This RDS database we believe is not in use. We've shut it down today and I plan to delete it next week on 4/30. -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

3 4

Infra Hack Report
by Phil Dibowitz 23 Apr '25

23 Apr '25

Hey all, Davide and I did another hackathon today, and got a lot done. First and foremost is that the Reg server is finally migrated. It's now a CentOS Stream 10 system (though with a CentOS7 container for reg itself since it still requires python2). That completes our systems migrations to modern OSes. Everything is on CentOS Stream 10 except lists which is CentOS Stream 9 (mailman3 is not yet available for C10). In addition we got a TON of other stuff done: * Upgraded chef-client to the latest version * Synced upstream cookbooks: fb_helpers, fb_dracut, fb_sawp, fb_apache, fb_sysctl, fb_syslog, fb_cron, fb_dnf, fb_yum_repos, fb_systemd, fb_securetty, fb_nsswitch, fb_fstab, fb_modprobe, fb_hostconf, fb_limits, fb_hosts, fb_logrotate, fb_motd, fb_iptables, fb_init, fb_postfix, fb_nginx * Add timesyncing with chrony (controlled by the fb_chrony cookbook) * Migrated to fb_hosts to control /etc/hosts (we had it as a required dependency, bit it was not enabled) * Migrated to fb_modprobe to control boot modules * Migrated to fb_logrotate to control log rotation * Migrated to fb_syslog to control rsyslog * Migrated to fb_securetty to control /etc/securetty * Migrated to fb_limits to control /etc/limits* * Migrated to fb_hostconf to control hostconf configs * Cleaned up the awscli2 hacks now that awscli2 is now properly packaged in CentOS10 * Moved mailman's RDS instance to have min CPU of 0.5 (instead of 1) so it can fully spin down when not in use and save us some money * Did a minor upgrade on mailman's RDS version * Close some old issues, filed some new ones for tracking I'll send a separate email about cleanup schedules. -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

Fwd: Cron <root@scale-lists-centos9> /usr/local/sbin/renew_certs.sh
by Phil Dibowitz 13 Apr '25

13 Apr '25

I fixed this. Something weird happened in EPEL and this got stuck on some set of incompatible packages. - Phil

1 0

Update from your Tech (online services) team!
by Phil Dibowitz 01 Feb '25

01 Feb '25

Hey all, As you probably saw in Slack, today we completed the lists migration. This includes: * Mailman2 (unsupported) => Mailman3 (supported!) * CentOS Stream 8 (unsupported) => CentOS Stream 9 (supported!) * PHPList 3.5.2 (doesn't work on modern PHP) => PHPList 3.6.14 (supports modern PHP!) * PHP 7.2 (unsupported) => PHP 8.2 (security support) This was part of a lot of work on the part of the team over the last 6-8 months, we only just finally pulled the last trigger today. I want to remind everyone: * For non-email interactions (moderation or tweaking settings is really the only thing here) you'll need to register for an account. It'll auto-associate with all your memberships (assuming you use the same email). * If you have any issues, please let me know, I'll happily sort you out, and attend to any moderation in the interim. I will answering any tags in slack or direct within 24 hours for the next week for sure, and in general until the conference. * The permission model is more sane in MM3, there's not some magical password you have to share (woo!), people are just assigned as owners, moderators, or members. So people can easily be added to share the moderation burden, or removed when they no longer should have access with minimal fuss. This brings us generally to a much better place with our webserver on CentOS Stream 10 and our Lists server on CentOS Stream 9 (mailman3 is not yet ready for 10, but it'll be a simple upgrade when it is). The next project will be moving reg. It's a bit trickier as it still requires python2, so we'll be containerizing it and moving it to CentOS Stream 10 after the conference (obviously this is reg's most critical time, so we won't be touching it until post-conf). Then we'll go about cleaning up all the old code throughout our codebase specific to CentOS 6, 7, and 8. So many thanks to Davide for countless Monday's hacking on all of this. And thanks to Hannah for helping with the migration today. -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

Final (probably) test
by Phil Dibowitz 31 Jan '25

31 Jan '25

Please ignore still. -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

test, please ignore
by Phil Dibowitz 31 Jan '25

31 Jan '25

test -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

test 2, please ignore
by Phil Dibowitz 31 Jan '25

31 Jan '25

test 2, please ignore. -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

2026

2025

2024

2023

2022

scale-infra