- scale-infra - lists.linuxfests.org

Recovered: [Synthetics] Check on www.socallinuxexpo.org redirects to current SCALE year
by Datadog Alerting 02 Jul '26

02 Jul '26

[Recovered] [Synthetics] Check on www.socallinuxexpo.org redirects to current SCALE year @bala@socallinuxexpo.org @ilan@linuxfests.org @phil@ipom.com @slack-SCALE-infra @slack-infra @scale-infra@lists.linuxfests.org This alert was raised by account LinuxFests. View Alert Details: https://app.datadoghq.com/synthetics/details/mmg-4xw-gd4?link_source=monito… View Test Run in Datadog: https://app.datadoghq.com/synthetics/details/mmg-4xw-gd4?resultId=422559116… To manage your Datadog subscription, click here: https://app.datadoghq.com/account/preferences ©2026 Datadog Inc, 620 8th Ave, NYC

1 0

Triggered: [Synthetics] Check on www.socallinuxexpo.org redirects to current SCALE year
by Datadog Alerting 02 Jul '26

02 Jul '26

[Triggered] [Synthetics] Check on www.socallinuxexpo.org redirects to current SCALE year @bala@socallinuxexpo.org @ilan@linuxfests.org @phil@ipom.com @slack-SCALE-infra @slack-infra @scale-infra@lists.linuxfests.org Your test failed at least once from locations Ohio (AWS) and Oregon (AWS) over the last 10 minutes. Last failed test run Jul 02, 2026, 04:47PM UTC | Oregon (AWS) | Duration: 20s 1 failing assertion: Assertion response Time less Than 2000 ms Actual Value 20098 ms Tags: env:prod, team:scale, site:socallinuxexpo.org, probe_dc:aws:us-east-2, probe_dc:aws:us-west-2, check_type:api, check_status:live, ci_execution_rule:blocking This alert was raised by account LinuxFests. View Alert Details: https://app.datadoghq.com/synthetics/details/mmg-4xw-gd4?link_source=monito… View Test Run in Datadog: https://app.datadoghq.com/synthetics/details/mmg-4xw-gd4?resultId=276893206… To manage your Datadog subscription, click here: https://app.datadoghq.com/account/preferences ©2026 Datadog Inc, 620 8th Ave, NYC

1 0

Recovered: [Synthetics] Check on www.socallinuxexpo.org redirects to current SCALE year
by Datadog Alerting 19 May '26

19 May '26

[Recovered] [Synthetics] Check on www.socallinuxexpo.org redirects to current SCALE year @bala@socallinuxexpo.org @ilan@linuxfests.org @phil@ipom.com @slack-SCALE-infra @slack-infra @scale-infra@lists.linuxfests.org This alert was raised by account LinuxFests. View Test Run in Datadog: https://app.datadoghq.com/synthetics/details/mmg-4xw-gd4?resultId=211562810… View Test: https://app.datadoghq.com/synthetics/details/mmg-4xw-gd4 To manage your Datadog subscription, click here: https://app.datadoghq.com/account/preferences ©2026 Datadog Inc, 620 8th Ave, NYC

1 0

Triggered: [Synthetics] Check on www.socallinuxexpo.org redirects to current SCALE year
by Datadog Alerting 19 May '26

19 May '26

[Triggered] [Synthetics] Check on www.socallinuxexpo.org redirects to current SCALE year @bala@socallinuxexpo.org @ilan@linuxfests.org @phil@ipom.com @slack-SCALE-infra @slack-infra @scale-infra@lists.linuxfests.org Your test failed at least once from locations Ohio (AWS) and Oregon (AWS) over the last 10 minutes. Last failed test run May 19, 2026, 06:57PM UTC | Oregon (AWS) | Duration: 30s 3 failing assertions: Assertion header content-type is text/html; charset=iso-8859-1 Actual Value Property doesn't exist Assertion response Time less Than 2000 ms Actual Value 30003 ms Assertion status Code is 302 Actual Value Tags: env:prod, team:scale, site:socallinuxexpo.org, probe_dc:aws:us-east-2, probe_dc:aws:us-west-2, check_type:api, check_status:live, ci_execution_rule:blocking This alert was raised by account LinuxFests. View Test Run in Datadog: https://app.datadoghq.com/synthetics/details/mmg-4xw-gd4?resultId=499528635… View Test: https://app.datadoghq.com/synthetics/details/mmg-4xw-gd4 To manage your Datadog subscription, click here: https://app.datadoghq.com/account/preferences ©2026 Datadog Inc, 620 8th Ave, NYC

1 0

Fwd: Cron <mailman@scale-lists-centos9> /usr/local/bin/backup-mailman.sh
by Ilan Rabinovitch 17 Mar '26

17 Mar '26

This job looks to still be archiving /var/lib/mailman instead of /var/lib/mailman3/ Not sure why it only started failing recently, or if its still needed after the mailman3 upgrade. ---------- Forwarded message ---------- From: "(Cron Daemon)" <mailman(a)lists.linuxfests.org> To: mailman(a)lists.linuxfests.org Cc: Bcc: Date: Mon, 16 Mar 2026 04:00:02 +0000 (UTC) Subject: Cron <mailman@scale-lists-centos9> /usr/local/bin/backup-mailman.sh Starting backup.... tar: /var/lib/mailman/: Cannot open: No such file or directory tar: Error is not recoverable: exiting now tar: /var/lib/mailman/: Cannot open: No such file or directory tar: Error is not recoverable: exiting now Uploading backup.... /usr/local/bin/backup-mailman.sh: line 19: aws: command not found /usr/local/bin/backup-mailman.sh: line 19: aws: command not found rmdir: failed to remove '/tmp/tmp.mZNllZXzcU': Directory not empty

3 5

Details of changes
by Phil Dibowitz 12 Feb '26

12 Feb '26

Just a record of changes for those of you who care about the infra: * Updated postfix on lists to strip DKIM/ARC headers for incoming messages. These get re-signed on the way out, and even when we have DKIM aligment to the new sig, some things don't like the old sig not verifying. * Updated mailman to always do DMARC From mitigation This will ensure we have DKIM alignment This also updated several lists that were doing Wrap based mitigation to do From mitigation * Updated SPF records To remove unneeded rackspace IPs * Updated our DMARC records Added some missing fields, removed rua/ruf addresses we don't need, added rua/ruf address for mailgun. * Updated DNS for lists.linuxfests.org Added an MX, rather than falling back to A record There's more to come, we need to move to enforcing our SPF/DKIM/DMARC policies, but I want to wait and make sure we're actually getting DMARC alignment first. -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

test
by Phil Dibowitz 13 Jan '26

13 Jan '26

test ignore -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

READ: deprecated systems to be deleted!
by Phil Dibowitz 30 Apr '25

30 Apr '25

Folks, ** scale-web-old and scale-lists-old deletion ** I plan to DESTROY scale-web-old (not used since November) and scale-lists-old (not used since February) next week on 4/30. If you think you might need something from those VMs, please power them up and remove the data you need, and power them back off. If you need help, let me know. If you need more time, no problem, let me know. But I figure they've been powered off for months, so this should be a pretty reasonable time frame. NOTE: Personal home directories were *NOT* migrated, only service data. ** scale-reg-old deletion ** I plan to DESTROY scale-reg-old in _two_ weeks, on 5/7. If you have data on this host you would like to keep, please power up the VM and remove it, and power it back off. If you need help, let me know. If you need more time, no problem, let me know. Once this is deleted we can merge the massive cleanup PR that removes all support for mailman2, centos6-8, etc. and that will be very helpful. ** testing-drupal-db-cluster RDS deletion ** This RDS database we believe is not in use. We've shut it down today and I plan to delete it next week on 4/30. -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

3 4

Infra Hack Report
by Phil Dibowitz 23 Apr '25

23 Apr '25

Hey all, Davide and I did another hackathon today, and got a lot done. First and foremost is that the Reg server is finally migrated. It's now a CentOS Stream 10 system (though with a CentOS7 container for reg itself since it still requires python2). That completes our systems migrations to modern OSes. Everything is on CentOS Stream 10 except lists which is CentOS Stream 9 (mailman3 is not yet available for C10). In addition we got a TON of other stuff done: * Upgraded chef-client to the latest version * Synced upstream cookbooks: fb_helpers, fb_dracut, fb_sawp, fb_apache, fb_sysctl, fb_syslog, fb_cron, fb_dnf, fb_yum_repos, fb_systemd, fb_securetty, fb_nsswitch, fb_fstab, fb_modprobe, fb_hostconf, fb_limits, fb_hosts, fb_logrotate, fb_motd, fb_iptables, fb_init, fb_postfix, fb_nginx * Add timesyncing with chrony (controlled by the fb_chrony cookbook) * Migrated to fb_hosts to control /etc/hosts (we had it as a required dependency, bit it was not enabled) * Migrated to fb_modprobe to control boot modules * Migrated to fb_logrotate to control log rotation * Migrated to fb_syslog to control rsyslog * Migrated to fb_securetty to control /etc/securetty * Migrated to fb_limits to control /etc/limits* * Migrated to fb_hostconf to control hostconf configs * Cleaned up the awscli2 hacks now that awscli2 is now properly packaged in CentOS10 * Moved mailman's RDS instance to have min CPU of 0.5 (instead of 1) so it can fully spin down when not in use and save us some money * Did a minor upgrade on mailman's RDS version * Close some old issues, filed some new ones for tracking I'll send a separate email about cleanup schedules. -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

Fwd: Cron <root@scale-lists-centos9> /usr/local/sbin/renew_certs.sh
by Phil Dibowitz 12 Apr '25

12 Apr '25

I fixed this. Something weird happened in EPEL and this got stuck on some set of incompatible packages. - Phil

1 0

2026

2025

2024

2023

2022

scale-infra