- scale-infra - lists.linuxfests.org

Fwd: [Notification] New charge for public IPv4 addresses starting February 1, 2024 [AWS Account: 355993445259]
by Ilan Rabinovitch 20 Nov '23

20 Nov '23

FYI ---------- Forwarded message --------- From: 'Amazon Web Services, Inc.' via aws-linuxfests <aws(a)linuxfests.org> Date: Sun, Nov 19, 2023 at 1:42 AM Subject: [Notification] New charge for public IPv4 addresses starting February 1, 2024 [AWS Account: 355993445259] To: <aws(a)linuxfests.org> Hello, You are receiving this notification because your AWS account has at least one public IPv4 address used in the last 30 days. On July 28, 2023, we announced [1] a new charge for public IPv4 addresses that will come into effect starting February 1, 2024. For details on your usage, you can look at the AWS Cost and Usage report. To monitor, analyze, and audit your use of public IPv4 addresses, you can look at Public IP Insights [2]. You can also read our blog post, Identify and Optimize Public IPv4 Address Usage on AWS [3], for information on how to optimize your use of public IPv4 addresses. For more details on public IPv4 address pricing, please see the VPC pricing page [4]. For questions, please reach out to AWS support [5]. [1] https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-… [2] https://aws.amazon.com/about-aws/whats-new/2023/07/aws-public-ip-insights-v… [3] https://aws.amazon.com/blogs/networking-and-content-delivery/identify-and-o… [4] https://aws.amazon.com/vpc/pricing/ [5] https://aws.amazon.com/support Sincerely, Amazon Web Services Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210 --- Reference: https://phd.aws.amazon.com/phd/home?region=us-east-1#/event-log?eventID=arn…

1 0

Two time-sensitive-ish PRs
by Phil Dibowitz 19 Nov '23

19 Nov '23

Infra folks... Process mail queues in the background should get merged before our next mass-mail: https://github.com/socallinuxexpo/scale-chef/pull/303 And drop apache restarts to 2x/daily instead of every-other-hour: https://github.com/socallinuxexpo/scale-chef/pull/304 Thanks, - Phil

1 0

incident response
by Ilan Rabinovitch 06 Oct '23

06 Oct '23

Hi folks, We're increasingly seeing outages on the website this year. So far we've gotten by on KC or I noticing and rebooting things. While we obviously need to stop the outages from happening at all, we probably should still have some plan in place for how we respond to them - especially during critical times of the year (eg today for reg, sponsorship, and CFP). I know this isn't anyone's "job", but given how critical the infra is to the conference, should we setup an oncall rotation for it? We seem to be getting caught at various times where nobody with access is available to respond. Thoughts? Ilan

1 1

Bootstrap issues
by Ilan Rabinovitch 15 Sep '23

15 Sep '23

Hello, I'm trying to spin up a centos7 test node for the web folks so we can test out some new HA configs for drupal as well as troubleshoot the ongoing daily outages. Unfortunately I'm running into a few issues following the instructions: 1) the bootstrap script fails on cert validation issues from cinc.sh. not a big deal, i can tell it to skip validation for now. 2) the next step in the instructions say to run chefctll -i, but chefctl never appears on the box. Any suggestions? Can keep troubleshooting, but wondering if we maybe broke centos7 provisioning when we did the upgrades for centos8 for the list servers? Thanks, Ilan

1 2

test
by Phil Dibowitz 22 Aug '23

22 Aug '23

This is a test. -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

Mailman CentOS upgrade prep
by Phil Dibowitz 27 Mar '23

27 Mar '23

[mostly copied from slack] Tonight Davide and I built an upgraded Mailmain box. `scale-lists2` is on *CentOS 8 Stream* and (we think) ready to go. We didn't do CentOS 9 Steam because mailman isn't (yet) packaged in EPEL9. Moved over all the configs and archives (will re-sync when we do the final cutover). Mailman seems happy - mailing lists, admin interface, etc. - it's on a newer version, 2.1.29. I haven't the slightest clue how to access/test/use phplist, so I haven't tested it, I'd appreciate if anyone (especially Ilan) could poke at it. Given we're pulling the same build, it's very likely it needs some s/python/python2/g somewhere, at the very least. The whole email config here is a bit odd, but it looks like the MX points to google, and then (I presume) google forwards lists stuff on (I don't believe I have access to that), presumably based on the name. So I believe the cut over would be something like: * shutdown postfix on lists1 * re-sync mailman archives+configs * move elastic IP from lists1 to lists2 * update DNS (aws dns?) to update the v6 address for lists.{linuxfests,socallinuxexpo}.org to lists2's v6 address (v4 doesn't need to change due to elastic IP) * test mail is flowing I think that's it. But Ilan, you've actually played with phplist and mailgun and friends a lot and I haven't, so any things you think I might be missing, please let me know.I'll give people a few days to poke at this before I do the cutover. A few side notes: * The mailman restore script doesn't appear to have ever worked. And if we wanted to fix it we'd need to port it to boto3, and it's not clear where some of the APIs we used went. This may be one of those "just downloading the tarball and extracting it may be sufficient if we ever need to do a restore * Mailman backups need to be ported to boto3 as well, but that looks more straight forward? * We cleaned up a lot of stuff while we were at it, but there's more to do, as always. We're on official mailman packages now, though we're still on the same crufty build of phplist since I couldn't find real packages for that. Whether that works in this world is another story. I'm happy to at least try to help debug if someone points me to ... anything. Many many thanks to Davide tonight for all of his help. - Phil

1 0

scale-chef default branch renamed - all hosts updated
by Phil Dibowitz 24 Feb '23

24 Feb '23

I updated the default branch in scale-chef to be `main` instead of `master`, as is the standard these days. I updated the chefctl config here: https://github.com/socallinuxexpo/scale-chef/pull/227 And updated all 4 hosts that pull from us: 3 SCALE and 1 LinuxFests. You will need to update any local checkouts and forks (when you view the main repo on GH it'll give you commands). -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

MySQL/Aurora DB upgrades complete
by Phil Dibowitz 24 Feb '23

24 Feb '23

All DBs are upgrade except for `scale-drupal-may23-snapshot-cluster` which isn't used, afaict. Ilan, can I nuke that? -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 1

Fwd: Amazon Aurora MySQL 1 (with MySQL 5.6 compatibility) will reach end of life on February 28, 2023 [AWS Account: 355993445259] [US-EAST-1]
by Ilan Rabinovitch 24 Feb '23

24 Feb '23

We really dont want the website down during or leading up to SCALE. Let me know if I need to grab this. ---------- Forwarded message --------- From: Amazon Web Services, Inc. <no-reply-aws(a)amazon.com> Date: Fri, Feb 24, 2023 at 4:29 AM Subject: Amazon Aurora MySQL 1 (with MySQL 5.6 compatibility) will reach end of life on February 28, 2023 [AWS Account: 355993445259] [US-EAST-1] To: <ilan(a)linuxfests.org> Hello, You are receiving this message because you have one or more Amazon Aurora MySQL clusters running a version of Aurora MySQL 1 (with MySQL 5.6 compatibility) in the US-EAST-1 Region. Amazon Aurora MySQL 1 (with MySQL 5.6 compatibility) will reach end of life on February 28, 2023. We are providing you with a one week notice so you have sufficient time to upgrade your database cluster(s). You can find additional information needed to plan your upgrade including a detailed timeline with milestones in the 'Preparing for Amazon Aurora MySQL-Compatible Edition version 1 end of life' documentation [1]. Clusters that are running these deprecated engine versions after March 1, 2023 will be upgraded on your behalf within a maintenance window after March 1, 2023. The major version upgrade will start within your maintenance window and typically finish within that timeframe. Depending on the cluster and the database activity at the time, the upgrade process may extend beyond the maintenance window. For more information, please refer to the Amazon RDS maintenance window [2]. The clusters we identified as affected are listed in your 'Affected resources' tab of your AWS Health Dashboard. You may also find clusters affected by this deprecation notice by utilizing the 'Finding clusters affected by this end-of-life process' guide [3]. Should you have any questions or concerns, the AWS Support Team is available on re:Post [4] and via Premium Support [5]. [1] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.MySQL56… [2] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_UpgradeDB… [3] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.MySQL56… [4] https://repost.aws/ [5] https://aws.amazon.com/support Sincerely, Amazon Web Services Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210 --- Reference: https://phd.aws.amazon.com/phd/home?region=us-east-1#/event-log?eventID=arn…

3 5

Fwd: Amazon Aurora MySQL 1 (with MySQL 5.6 compatibility) will reach end of life on February 28, 2023 [AWS Account: 355993445259] [US-EAST-1]
by Ilan Rabinovitch 19 Feb '23

19 Feb '23

---------- Forwarded message --------- From: Amazon Web Services, Inc. <no-reply-aws(a)amazon.com> Date: Wed, Feb 15, 2023, 7:40 AM Subject: Amazon Aurora MySQL 1 (with MySQL 5.6 compatibility) will reach end of life on February 28, 2023 [AWS Account: 355993445259] [US-EAST-1] To: <ilan(a)linuxfests.org> Hello, You are receiving this message because you have one or more Amazon Aurora MySQL clusters running a version of Aurora MySQL 1 (with MySQL 5.6 compatibility) in the US-EAST-1 Region. Amazon Aurora MySQL 1 (with MySQL 5.6 compatibility) will reach end of life on February 28, 2023. We are providing you with a 2 week notice so you have sufficient time to upgrade your database cluster(s). You can find additional information needed to plan your upgrade including a detailed timeline with milestones in the 'Preparing for Amazon Aurora MySQL-Compatible Edition version 1 end of life' documentation [1]. Clusters that are running these deprecated engine versions after March 1, 2023 will be upgraded on your behalf within a maintenance window after March 1, 2023. The major version upgrade will start within your maintenance window and typically finish within that timeframe. Depending on the cluster and the database activity at the time, the upgrade process may extend beyond the maintenance window. For more information, please refer to the Amazon RDS maintenance window [2]. The clusters we identified as affected are listed in your 'Affected resources' tab of your AWS Health Dashboard. You may also find clusters affected by this deprecation notice by utilizing the 'Finding clusters affected by this end-of-life process' guide [3]. Should you have any questions or concerns, the AWS Support Team is available on re:Post [4] and via Premium Support [5]. [1] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.MySQL56… [2] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_UpgradeDB… [3] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.MySQL56… [4] https://repost.aws/ [5] https://aws.amazon.com/support Sincerely, Amazon Web Services Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210 --- Reference: https://phd.aws.amazon.com/phd/home?region=us-east-1#/event-log?eventID=arn…

2 3

2025

2024

2023

2022

scale-infra