- scale-infra - lists.linuxfests.org

[P1] Recovered: [Synthetics] CFP Login
by Datadog Alerting 13 Dec '23

13 Dec '23

[P1] [Recovered] [Synthetics] CFP Login @slack-infra @slack-SCALE-infra @bala@socallinuxexpo.org @ilan@linuxfests.org @kcb@socallinuxexpo.org @phil@ipom.com @justinleegarrison@gmail.com @scale-infra@lists.linuxfests.org The monitor was marked as Recovered by Ilan Rabinovitch. This alert was raised by account LinuxFests. To manage your Datadog subscription, click here: https://app.datadoghq.com/account/preferences ©2023 Datadog Inc, 620 8th Ave, NYC

1 0

[P1] Re-Triggered: [Synthetics] CFP Login
by Datadog Alerting 11 Dec '23

11 Dec '23

[P1] [Re-Triggered] [Synthetics] CFP Login @slack-infra @slack-SCALE-infra @bala@socallinuxexpo.org @ilan@linuxfests.org @kcb@socallinuxexpo.org @phil@ipom.com @justinleegarrison@gmail.com @scale-infra@lists.linuxfests.org Your test failed at least once from locations N. California (AWS) and Oregon (AWS) over the last 0 seconds. Last failed test run Dec 12, 2023, 01:52AM UTC | N. California (AWS) | Laptop Large | Chrome 1 failing step: Step 3 | Click on link "CFP Tool Login" Error Message Error: Cannot locate element. This alert was raised by account LinuxFests. View Test Run in Datadog: https://app.datadoghq.com/synthetics/details/cew-9vp-wfp?resultId=226523716… View Test: https://app.datadoghq.com/synthetics/details/cew-9vp-wfp To manage your Datadog subscription, click here: https://app.datadoghq.com/account/preferences ©2023 Datadog Inc, 620 8th Ave, NYC

1 0

[P1] Triggered: [Synthetics] CFP Login
by Datadog Alerting 11 Dec '23

11 Dec '23

[P1] [Triggered] [Synthetics] CFP Login @slack-infra @slack-SCALE-infra @bala@socallinuxexpo.org @ilan@linuxfests.org @kcb@socallinuxexpo.org @phil@ipom.com @justinleegarrison@gmail.com @scale-infra@lists.linuxfests.org Your test failed at least once from locations N. California (AWS) and Oregon (AWS) over the last 0 seconds. Last failed test run Dec 12, 2023, 01:42AM UTC | N. California (AWS) | Laptop Large | Chrome 1 failing step: Step 3 | Click on link "CFP Tool Login" Error Message Error: Cannot locate element. This alert was raised by account LinuxFests. View Test Run in Datadog: https://app.datadoghq.com/synthetics/details/cew-9vp-wfp?resultId=254245456… View Test: https://app.datadoghq.com/synthetics/details/cew-9vp-wfp To manage your Datadog subscription, click here: https://app.datadoghq.com/account/preferences ©2023 Datadog Inc, 620 8th Ave, NYC

1 0

Fwd: [Notification] New charge for public IPv4 addresses starting February 1, 2024 [AWS Account: 355993445259]
by Ilan Rabinovitch 20 Nov '23

20 Nov '23

FYI ---------- Forwarded message --------- From: 'Amazon Web Services, Inc.' via aws-linuxfests <aws(a)linuxfests.org> Date: Sun, Nov 19, 2023 at 1:42 AM Subject: [Notification] New charge for public IPv4 addresses starting February 1, 2024 [AWS Account: 355993445259] To: <aws(a)linuxfests.org> Hello, You are receiving this notification because your AWS account has at least one public IPv4 address used in the last 30 days. On July 28, 2023, we announced [1] a new charge for public IPv4 addresses that will come into effect starting February 1, 2024. For details on your usage, you can look at the AWS Cost and Usage report. To monitor, analyze, and audit your use of public IPv4 addresses, you can look at Public IP Insights [2]. You can also read our blog post, Identify and Optimize Public IPv4 Address Usage on AWS [3], for information on how to optimize your use of public IPv4 addresses. For more details on public IPv4 address pricing, please see the VPC pricing page [4]. For questions, please reach out to AWS support [5]. [1] https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-… [2] https://aws.amazon.com/about-aws/whats-new/2023/07/aws-public-ip-insights-v… [3] https://aws.amazon.com/blogs/networking-and-content-delivery/identify-and-o… [4] https://aws.amazon.com/vpc/pricing/ [5] https://aws.amazon.com/support Sincerely, Amazon Web Services Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210 --- Reference: https://phd.aws.amazon.com/phd/home?region=us-east-1#/event-log?eventID=arn…

1 0

Two time-sensitive-ish PRs
by Phil Dibowitz 19 Nov '23

19 Nov '23

Infra folks... Process mail queues in the background should get merged before our next mass-mail: https://github.com/socallinuxexpo/scale-chef/pull/303 And drop apache restarts to 2x/daily instead of every-other-hour: https://github.com/socallinuxexpo/scale-chef/pull/304 Thanks, - Phil

1 0

incident response
by Ilan Rabinovitch 06 Oct '23

06 Oct '23

Hi folks, We're increasingly seeing outages on the website this year. So far we've gotten by on KC or I noticing and rebooting things. While we obviously need to stop the outages from happening at all, we probably should still have some plan in place for how we respond to them - especially during critical times of the year (eg today for reg, sponsorship, and CFP). I know this isn't anyone's "job", but given how critical the infra is to the conference, should we setup an oncall rotation for it? We seem to be getting caught at various times where nobody with access is available to respond. Thoughts? Ilan

1 1

Bootstrap issues
by Ilan Rabinovitch 15 Sep '23

15 Sep '23

Hello, I'm trying to spin up a centos7 test node for the web folks so we can test out some new HA configs for drupal as well as troubleshoot the ongoing daily outages. Unfortunately I'm running into a few issues following the instructions: 1) the bootstrap script fails on cert validation issues from cinc.sh. not a big deal, i can tell it to skip validation for now. 2) the next step in the instructions say to run chefctll -i, but chefctl never appears on the box. Any suggestions? Can keep troubleshooting, but wondering if we maybe broke centos7 provisioning when we did the upgrades for centos8 for the list servers? Thanks, Ilan

1 2

test
by Phil Dibowitz 22 Aug '23

22 Aug '23

This is a test. -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

Mailman CentOS upgrade prep
by Phil Dibowitz 27 Mar '23

27 Mar '23

[mostly copied from slack] Tonight Davide and I built an upgraded Mailmain box. `scale-lists2` is on *CentOS 8 Stream* and (we think) ready to go. We didn't do CentOS 9 Steam because mailman isn't (yet) packaged in EPEL9. Moved over all the configs and archives (will re-sync when we do the final cutover). Mailman seems happy - mailing lists, admin interface, etc. - it's on a newer version, 2.1.29. I haven't the slightest clue how to access/test/use phplist, so I haven't tested it, I'd appreciate if anyone (especially Ilan) could poke at it. Given we're pulling the same build, it's very likely it needs some s/python/python2/g somewhere, at the very least. The whole email config here is a bit odd, but it looks like the MX points to google, and then (I presume) google forwards lists stuff on (I don't believe I have access to that), presumably based on the name. So I believe the cut over would be something like: * shutdown postfix on lists1 * re-sync mailman archives+configs * move elastic IP from lists1 to lists2 * update DNS (aws dns?) to update the v6 address for lists.{linuxfests,socallinuxexpo}.org to lists2's v6 address (v4 doesn't need to change due to elastic IP) * test mail is flowing I think that's it. But Ilan, you've actually played with phplist and mailgun and friends a lot and I haven't, so any things you think I might be missing, please let me know.I'll give people a few days to poke at this before I do the cutover. A few side notes: * The mailman restore script doesn't appear to have ever worked. And if we wanted to fix it we'd need to port it to boto3, and it's not clear where some of the APIs we used went. This may be one of those "just downloading the tarball and extracting it may be sufficient if we ever need to do a restore * Mailman backups need to be ported to boto3 as well, but that looks more straight forward? * We cleaned up a lot of stuff while we were at it, but there's more to do, as always. We're on official mailman packages now, though we're still on the same crufty build of phplist since I couldn't find real packages for that. Whether that works in this world is another story. I'm happy to at least try to help debug if someone points me to ... anything. Many many thanks to Davide tonight for all of his help. - Phil

1 0

scale-chef default branch renamed - all hosts updated
by Phil Dibowitz 24 Feb '23

24 Feb '23

I updated the default branch in scale-chef to be `main` instead of `master`, as is the standard these days. I updated the chefctl config here: https://github.com/socallinuxexpo/scale-chef/pull/227 And updated all 4 hosts that pull from us: 3 SCALE and 1 LinuxFests. You will need to update any local checkouts and forks (when you view the main repo on GH it'll give you commands). -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

2025

2024

2023

2022

scale-infra