- scale-infra - lists.linuxfests.org

Infra Hackathon @ SCALE
by Phil Dibowitz 26 Feb '24

26 Feb '24

Hey all, I was talking to Davide about doing an Infra hackathon at SCALE. We think we could both knock down a lot of the backlog, but also, others interested in helping who maybe would prefer to do it with some guidance could help as well. Things that would be a priority: * Re-syncing upstream cookbooks * Upgrading Chef/Cinc * Porting things to C9 (and cleaning up old support) * Debugging Web instability I was thinking perhaps a few hours on Thursday and Friday afternoon which are a bit quieter. And maybe Ilan can even find us a room! So several questions: * Are people interested in joining us? * Do people have day/time preferences? * Any other thoughts? -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

Re: [scale-infra] Mail server slow
by Phil Dibowitz 14 Feb '24

14 Feb '24

Adding -infra (back?) The docs are a lie! Thanks for fixing that, good catch. On 2/14/24 09:48, Ilan Rabinovitch wrote: > I dont actually think this is working. The paths listed in the cron > job were pointing to non-existent files/directories. > > https://github.com/socallinuxexpo/scale-chef/pull/308 > > On Sun, Nov 19, 2023 at 8:17 PM Phil Dibowitz <phil(a)ipom.com> wrote: >> >> https://github.com/socallinuxexpo/scale-chef/pull/303 should do the trick. >> >> >> >> On 11/18/23 10:45, Ilan Rabinovitch wrote: >>> OK. Hopefully we can get to it before Hannah has to send the next set >>> of emails, or maybe you can help her run the commands manually until >>> we get cron jobs set up. >>> >>> Hannah, also please let us know once you've picked a new mailing list too. >>> >>> On Tue, Nov 7, 2023 at 10:17 PM Phil Dibowitz <phil(a)ipom.com> wrote: >>>> >>>> Sorry, new job is really keeping me busy. Plus last weekend and this >>>> weekend are the last two weekends of the Metallica 2023 tour and I'm >>>> doing both. >>>> >>>> I will try to do the cron next week when I get back (I put it on my todo >>>> list app), but if you make a PR before then I won't take any offense. >>>> >>>> >>>> >>>> On 11/7/23 06:00, Ilan Rabinovitch wrote: >>>>> Phil, Let us know if you need a hand with setting up the cron jobs. >>>>> >>>>> Hannah, Let us know if you still want to switch mailing list tooling. >>>>> >>>>> On Sun, Oct 29, 2023 at 5:19 PM Ilan Rabinovitch <ilan(a)linuxfests.org >>>>> <mailto:ilan@linuxfests.org>> wrote: >>>>> >>>>> Running the queue from the CLI resulted in it completing in <5 >>>>> minutes. Definitely worth configuring it in cron when Phil has time. >>>>> >>>>> On Sun, Oct 29, 2023 at 6:06 PM Ilan Rabinovitch >>>>> <ilan(a)linuxfests.org <mailto:ilan@linuxfests.org>> wrote: >>>>> >>>>> Hannah, >>>>> >>>>> You mentioned a few times wanting to switch tools, have you >>>>> identified the new tool you prefer to use? >>>>> >>>>> PHPList by default expects the browser to remain open until the >>>>> send is complete. You can however set up a cronjob to >>>>> automatically process the queue in the background: >>>>> (https://www.phplist.org/manual/books/phplist-manual/page/methods-of-sending… <https://www.phplist.org/manual/books/phplist-manual/page/methods-of-sending…>) >>>>> >>>>> >>>>> Phil, Maybe the infra team can set that up to run per the docs >>>>> above? >>>>> >>>>> /usr/bin/php /var/www/html/lists/admin/index.php -pprocessqueue >>>>> -c/var/www/html/lists/config/config.php >>>>> >>>>> There's also a bunch of phplist settings the define how fast >>>>> mail is sent. >>>>> >>>>> You might what to look at these: >>>>> >>>>> MAX_PROCESS_MESSAGE >>>>> MAILQUEUE_BATCH_SIZE >>>>> MAILQUEUE_BATCH_PERIOD >>>>> MAILQUEUE_THROTTLE >>>>> MAILQUEUE_AUTOTHROTTLE >>>>> >>>>> I imagine they're set to defaults rather than tuned for our >>>>> workloads. >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, Oct 29, 2023 at 4:23 PM Hannah Anderson >>>>> <hsanderson707(a)gmail.com <mailto:hsanderson707@gmail.com>> wrote: >>>>> >>>>> The server definitely does not send unless my laptop is on. >>>>> >>>>> I sent an email three days ago and it is still only at 30% >>>>> complete. It sends a bunch quickly then slows to a halt. >>>>> >>>>> On Sun, Oct 29, 2023, 7:01 AM Ilan Rabinovitch >>>>> <ilan(a)linuxfests.org <mailto:ilan@linuxfests.org>> wrote: >>>>> >>>>> If we're not moving off phplist to another tool we >>>>> might want to try the mailgun plugin for phplist so that >>>>> we get bounce handling. Not sure if it'll speed things >>>>> up to use the API without testing. >>>>> >>>>> https://resources.phplist.com/plugin/mailgun >>>>> <https://resources.phplist.com/plugin/mailgun> >>>>> >>>>> >>>>> On Wed, Oct 11, 2023 at 3:03 PM Ilan Rabinovitch >>>>> <ilan(a)linuxfests.org <mailto:ilan@linuxfests.org>> wrote: >>>>> >>>>> I believe the flow for emails here is phplist -> >>>>> postfix -> mailgun. We use mailgun here because EC2 >>>>> ip addresses had deliverability issues for us in the >>>>> past. Mailgun also handles signing all our emails >>>>> with DKIM, etc which simplifies things for us. Can >>>>> easily change it if we find it problematic. >>>>> >>>>> We're using serverless RDS for this which scales to >>>>> 0 anytime somebody hasn't touched phplist in 5-10 >>>>> minutes. >>>>> This saves us money but means the first attempt to >>>>> access it in a while will be slow. It shouldn't >>>>> impact sending, but we can extend that to an hour or >>>>> two of idle time if we want and see if that helps. >>>>> >>>>> In terms of when the sends were: >>>>> >>>>> Mailgun graph suggests emails were sent on 10/3/23 >>>>> at 6pm ET, 10/4 at 3am ET, and 10/6 at 1pm ET. >>>>> >>>>> Screenshot 2023-10-11 at 9.32.30 PM.png >>>>> >>>>> I configured the mailgun integration >>>>> <https://docs.datadoghq.com/integrations/mailgun/> >>>>> with Datadog so that we can have those stats >>>>> alongside system stats and postfix stats moving >>>>> forward. It wont help for past sends just new email >>>>> moving forward.. >>>>> >>>>> PHP List confirms the timeline above. >>>>> >>>>> Screenshot 2023-10-11 at 9.45.41 PM.png >>>>> Looks like postfix stats are failing to collect due >>>>> to a missing sudo configuration, but I'm not sure >>>>> it'd have anything useful for this investigation. >>>>> Not sure which cookbook/recipe you want that in. We >>>>> probably want it on all hosts, not just >>>>> mailman/phplist boxes, but defer to you. >>>>> >>>>> Oct 11 18:52:18 scale-lists2 sudo[3737505]: dd-agent >>>>> : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=list >>>>> Oct 11 18:52:18 scale-lists2 sudo[3737507]: >>>>> pam_unix(sudo:auth): conversation failed >>>>> Oct 11 18:52:18 scale-lists2 sudo[3737507]: >>>>> pam_unix(sudo:auth): auth could not identify >>>>> password for [dd-agent] >>>>> Oct 11 18:52:20 scale-lists2 sudo[3737507]: dd-agent >>>>> : command not allowed ; TTY=unknown ; PWD=/ ; >>>>> USER=root ; COMMAND=/bin/find >>>>> /var/spool/postfix/active -type f >>>>> >>>>> Looking at system graphs >>>>> <https://app.datadoghq.com/metric/explorer?start=1696194000000&end=169671234…> there's no spikes in memory, cpu or network around that time. >>>>> >>>>> One thing we should figure out outside of perf is >>>>> bounce handling. PHPlist says none of the emails we >>>>> send are bouncing, but mailgun shows over 1K bounces >>>>> per email campaign sent out. I imagine that's not >>>>> helping speed, but also makes us look like spammers. >>>>> >>>>> Screenshot 2023-10-11 at 9.50.24 PM.png >>>>> >>>>> >>>>> Anyways, that's all I can offer before I crash tonight. >>>>> >>>>> Good luck with the investigation. >>>>> >>>>> -Ilan >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Ilan Rabinovitch >>>>> Conference Chair >>>>> Southern California Linux Expo >>>>> 877-831-2569 x11 Voice >>>>> 818-442-1865 Mobile >>>>> ilan(a)linuxfests.org <mailto:ilan@linuxfests.org> Email >>>>> --- >>>>> Ask about sponsorship and speaking opportunities at >>>>> LinuxFests.org's upcoming events: >>>>> * SCALE 21x - March 14-17 2024 >>>>> * DevOpsDay LA - March 15, 2024 >>>>> * Texas Linux Fest - April 12-13, 2024 >>>>> >>>>> >>>>> On Wed, Oct 11, 2023 at 5:49 AM Phil Dibowitz >>>>> <phil(a)ipom.com <mailto:phil@ipom.com>> wrote: >>>>> >>>>> I'll dig into this tomorrow afternoon, thanks >>>>> for reporting. >>>>> >>>>> One thing that would be helpful is to know what >>>>> time you sent emails, >>>>> and what time it appeared to be done processing >>>>> so I could correlate >>>>> with the events you saw. >>>>> >>>>> Thanks! >>>>> >>>>> On 10/10/23 10:20, Hannah Anderson wrote: >>>>> > Hey Phil, >>>>> > >>>>> > I have noticed that the mail server has been >>>>> slow. It took 3 hours to >>>>> > process the last email I sent and did not >>>>> make it to 100% of the list. >>>>> > >>>>> > I thought this was a fluke, but it has >>>>> happened for the last 3 emails >>>>> > now. Any idea what could be wrong? >>>>> > >>>>> > Thanks, >>>>> > Hannah >>>>> > >>>>> > -- >>>>> > Without Wax >>>>> >>>>> -- >>>>> Phil Dibowitz phil(a)ipom.com <mailto:phil@ipom.com> >>>>> Open Source software and tech docs >>>>> Insanity Palace of Metallica >>>>> http://www.phildev.net/ >>>>> <http://www.phildev.net/> http://www.ipom.com/ >>>>> <http://www.ipom.com/> >>>>> >>>>> "Be who you are and say what you feel, because >>>>> those who mind don't >>>>> matter and those who matter don't mind." >>>>> - Dr. Seuss >>>>> >>>>> >>>> >>>> -- >>>> Phil Dibowitz phil(a)ipom.com >>>> Open Source software and tech docs Insanity Palace of Metallica >>>> http://www.phildev.net/ http://www.ipom.com/ >>>> >>>> "Be who you are and say what you feel, because those who mind don't >>>> matter and those who matter don't mind." >>>> - Dr. Seuss >>>> >>>> >> -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

web to Centos8
by Ilan Rabinovitch 12 Feb '24

12 Feb '24

Hi, I took a stab at updating the cookbooks we use for the website to use centos8. Motivation was getting a new version of php so we can use some drupal plugins that dont support php5.4, but i think this was on the backlog already for a while. https://github.com/socallinuxexpo/scale-chef/pull/283 To test this out, I spun up scale-web2 with a clone of the production scale-drupal database, and then pointed my host file at it for www.socallinuxexpo.org: I then tested the following: - chef runs complete end to end - Backing up / restoring static assets (see backup scripts in /usr/local/bin or in the scale-drupal cookbook) - browsing legacy static website (eg https://socallinuxexpo.org/past/2002/ https://socallinuxexpo.org/past/2003/) - Registering as a speaker, confirming that I got the email and that my account worked. - publishing the submitted talk - creating sponsors/exhibitors/blog post/events/etc Pending things: - Centos8 seems to default to using php-fpm instead of mod_php. Not opposed but it's different. Seems to work fine, but I've not load tested it or anything. - drush (https://www.drush.org/) packages no longer seemed to be present in centos or epel. remi seems to have it but the drush docs suggest we should be installing this in our drupal code base via composer instead of as a package. I dont have strong opinions, but we should figure out the right path before merging. Cheers, Ilan

4 43

Fwd: [Action Required] Upgrade Amazon Aurora Serverless v1 by December 31, 2024 [AWS Account: 355993445259] [US-EAST-1]
by Ilan Rabinovitch 28 Dec '23

28 Dec '23

looks like we have to migrate all our RDS instances. ---------- Forwarded message --------- From: 'Amazon Web Services, Inc.' via aws-linuxfests <aws(a)linuxfests.org> Date: Thu, Dec 28, 2023 at 1:42 AM Subject: [Action Required] Upgrade Amazon Aurora Serverless v1 by December 31, 2024 [AWS Account: 355993445259] [US-EAST-1] To: <aws(a)linuxfests.org> Hello, We are reaching out to let you know that as of December 31, 2024, Amazon Aurora will no longer support Serverless version 1 (v1). As per the Aurora Version Policy [1], we are providing 12 months notice to give you time to upgrade your database cluster(s). Aurora supports two versions of Serverless. We are only announcing the end of support for Serverless v1. Aurora Serverless v2 continues to be supported. We recommend that you proactively upgrade your databases running Amazon Aurora Serverless v1 to Amazon Aurora Serverless v2 at your convenience before December 31, 2024. Starting on or after 12:00 PM PST on December 31, 2024, if your Amazon Aurora Serverless v1 cluster has not been upgraded, Amazon Aurora will automatically upgrade your Amazon Aurora Serverless v1 cluster to Amazon Aurora Serverless v2 during your next scheduled cluster maintenance window. If your Aurora Serverless v1 cluster is running an engine version that is not available for Aurora Serverless v2, it will be upgraded to an engine version compatible with Aurora Serverless v2. How to determine which clusters are running Aurora Serverless v1? You can find information about the DB cluster on the AWS RDS console. Aurora Serverless v1 DB clusters show Serverless for Role in the Summary section of the cluster detail page [2]. Also a list of your affected clusters can be found in the 'Affected resources' tab of your AWS Health Dashboard. How to Upgrade to Aurora Serverless v2? You can upgrade your database cluster from Aurora Serverless v1 to Aurora Serverless v2 using the AWS SDK or CLI [3] by following these steps: * Convert your Aurora Serverless v1 database cluster to a provisioned cluster * Upgrade the provisioned cluster to the version that is supported with Aurora Serverless v2 * Add a Aurora Serverless v2 reader to the upgraded provisioned cluster and failover to it. Refer to our Aurora Serverless documentation [3] and also the blog on upgrading from Serverless v1 to Serverless v2 [4]. Please be aware of the following timeline: - From now through to December 31, 2024 - You can initiate upgrades of Amazon Aurora Serverless v1 to Amazon Aurora Serverless v2 at any time. - Starting September 1, 2024, you will no longer be able to create new Aurora Serverless v1 clusters or instances with either the AWS Console or the CLI. - Starting December 31, 2024, Amazon Aurora will upgrade your Aurora Serverless v1 cluster during your next maintenance window to Aurora Serverless v2. If you have any questions or concerns, please reach out to AWS Premium Support [5]. [1] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Version… [2] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverl… [3] https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverl… [4] https://aws.amazon.com/blogs/database/upgrade-from-amazon-aurora-serverless… [5] https://aws.amazon.com/support Sincerely, Amazon Web Services Amazon Web Services, Inc. is a subsidiary of Amazon.com, Inc. Amazon.com is a registered trademark of Amazon.com, Inc. This message was produced and distributed by Amazon Web Services Inc., 410 Terry Ave. North, Seattle, WA 98109-5210 --- Reference: https://health.aws.amazon.com/health/home?region=us-east-1#/event-log?event…

1 0

Recovered: [Synthetics] Check on www.socallinuxexpo.org redirects to current SCALE year
by Datadog Alerting 24 Dec '23

24 Dec '23

[Recovered] [Synthetics] Check on www.socallinuxexpo.org redirects to current SCALE year @bala@socallinuxexpo.org @ilan@linuxfests.org @phil@ipom.com @slack-SCALE-infra @slack-infra @scale-infra@lists.linuxfests.org This alert was raised by account LinuxFests. View Test Run in Datadog: https://app.datadoghq.com/synthetics/details/mmg-4xw-gd4?resultId=235868048… View Test: https://app.datadoghq.com/synthetics/details/mmg-4xw-gd4 To manage your Datadog subscription, click here: https://app.datadoghq.com/account/preferences ©2023 Datadog Inc, 620 8th Ave, NYC

1 0

Triggered: [Synthetics] Check on www.socallinuxexpo.org redirects to current SCALE year
by Datadog Alerting 24 Dec '23

24 Dec '23

[Triggered] [Synthetics] Check on www.socallinuxexpo.org redirects to current SCALE year @bala@socallinuxexpo.org @ilan@linuxfests.org @phil@ipom.com @slack-SCALE-infra @slack-infra @scale-infra@lists.linuxfests.org Your test failed at least once from locations Ohio (AWS) and Oregon (AWS) over the last 10 minutes. Last failed test run Dec 25, 2023, 01:42AM UTC | Ohio (AWS) 1 failing assertion: Assertion response Time - less Than 2000 ms Actual Value 20457 ms This alert was raised by account LinuxFests. View Test Run in Datadog: https://app.datadoghq.com/synthetics/details/mmg-4xw-gd4?resultId=286980735… View Test: https://app.datadoghq.com/synthetics/details/mmg-4xw-gd4 To manage your Datadog subscription, click here: https://app.datadoghq.com/account/preferences ©2023 Datadog Inc, 620 8th Ave, NYC

1 0

Mailman now in EPEL 9!
by Phil Dibowitz 22 Dec '23

22 Dec '23

Thanks to a year of work by Michel, EPEL9 had mailman including the WebUI! It's mm3 so as I recall theres a migration there, but that's ok. I won't have time anytime soon, but we have a path too Cent9 now. And if someone else has time before me, go for it! ⁣- Phil

2 1

[P1] Recovered: [Synthetics] CFP Login
by Datadog Alerting 13 Dec '23

13 Dec '23

[P1] [Recovered] [Synthetics] CFP Login @slack-infra @slack-SCALE-infra @bala@socallinuxexpo.org @ilan@linuxfests.org @kcb@socallinuxexpo.org @phil@ipom.com @justinleegarrison@gmail.com @scale-infra@lists.linuxfests.org The monitor was marked as Recovered by Ilan Rabinovitch. This alert was raised by account LinuxFests. To manage your Datadog subscription, click here: https://app.datadoghq.com/account/preferences ©2023 Datadog Inc, 620 8th Ave, NYC

1 0

[P1] Re-Triggered: [Synthetics] CFP Login
by Datadog Alerting 11 Dec '23

11 Dec '23

[P1] [Re-Triggered] [Synthetics] CFP Login @slack-infra @slack-SCALE-infra @bala@socallinuxexpo.org @ilan@linuxfests.org @kcb@socallinuxexpo.org @phil@ipom.com @justinleegarrison@gmail.com @scale-infra@lists.linuxfests.org Your test failed at least once from locations N. California (AWS) and Oregon (AWS) over the last 0 seconds. Last failed test run Dec 12, 2023, 01:52AM UTC | N. California (AWS) | Laptop Large | Chrome 1 failing step: Step 3 | Click on link "CFP Tool Login" Error Message Error: Cannot locate element. This alert was raised by account LinuxFests. View Test Run in Datadog: https://app.datadoghq.com/synthetics/details/cew-9vp-wfp?resultId=226523716… View Test: https://app.datadoghq.com/synthetics/details/cew-9vp-wfp To manage your Datadog subscription, click here: https://app.datadoghq.com/account/preferences ©2023 Datadog Inc, 620 8th Ave, NYC

1 0

[P1] Triggered: [Synthetics] CFP Login
by Datadog Alerting 11 Dec '23

11 Dec '23

[P1] [Triggered] [Synthetics] CFP Login @slack-infra @slack-SCALE-infra @bala@socallinuxexpo.org @ilan@linuxfests.org @kcb@socallinuxexpo.org @phil@ipom.com @justinleegarrison@gmail.com @scale-infra@lists.linuxfests.org Your test failed at least once from locations N. California (AWS) and Oregon (AWS) over the last 0 seconds. Last failed test run Dec 12, 2023, 01:42AM UTC | N. California (AWS) | Laptop Large | Chrome 1 failing step: Step 3 | Click on link "CFP Tool Login" Error Message Error: Cannot locate element. This alert was raised by account LinuxFests. View Test Run in Datadog: https://app.datadoghq.com/synthetics/details/cew-9vp-wfp?resultId=254245456… View Test: https://app.datadoghq.com/synthetics/details/cew-9vp-wfp To manage your Datadog subscription, click here: https://app.datadoghq.com/account/preferences ©2023 Datadog Inc, 620 8th Ave, NYC

1 0

2025

2024

2023

2022

scale-infra