scale-infra February 2024

scale-infra@lists.linuxfests.org

2 participants
3 discussions

Infra Hackathon @ SCALE
by Phil Dibowitz 26 Feb '24

26 Feb '24

Hey all, I was talking to Davide about doing an Infra hackathon at SCALE. We think we could both knock down a lot of the backlog, but also, others interested in helping who maybe would prefer to do it with some guidance could help as well. Things that would be a priority: * Re-syncing upstream cookbooks * Upgrading Chef/Cinc * Porting things to C9 (and cleaning up old support) * Debugging Web instability I was thinking perhaps a few hours on Thursday and Friday afternoon which are a bit quieter. And maybe Ilan can even find us a room! So several questions: * Are people interested in joining us? * Do people have day/time preferences? * Any other thoughts? -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

Re: [scale-infra] Mail server slow
by Phil Dibowitz 14 Feb '24

14 Feb '24

Adding -infra (back?) The docs are a lie! Thanks for fixing that, good catch. On 2/14/24 09:48, Ilan Rabinovitch wrote: > I dont actually think this is working. The paths listed in the cron > job were pointing to non-existent files/directories. > > https://github.com/socallinuxexpo/scale-chef/pull/308 > > On Sun, Nov 19, 2023 at 8:17 PM Phil Dibowitz <phil(a)ipom.com> wrote: >> >> https://github.com/socallinuxexpo/scale-chef/pull/303 should do the trick. >> >> >> >> On 11/18/23 10:45, Ilan Rabinovitch wrote: >>> OK. Hopefully we can get to it before Hannah has to send the next set >>> of emails, or maybe you can help her run the commands manually until >>> we get cron jobs set up. >>> >>> Hannah, also please let us know once you've picked a new mailing list too. >>> >>> On Tue, Nov 7, 2023 at 10:17 PM Phil Dibowitz <phil(a)ipom.com> wrote: >>>> >>>> Sorry, new job is really keeping me busy. Plus last weekend and this >>>> weekend are the last two weekends of the Metallica 2023 tour and I'm >>>> doing both. >>>> >>>> I will try to do the cron next week when I get back (I put it on my todo >>>> list app), but if you make a PR before then I won't take any offense. >>>> >>>> >>>> >>>> On 11/7/23 06:00, Ilan Rabinovitch wrote: >>>>> Phil, Let us know if you need a hand with setting up the cron jobs. >>>>> >>>>> Hannah, Let us know if you still want to switch mailing list tooling. >>>>> >>>>> On Sun, Oct 29, 2023 at 5:19 PM Ilan Rabinovitch <ilan(a)linuxfests.org >>>>> <mailto:ilan@linuxfests.org>> wrote: >>>>> >>>>> Running the queue from the CLI resulted in it completing in <5 >>>>> minutes. Definitely worth configuring it in cron when Phil has time. >>>>> >>>>> On Sun, Oct 29, 2023 at 6:06 PM Ilan Rabinovitch >>>>> <ilan(a)linuxfests.org <mailto:ilan@linuxfests.org>> wrote: >>>>> >>>>> Hannah, >>>>> >>>>> You mentioned a few times wanting to switch tools, have you >>>>> identified the new tool you prefer to use? >>>>> >>>>> PHPList by default expects the browser to remain open until the >>>>> send is complete. You can however set up a cronjob to >>>>> automatically process the queue in the background: >>>>> (https://www.phplist.org/manual/books/phplist-manual/page/methods-of-sending… <https://www.phplist.org/manual/books/phplist-manual/page/methods-of-sending…>) >>>>> >>>>> >>>>> Phil, Maybe the infra team can set that up to run per the docs >>>>> above? >>>>> >>>>> /usr/bin/php /var/www/html/lists/admin/index.php -pprocessqueue >>>>> -c/var/www/html/lists/config/config.php >>>>> >>>>> There's also a bunch of phplist settings the define how fast >>>>> mail is sent. >>>>> >>>>> You might what to look at these: >>>>> >>>>> MAX_PROCESS_MESSAGE >>>>> MAILQUEUE_BATCH_SIZE >>>>> MAILQUEUE_BATCH_PERIOD >>>>> MAILQUEUE_THROTTLE >>>>> MAILQUEUE_AUTOTHROTTLE >>>>> >>>>> I imagine they're set to defaults rather than tuned for our >>>>> workloads. >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, Oct 29, 2023 at 4:23 PM Hannah Anderson >>>>> <hsanderson707(a)gmail.com <mailto:hsanderson707@gmail.com>> wrote: >>>>> >>>>> The server definitely does not send unless my laptop is on. >>>>> >>>>> I sent an email three days ago and it is still only at 30% >>>>> complete. It sends a bunch quickly then slows to a halt. >>>>> >>>>> On Sun, Oct 29, 2023, 7:01 AM Ilan Rabinovitch >>>>> <ilan(a)linuxfests.org <mailto:ilan@linuxfests.org>> wrote: >>>>> >>>>> If we're not moving off phplist to another tool we >>>>> might want to try the mailgun plugin for phplist so that >>>>> we get bounce handling. Not sure if it'll speed things >>>>> up to use the API without testing. >>>>> >>>>> https://resources.phplist.com/plugin/mailgun >>>>> <https://resources.phplist.com/plugin/mailgun> >>>>> >>>>> >>>>> On Wed, Oct 11, 2023 at 3:03 PM Ilan Rabinovitch >>>>> <ilan(a)linuxfests.org <mailto:ilan@linuxfests.org>> wrote: >>>>> >>>>> I believe the flow for emails here is phplist -> >>>>> postfix -> mailgun. We use mailgun here because EC2 >>>>> ip addresses had deliverability issues for us in the >>>>> past. Mailgun also handles signing all our emails >>>>> with DKIM, etc which simplifies things for us. Can >>>>> easily change it if we find it problematic. >>>>> >>>>> We're using serverless RDS for this which scales to >>>>> 0 anytime somebody hasn't touched phplist in 5-10 >>>>> minutes. >>>>> This saves us money but means the first attempt to >>>>> access it in a while will be slow. It shouldn't >>>>> impact sending, but we can extend that to an hour or >>>>> two of idle time if we want and see if that helps. >>>>> >>>>> In terms of when the sends were: >>>>> >>>>> Mailgun graph suggests emails were sent on 10/3/23 >>>>> at 6pm ET, 10/4 at 3am ET, and 10/6 at 1pm ET. >>>>> >>>>> Screenshot 2023-10-11 at 9.32.30 PM.png >>>>> >>>>> I configured the mailgun integration >>>>> <https://docs.datadoghq.com/integrations/mailgun/> >>>>> with Datadog so that we can have those stats >>>>> alongside system stats and postfix stats moving >>>>> forward. It wont help for past sends just new email >>>>> moving forward.. >>>>> >>>>> PHP List confirms the timeline above. >>>>> >>>>> Screenshot 2023-10-11 at 9.45.41 PM.png >>>>> Looks like postfix stats are failing to collect due >>>>> to a missing sudo configuration, but I'm not sure >>>>> it'd have anything useful for this investigation. >>>>> Not sure which cookbook/recipe you want that in. We >>>>> probably want it on all hosts, not just >>>>> mailman/phplist boxes, but defer to you. >>>>> >>>>> Oct 11 18:52:18 scale-lists2 sudo[3737505]: dd-agent >>>>> : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=list >>>>> Oct 11 18:52:18 scale-lists2 sudo[3737507]: >>>>> pam_unix(sudo:auth): conversation failed >>>>> Oct 11 18:52:18 scale-lists2 sudo[3737507]: >>>>> pam_unix(sudo:auth): auth could not identify >>>>> password for [dd-agent] >>>>> Oct 11 18:52:20 scale-lists2 sudo[3737507]: dd-agent >>>>> : command not allowed ; TTY=unknown ; PWD=/ ; >>>>> USER=root ; COMMAND=/bin/find >>>>> /var/spool/postfix/active -type f >>>>> >>>>> Looking at system graphs >>>>> <https://app.datadoghq.com/metric/explorer?start=1696194000000&end=169671234…> there's no spikes in memory, cpu or network around that time. >>>>> >>>>> One thing we should figure out outside of perf is >>>>> bounce handling. PHPlist says none of the emails we >>>>> send are bouncing, but mailgun shows over 1K bounces >>>>> per email campaign sent out. I imagine that's not >>>>> helping speed, but also makes us look like spammers. >>>>> >>>>> Screenshot 2023-10-11 at 9.50.24 PM.png >>>>> >>>>> >>>>> Anyways, that's all I can offer before I crash tonight. >>>>> >>>>> Good luck with the investigation. >>>>> >>>>> -Ilan >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Ilan Rabinovitch >>>>> Conference Chair >>>>> Southern California Linux Expo >>>>> 877-831-2569 x11 Voice >>>>> 818-442-1865 Mobile >>>>> ilan(a)linuxfests.org <mailto:ilan@linuxfests.org> Email >>>>> --- >>>>> Ask about sponsorship and speaking opportunities at >>>>> LinuxFests.org's upcoming events: >>>>> * SCALE 21x - March 14-17 2024 >>>>> * DevOpsDay LA - March 15, 2024 >>>>> * Texas Linux Fest - April 12-13, 2024 >>>>> >>>>> >>>>> On Wed, Oct 11, 2023 at 5:49 AM Phil Dibowitz >>>>> <phil(a)ipom.com <mailto:phil@ipom.com>> wrote: >>>>> >>>>> I'll dig into this tomorrow afternoon, thanks >>>>> for reporting. >>>>> >>>>> One thing that would be helpful is to know what >>>>> time you sent emails, >>>>> and what time it appeared to be done processing >>>>> so I could correlate >>>>> with the events you saw. >>>>> >>>>> Thanks! >>>>> >>>>> On 10/10/23 10:20, Hannah Anderson wrote: >>>>> > Hey Phil, >>>>> > >>>>> > I have noticed that the mail server has been >>>>> slow. It took 3 hours to >>>>> > process the last email I sent and did not >>>>> make it to 100% of the list. >>>>> > >>>>> > I thought this was a fluke, but it has >>>>> happened for the last 3 emails >>>>> > now. Any idea what could be wrong? >>>>> > >>>>> > Thanks, >>>>> > Hannah >>>>> > >>>>> > -- >>>>> > Without Wax >>>>> >>>>> -- >>>>> Phil Dibowitz phil(a)ipom.com <mailto:phil@ipom.com> >>>>> Open Source software and tech docs >>>>> Insanity Palace of Metallica >>>>> http://www.phildev.net/ >>>>> <http://www.phildev.net/> http://www.ipom.com/ >>>>> <http://www.ipom.com/> >>>>> >>>>> "Be who you are and say what you feel, because >>>>> those who mind don't >>>>> matter and those who matter don't mind." >>>>> - Dr. Seuss >>>>> >>>>> >>>> >>>> -- >>>> Phil Dibowitz phil(a)ipom.com >>>> Open Source software and tech docs Insanity Palace of Metallica >>>> http://www.phildev.net/ http://www.ipom.com/ >>>> >>>> "Be who you are and say what you feel, because those who mind don't >>>> matter and those who matter don't mind." >>>> - Dr. Seuss >>>> >>>> >> -- Phil Dibowitz phil(a)ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss

1 0

web to Centos8
by Ilan Rabinovitch 12 Feb '24

12 Feb '24

Hi, I took a stab at updating the cookbooks we use for the website to use centos8. Motivation was getting a new version of php so we can use some drupal plugins that dont support php5.4, but i think this was on the backlog already for a while. https://github.com/socallinuxexpo/scale-chef/pull/283 To test this out, I spun up scale-web2 with a clone of the production scale-drupal database, and then pointed my host file at it for www.socallinuxexpo.org: I then tested the following: - chef runs complete end to end - Backing up / restoring static assets (see backup scripts in /usr/local/bin or in the scale-drupal cookbook) - browsing legacy static website (eg https://socallinuxexpo.org/past/2002/ https://socallinuxexpo.org/past/2003/) - Registering as a speaker, confirming that I got the email and that my account worked. - publishing the submitted talk - creating sponsors/exhibitors/blog post/events/etc Pending things: - Centos8 seems to default to using php-fpm instead of mod_php. Not opposed but it's different. Seems to work fine, but I've not load tested it or anything. - drush (https://www.drush.org/) packages no longer seemed to be present in centos or epel. remi seems to have it but the drush docs suggest we should be installing this in our drupal code base via composer instead of as a package. I dont have strong opinions, but we should figure out the right path before merging. Cheers, Ilan

4 43

2025

2024

2023

2022

scale-infra February 2024