14 Feb
2024
14 Feb
'24
6:46 p.m.
Adding -infra (back?)
The docs are a lie! Thanks for fixing that, good catch.
On 2/14/24 09:48, Ilan Rabinovitch wrote:
I dont actually think this is working. The paths listed in the cron job were pointing to non-existent files/directories.
https://github.com/socallinuxexpo/scale-chef/pull/308
On Sun, Nov 19, 2023 at 8:17 PM Phil Dibowitz phil@ipom.com wrote:
https://github.com/socallinuxexpo/scale-chef/pull/303 should do the trick.
On 11/18/23 10:45, Ilan Rabinovitch wrote:
OK. Hopefully we can get to it before Hannah has to send the next set of emails, or maybe you can help her run the commands manually until we get cron jobs set up.
Hannah, also please let us know once you've picked a new mailing list too.
On Tue, Nov 7, 2023 at 10:17 PM Phil Dibowitz phil@ipom.com wrote:
Sorry, new job is really keeping me busy. Plus last weekend and this weekend are the last two weekends of the Metallica 2023 tour and I'm doing both.
I will try to do the cron next week when I get back (I put it on my todo list app), but if you make a PR before then I won't take any offense.
On 11/7/23 06:00, Ilan Rabinovitch wrote:
Phil, Let us know if you need a hand with setting up the cron jobs.
Hannah, Let us know if you still want to switch mailing list tooling.
On Sun, Oct 29, 2023 at 5:19 PM Ilan Rabinovitch <ilan@linuxfests.org mailto:ilan@linuxfests.org> wrote:
Running the queue from the CLI resulted in it completing in <5 minutes. Definitely worth configuring it in cron when Phil has time. On Sun, Oct 29, 2023 at 6:06 PM Ilan Rabinovitch <ilan@linuxfests.org <mailto:ilan@linuxfests.org>> wrote: Hannah, You mentioned a few times wanting to switch tools, have you identified the new tool you prefer to use? PHPList by default expects the browser to remain open until the send is complete. You can however set up a cronjob to automatically process the queue in the background: (https://www.phplist.org/manual/books/phplist-manual/page/methods-of-sending-%28browser-cron-command-line%29#bkmrk-automatic-queue-proc <https://www.phplist.org/manual/books/phplist-manual/page/methods-of-sending-%28browser-cron-command-line%29#bkmrk-automatic-queue-proc>) Phil, Maybe the infra team can set that up to run per the docs above? /usr/bin/php /var/www/html/lists/admin/index.php -pprocessqueue -c/var/www/html/lists/config/config.php There's also a bunch of phplist settings the define how fast mail is sent. You might what to look at these: MAX_PROCESS_MESSAGE MAILQUEUE_BATCH_SIZE MAILQUEUE_BATCH_PERIOD MAILQUEUE_THROTTLE MAILQUEUE_AUTOTHROTTLE I imagine they're set to defaults rather than tuned for our workloads. On Sun, Oct 29, 2023 at 4:23 PM Hannah Anderson <hsanderson707@gmail.com <mailto:hsanderson707@gmail.com>> wrote: The server definitely does not send unless my laptop is on. I sent an email three days ago and it is still only at 30% complete. It sends a bunch quickly then slows to a halt. On Sun, Oct 29, 2023, 7:01 AM Ilan Rabinovitch <ilan@linuxfests.org <mailto:ilan@linuxfests.org>> wrote: If we're not moving off phplist to another tool we might want to try the mailgun plugin for phplist so that we get bounce handling. Not sure if it'll speed things up to use the API without testing. https://resources.phplist.com/plugin/mailgun <https://resources.phplist.com/plugin/mailgun> On Wed, Oct 11, 2023 at 3:03 PM Ilan Rabinovitch <ilan@linuxfests.org <mailto:ilan@linuxfests.org>> wrote: I believe the flow for emails here is phplist -> postfix -> mailgun. We use mailgun here because EC2 ip addresses had deliverability issues for us in the past. Mailgun also handles signing all our emails with DKIM, etc which simplifies things for us. Can easily change it if we find it problematic. We're using serverless RDS for this which scales to 0 anytime somebody hasn't touched phplist in 5-10 minutes. This saves us money but means the first attempt to access it in a while will be slow. It shouldn't impact sending, but we can extend that to an hour or two of idle time if we want and see if that helps. In terms of when the sends were: Mailgun graph suggests emails were sent on 10/3/23 at 6pm ET, 10/4 at 3am ET, and 10/6 at 1pm ET. Screenshot 2023-10-11 at 9.32.30 PM.png I configured the mailgun integration <https://docs.datadoghq.com/integrations/mailgun/> with Datadog so that we can have those stats alongside system stats and postfix stats moving forward. It wont help for past sends just new email moving forward.. PHP List confirms the timeline above. Screenshot 2023-10-11 at 9.45.41 PM.png Looks like postfix stats are failing to collect due to a missing sudo configuration, but I'm not sure it'd have anything useful for this investigation. Not sure which cookbook/recipe you want that in. We probably want it on all hosts, not just mailman/phplist boxes, but defer to you. Oct 11 18:52:18 scale-lists2 sudo[3737505]: dd-agent : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=list Oct 11 18:52:18 scale-lists2 sudo[3737507]: pam_unix(sudo:auth): conversation failed Oct 11 18:52:18 scale-lists2 sudo[3737507]: pam_unix(sudo:auth): auth could not identify password for [dd-agent] Oct 11 18:52:20 scale-lists2 sudo[3737507]: dd-agent : command not allowed ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/find /var/spool/postfix/active -type f Looking at system graphs <https://app.datadoghq.com/metric/explorer?start=1696194000000&end=1696712340000&paused=true#N4Ig7glgJg5gpgFxALlAGwIYE8D2BXJVEADxQEYAaELcqyKBAC1pEbghkcLIF8qo4AMwgA7CAgg4RKUAiwAHOChASAtnADOcAE4RNIKtrgBHPJoQaUAbVBGN8qVoD6gnNtUZCKiOq279VKY6epbINiAiGOrKQdpYZAYgUJ4YThr42gDGSsgg6gi6mZaBZnHKGniqyBpYGghwqgB0mfJ4jTV1DcCMOHXVmRhocAC0aBB1GgBMPCB8oJHRubFYk4nJCKnpeFk5eYiFxSDL5ZXVtfVNLW14ft29CP2DI2MT07MU81G7ywDMaylpDLZZT5A6JY65CpVDoXZqtRqSMAYcR3PoaAZDUbjCxvOYRL4xUpYAAs-w2gO2wNyoIgRXBRJOVQwYA0jTgmUmjREiDAbgA1k58AhUQ90U8sa8ZniFt8iQBWMmbIG7Gl0ko6GiQ07M1nsznchC87QC+QYTJ8xAaQUEEWPTEvHFSgC6VFc7jwmFC4Tdqg9GEJGoSeJ9foDcVWwbcvswYawf0j7pjSyJpIT0f9yY1Cp4LpAdSwQxkIFNQwQ9WUUBwMCcmU9GlpiTG3KcckU5RwYygjdEcCc9CYyhEUcG7yS43kmCwLYUuybShzPD4eYn4gAwlJhDAUCIPWgeEA> there's no spikes in memory, cpu or network around that time. One thing we should figure out outside of perf is bounce handling. PHPlist says none of the emails we send are bouncing, but mailgun shows over 1K bounces per email campaign sent out. I imagine that's not helping speed, but also makes us look like spammers. Screenshot 2023-10-11 at 9.50.24 PM.png Anyways, that's all I can offer before I crash tonight. Good luck with the investigation. -Ilan Ilan Rabinovitch Conference Chair Southern California Linux Expo 877-831-2569 x11 Voice 818-442-1865 Mobile ilan@linuxfests.org <mailto:ilan@linuxfests.org> Email --- Ask about sponsorship and speaking opportunities at LinuxFests.org's upcoming events: * SCALE 21x - March 14-17 2024 * DevOpsDay LA - March 15, 2024 * Texas Linux Fest - April 12-13, 2024 On Wed, Oct 11, 2023 at 5:49 AM Phil Dibowitz <phil@ipom.com <mailto:phil@ipom.com>> wrote: I'll dig into this tomorrow afternoon, thanks for reporting. One thing that would be helpful is to know what time you sent emails, and what time it appeared to be done processing so I could correlate with the events you saw. Thanks! On 10/10/23 10:20, Hannah Anderson wrote: > Hey Phil, > > I have noticed that the mail server has been slow. It took 3 hours to > process the last email I sent and did not make it to 100% of the list. > > I thought this was a fluke, but it has happened for the last 3 emails > now. Any idea what could be wrong? > > Thanks, > Hannah > > -- > Without Wax -- Phil Dibowitz phil@ipom.com <mailto:phil@ipom.com> Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ <http://www.phildev.net/> http://www.ipom.com/ <http://www.ipom.com/> "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
--
Phil Dibowitz phil@ipom.com
Open Source software and tech docs Insanity Palace of Metallica
http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't
matter and those who matter don't mind."
- Dr. Seuss