Theres definitely something continuing to go on with drupal and/or httpd. Something continues to eat up all the worker threads from httpd. Before we set maxworkers to something reasonable it would eat up all the memory, now it just consumes all the workers and the site becomes unresponsive. I dont think this relates to the upgrade, as it was happening before.
We've had the site down three times in the past 2 days as a result. Monitoring detected it and I was able to respond to it fairly quickly, but I'm about to board a 30 hour flight, so I'm likely not going to be able to respond moving forward.
Given the importance of the CFP this month I hope someone has a chance to dig in.
On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz phil@ipom.com wrote:
Copy that, I'll keep an eye out. Nice work, and thanks!
On 10/3/23 18:06, Ilan Rabinovitch wrote:
This is done. Here's the specific ids for any roll backs:
- start scale-web1 up again (i-017df11978f41d9ae)
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ from
scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae) 3) move EIP (23.21.71.118) from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.
Assuming no issues the next change to think about is:
- enable the s3fs plugin work Phil Smith did recently so static files
aren't stored on box anymore.
- move scale-web2 behind an ELB and auto scaling group.
- remove the cron jobs that back up static files
This will enable some reasonable failover that doesn't require human intervention.
On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday.
On 10/3/23 16:08, Ilan Rabinovitch wrote:
Cutover:
- RDS Database snapshot
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
hosts in case users have uploaded static assets
- Move EIP from scale-web1 host to scale-web2
Rollback:
- re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on
both hosts in case users have uploaded static assets
- Move EIP from scale-web2 host to scale-web1
On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com wrote:
Feel free to do it today.
What's the migrate/failback process? It's somewhere in cloudflair?
If not I can do it next week.
On 10/3/23 11:09, Ilan Rabinovitch wrote:
Sounds like we should do this today then, or you can grab next week. I'm going to be offline until November after Thursday morning.
On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote: > > I'm out of town Fri-Sun, and will only be around a laptop sporadically. > I'm in town all week next week though. > > > > On 10/2/23 13:22, Ilan Rabinovitch wrote: >> Phil Smith, myself and a few others have banged on scale-web2 and we >> think we're all set to do a cutover there. >> However, I'm going to be out of town for a few weeks starting Thursday >> and do not yet know what my connectivity will be like. >> >> Would love to cutover this week, but only want to do it if others will >> be available to roll back to the old server should something go awry >> while I'm on vacation. >> >> On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch >> ilan@socallinuxexpo.org wrote: >>> >>> I believe Ive addressed everyone's feedback. >>> >>> scale-web2 is online on running off the scale_web_centos8 branch. >>> it's using a test db running from a snapshot of prod 2~ weeks ago. >>> you can change it as you see fit without worrying about impacting production. >>> >>> To test you'll need to update /etc/hosts to point at the new server. >>> following to your /etc/hosts file. >>> >>> 54.205.247.164 www.socallinuxexpo.org >>> >>> Ideally would like to merge and migrate product this week >>> >>> >>> On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote: >>>> >>>> Sorry I was in Mexico. >>>> >>>> Left a review to DRY it up. But I don't see any critical things. >>>> >>>> - Phil >>>> >>>> On 9/22/23 15:23, Ilan Rabinovitch wrote: >>>>> Ping? Would like to wrap this up as the non-stop daily outages are no >>>>> fun, Managed to mitigate it somewhat, but given none of us are on call >>>>> getting to some reasonable sense of HA would be appreciated so folks >>>>> can go on vacation without worrying about who will reboot the servers. >>>>> >>>>> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>> >>>>>> Thanks Davide for the initial review. I think I addressed the >>>>>> remaining feedback. >>>>>> >>>>>> Anything else we want to test before merging? >>>>>> >>>>>> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I took a stab at updating the cookbooks we use for the website to use >>>>>>> centos8. Motivation was getting a new version of php so we can use >>>>>>> some drupal plugins that dont support php5.4, but i think this was on >>>>>>> the backlog already for a while. >>>>>>> >>>>>>> https://github.com/socallinuxexpo/scale-chef/pull/283 >>>>>>> >>>>>>> To test this out, I spun up scale-web2 with a clone of the production >>>>>>> scale-drupal database, and then pointed my host file at it for >>>>>>> www.socallinuxexpo.org: >>>>>>> >>>>>>> I then tested the following: >>>>>>> >>>>>>> - chef runs complete end to end >>>>>>> - Backing up / restoring static assets (see backup scripts in >>>>>>> /usr/local/bin or in the scale-drupal cookbook) >>>>>>> - browsing legacy static website (eg >>>>>>> https://socallinuxexpo.org/past/2002/ >>>>>>> https://socallinuxexpo.org/past/2003/) >>>>>>> - Registering as a speaker, confirming that I got the email and that >>>>>>> my account worked. >>>>>>> - publishing the submitted talk >>>>>>> - creating sponsors/exhibitors/blog post/events/etc >>>>>>> >>>>>>> Pending things: >>>>>>> - Centos8 seems to default to using php-fpm instead of mod_php. Not >>>>>>> opposed but it's different. Seems to work fine, but I've not load >>>>>>> tested it or anything. >>>>>>> >>>>>>> - drush (https://www.drush.org/) packages no longer seemed to be >>>>>>> present in centos or epel. remi seems to have it but the drush docs >>>>>>> suggest we should be installing this in our drupal code base via >>>>>>> composer instead of as a package. I dont have strong opinions, but we >>>>>>> should figure out the right path before merging. >>>>>>> >>>>>>> Cheers, >>>>>>> Ilan >>>>> _______________________________________________ >>>>> scale-infra mailing list >>>>> scale-infra@lists.linuxfests.org >>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>>> >>>> -- >>>> Phil Dibowitz phil@ipom.com >>>> Open Source software and tech docs Insanity Palace of Metallica >>>> http://www.phildev.net/ http://www.ipom.com/ >>>> >>>> "Be who you are and say what you feel, because those who mind don't >>>> matter and those who matter don't mind." >>>> - Dr. Seuss >>>> >>>> >>>> _______________________________________________ >>>> scale-infra mailing list >>>> scale-infra@lists.linuxfests.org >>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org >> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra > > -- > Phil Dibowitz phil@ipom.com > Open Source software and tech docs Insanity Palace of Metallica > http://www.phildev.net/ http://www.ipom.com/ > > "Be who you are and say what you feel, because those who mind don't > matter and those who matter don't mind." > - Dr. Seuss > > > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra _______________________________________________ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra