Restarted. Added cron to restart. Will find time to pair debug with Smith this week.
Get BlueMail for Android
On Oct 7, 2023, 1:50 AM, at 1:50 AM, Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
Looks like we were down again for a few hours this morning.
On Fri, Oct 6, 2023 at 10:03 PM Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
(resending with offlist folks)
Looks like we had another outage this morning. I'm online for the
next
45-60 min if someone wants to chat about it. Otherwise, I would greatly appreciate it if someone on the infra team could keep an eye on monitoring and at least restart things.
On Fri, Oct 6, 2023 at 1:11 AM Phil Dibowitz phil@ipom.com wrote:
Phil Smith is probably the best person since it seems to be a
Drupal
issue, best I can tell (I scoured logs a few months back and found nothing other than what you just described: apache threads just
stay
around, busy, until they eat up all the memory).
I'm not sure he's on this list so I'm CC'ing hom.
Phil - I'm happy to get you root access (send me an SSH pubkey), or anything else you may not already have to help you debug. I spent
quite
a while last SCALE trying to figure this out, Ilan has spent time
on
this, and it seems to be drupal-specific, best I can tell.
On 10/5/23 14:49, Ilan Rabinovitch wrote:
Theres definitely something continuing to go on with drupal
and/or
httpd. Something continues to eat up all the worker threads from httpd. Before we set maxworkers to something reasonable it would
eat
up all the memory, now it just consumes all the workers and the
site
becomes unresponsive. I dont think this relates to the upgrade, as it was happening
before.
We've had the site down three times in the past 2 days as a
result.
Monitoring detected it and I was able to respond to it fairly
quickly,
but I'm about to board a 30 hour flight, so I'm likely not going
to be
able to respond moving forward.
Given the importance of the CFP this month I hope someone has a
chance
to dig in.
On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz phil@ipom.com
wrote:
Copy that, I'll keep an eye out. Nice work, and thanks!
On 10/3/23 18:06, Ilan Rabinovitch wrote:
This is done. Here's the specific ids for any roll backs:
- start scale-web1 up again (i-017df11978f41d9ae)
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/
from
scale-web2 (i-0bf1dd453774f2c3b) to scale-web1
(i-017df11978f41d9ae)
- move EIP (23.21.71.118) from scale-web2
(i-0bf1dd453774f2c3b) to scale-web1.
Assuming no issues the next change to think about is:
- enable the s3fs plugin work Phil Smith did recently so static
files
aren't stored on box anymore.
- move scale-web2 behind an ELB and auto scaling group.
- remove the cron jobs that back up static files
This will enable some reasonable failover that doesn't require
human
intervention.
On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com
wrote:
> > Easy-peasey. Feel free to do it as your leisure today. I have
a few
> errands to run, but will never be more than 30m from home and
will be
> able to roll back today, and will be around all day tomorrow
and Thursday.
> > > On 10/3/23 16:08, Ilan Rabinovitch wrote: >> Cutover: >> - RDS Database snapshot >> - sync
/home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
>> hosts in case users have uploaded static assets >> - Move EIP from scale-web1 host to scale-web2 >> >> Rollback: >> - re-sync
/home/drupal/scale-drupal/httpdocs/sites/default/files/ on
>> both hosts in case users have uploaded static assets >> - Move EIP from scale-web2 host to scale-web1 >> >> >> >> >> >> On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com
wrote:
>>> >>> Feel free to do it today. >>> >>> What's the migrate/failback process? It's somewhere in
cloudflair?
>>> >>> If not I can do it next week. >>> >>> >>> On 10/3/23 11:09, Ilan Rabinovitch wrote: >>>> Sounds like we should do this today then, or you can grab
next week.
>>>> I'm going to be offline until November after Thursday
morning.
>>>> >>>> On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz
phil@ipom.com wrote:
>>>>> >>>>> I'm out of town Fri-Sun, and will only be around a laptop
sporadically.
>>>>> I'm in town all week next week though. >>>>> >>>>> >>>>> >>>>> On 10/2/23 13:22, Ilan Rabinovitch wrote: >>>>>> Phil Smith, myself and a few others have banged on
scale-web2 and we
>>>>>> think we're all set to do a cutover there. >>>>>> However, I'm going to be out of town for a few weeks
starting Thursday
>>>>>> and do not yet know what my connectivity will be like. >>>>>> >>>>>> Would love to cutover this week, but only want to do it
if others will
>>>>>> be available to roll back to the old server should
something go awry
>>>>>> while I'm on vacation. >>>>>> >>>>>> On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch >>>>>> ilan@socallinuxexpo.org wrote: >>>>>>> >>>>>>> I believe Ive addressed everyone's feedback. >>>>>>> >>>>>>> scale-web2 is online on running off the
scale_web_centos8 branch.
>>>>>>> it's using a test db running from a snapshot of prod 2~
weeks ago.
>>>>>>> you can change it as you see fit without worrying about
impacting production.
>>>>>>> >>>>>>> To test you'll need to update /etc/hosts to point at the
new server.
>>>>>>> following to your /etc/hosts file. >>>>>>> >>>>>>> 54.205.247.164 www.socallinuxexpo.org >>>>>>> >>>>>>> Ideally would like to merge and migrate product this
week
>>>>>>> >>>>>>> >>>>>>> On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz
phil@ipom.com wrote:
>>>>>>>> >>>>>>>> Sorry I was in Mexico. >>>>>>>> >>>>>>>> Left a review to DRY it up. But I don't see any
critical things.
>>>>>>>> >>>>>>>> - Phil >>>>>>>> >>>>>>>> On 9/22/23 15:23, Ilan Rabinovitch wrote: >>>>>>>>> Ping? Would like to wrap this up as the non-stop daily
outages are no
>>>>>>>>> fun, Managed to mitigate it somewhat, but given none
of us are on call
>>>>>>>>> getting to some reasonable sense of HA would be
appreciated so folks
>>>>>>>>> can go on vacation without worrying about who will
reboot the servers.
>>>>>>>>> >>>>>>>>> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch
ilan@linuxfests.org wrote:
>>>>>>>>>> >>>>>>>>>> Thanks Davide for the initial review. I think I
addressed the
>>>>>>>>>> remaining feedback. >>>>>>>>>> >>>>>>>>>> Anything else we want to test before merging? >>>>>>>>>> >>>>>>>>>> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch
ilan@linuxfests.org wrote:
>>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I took a stab at updating the cookbooks we use for
the website to use
>>>>>>>>>>> centos8. Motivation was getting a new version of php
so we can use
>>>>>>>>>>> some drupal plugins that dont support php5.4, but i
think this was on
>>>>>>>>>>> the backlog already for a while. >>>>>>>>>>> >>>>>>>>>>>
https://github.com/socallinuxexpo/scale-chef/pull/283
>>>>>>>>>>> >>>>>>>>>>> To test this out, I spun up scale-web2 with a clone
of the production
>>>>>>>>>>> scale-drupal database, and then pointed my host file
at it for
>>>>>>>>>>> www.socallinuxexpo.org: >>>>>>>>>>> >>>>>>>>>>> I then tested the following: >>>>>>>>>>> >>>>>>>>>>> - chef runs complete end to end >>>>>>>>>>> - Backing up / restoring static assets (see backup
scripts in
>>>>>>>>>>> /usr/local/bin or in the scale-drupal cookbook) >>>>>>>>>>> - browsing legacy static website (eg >>>>>>>>>>> https://socallinuxexpo.org/past/2002/ >>>>>>>>>>> https://socallinuxexpo.org/past/2003/) >>>>>>>>>>> - Registering as a speaker, confirming that I got
the email and that
>>>>>>>>>>> my account worked. >>>>>>>>>>> - publishing the submitted talk >>>>>>>>>>> - creating sponsors/exhibitors/blog post/events/etc >>>>>>>>>>> >>>>>>>>>>> Pending things: >>>>>>>>>>> - Centos8 seems to default to using php-fpm instead
of mod_php. Not
>>>>>>>>>>> opposed but it's different. Seems to work fine, but
I've not load
>>>>>>>>>>> tested it or anything. >>>>>>>>>>> >>>>>>>>>>> - drush (https://www.drush.org/) packages no longer
seemed to be
>>>>>>>>>>> present in centos or epel. remi seems to have it but
the drush docs
>>>>>>>>>>> suggest we should be installing this in our drupal
code base via
>>>>>>>>>>> composer instead of as a package. I dont have
strong opinions, but we
>>>>>>>>>>> should figure out the right path before merging. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Ilan >>>>>>>>> _______________________________________________ >>>>>>>>> scale-infra mailing list >>>>>>>>> scale-infra@lists.linuxfests.org >>>>>>>>>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
>>>>>>>> >>>>>>>> -- >>>>>>>> Phil Dibowitz phil@ipom.com >>>>>>>> Open Source software and tech docs Insanity
Palace of Metallica
>>>>>>>> http://www.phildev.net/
>>>>>>>> >>>>>>>> "Be who you are and say what you feel, because those
who mind don't
>>>>>>>> matter and those who matter don't mind." >>>>>>>> - Dr. Seuss >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> scale-infra mailing list >>>>>>>> scale-infra@lists.linuxfests.org >>>>>>>>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
>>>>>> _______________________________________________ >>>>>> scale-infra mailing list >>>>>> scale-infra@lists.linuxfests.org >>>>>>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
>>>>> >>>>> -- >>>>> Phil Dibowitz phil@ipom.com >>>>> Open Source software and tech docs Insanity Palace
of Metallica
>>>>> http://www.phildev.net/
>>>>> >>>>> "Be who you are and say what you feel, because those who
mind don't
>>>>> matter and those who matter don't mind." >>>>> - Dr. Seuss >>>>> >>>>> >>>>> _______________________________________________ >>>>> scale-infra mailing list >>>>> scale-infra@lists.linuxfests.org >>>>>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
>>>> _______________________________________________ >>>> scale-infra mailing list >>>> scale-infra@lists.linuxfests.org >>>>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
>>> >>> -- >>> Phil Dibowitz phil@ipom.com >>> Open Source software and tech docs Insanity Palace of
Metallica
>>> >>> "Be who you are and say what you feel, because those who
mind don't
>>> matter and those who matter don't mind." >>> - Dr. Seuss >>> >>> >>> _______________________________________________ >>> scale-infra mailing list >>> scale-infra@lists.linuxfests.org >>>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
>> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org >>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
> > -- > Phil Dibowitz phil@ipom.com > Open Source software and tech docs Insanity Palace of
Metallica
> http://www.phildev.net/ http://www.ipom.com/ > > "Be who you are and say what you feel, because those who mind
don't
> matter and those who matter don't mind." > - Dr. Seuss > > > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org >
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of
Metallica
http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind
don't
matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of
Metallica
http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra