Hi,
I took a stab at updating the cookbooks we use for the website to use centos8. Motivation was getting a new version of php so we can use some drupal plugins that dont support php5.4, but i think this was on the backlog already for a while.
https://github.com/socallinuxexpo/scale-chef/pull/283
To test this out, I spun up scale-web2 with a clone of the production scale-drupal database, and then pointed my host file at it for www.socallinuxexpo.org:
I then tested the following:
- chef runs complete end to end - Backing up / restoring static assets (see backup scripts in /usr/local/bin or in the scale-drupal cookbook) - browsing legacy static website (eg https://socallinuxexpo.org/past/2002/ https://socallinuxexpo.org/past/2003/) - Registering as a speaker, confirming that I got the email and that my account worked. - publishing the submitted talk - creating sponsors/exhibitors/blog post/events/etc
Pending things: - Centos8 seems to default to using php-fpm instead of mod_php. Not opposed but it's different. Seems to work fine, but I've not load tested it or anything.
- drush (https://www.drush.org/) packages no longer seemed to be present in centos or epel. remi seems to have it but the drush docs suggest we should be installing this in our drupal code base via composer instead of as a package. I dont have strong opinions, but we should figure out the right path before merging.
Cheers, Ilan
Thanks Davide for the initial review. I think I addressed the remaining feedback.
Anything else we want to test before merging?
On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Hi,
I took a stab at updating the cookbooks we use for the website to use centos8. Motivation was getting a new version of php so we can use some drupal plugins that dont support php5.4, but i think this was on the backlog already for a while.
https://github.com/socallinuxexpo/scale-chef/pull/283
To test this out, I spun up scale-web2 with a clone of the production scale-drupal database, and then pointed my host file at it for www.socallinuxexpo.org:
I then tested the following:
- chef runs complete end to end
- Backing up / restoring static assets (see backup scripts in
/usr/local/bin or in the scale-drupal cookbook)
- browsing legacy static website (eg
https://socallinuxexpo.org/past/2002/ https://socallinuxexpo.org/past/2003/)
- Registering as a speaker, confirming that I got the email and that
my account worked.
- publishing the submitted talk
- creating sponsors/exhibitors/blog post/events/etc
Pending things:
- Centos8 seems to default to using php-fpm instead of mod_php. Not
opposed but it's different. Seems to work fine, but I've not load tested it or anything.
- drush (https://www.drush.org/) packages no longer seemed to be
present in centos or epel. remi seems to have it but the drush docs suggest we should be installing this in our drupal code base via composer instead of as a package. I dont have strong opinions, but we should figure out the right path before merging.
Cheers, Ilan
Ping? Would like to wrap this up as the non-stop daily outages are no fun, Managed to mitigate it somewhat, but given none of us are on call getting to some reasonable sense of HA would be appreciated so folks can go on vacation without worrying about who will reboot the servers.
On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Thanks Davide for the initial review. I think I addressed the remaining feedback.
Anything else we want to test before merging?
On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Hi,
I took a stab at updating the cookbooks we use for the website to use centos8. Motivation was getting a new version of php so we can use some drupal plugins that dont support php5.4, but i think this was on the backlog already for a while.
https://github.com/socallinuxexpo/scale-chef/pull/283
To test this out, I spun up scale-web2 with a clone of the production scale-drupal database, and then pointed my host file at it for www.socallinuxexpo.org:
I then tested the following:
- chef runs complete end to end
- Backing up / restoring static assets (see backup scripts in
/usr/local/bin or in the scale-drupal cookbook)
- browsing legacy static website (eg
https://socallinuxexpo.org/past/2002/ https://socallinuxexpo.org/past/2003/)
- Registering as a speaker, confirming that I got the email and that
my account worked.
- publishing the submitted talk
- creating sponsors/exhibitors/blog post/events/etc
Pending things:
- Centos8 seems to default to using php-fpm instead of mod_php. Not
opposed but it's different. Seems to work fine, but I've not load tested it or anything.
- drush (https://www.drush.org/) packages no longer seemed to be
present in centos or epel. remi seems to have it but the drush docs suggest we should be installing this in our drupal code base via composer instead of as a package. I dont have strong opinions, but we should figure out the right path before merging.
Cheers, Ilan
Sorry I was in Mexico.
Left a review to DRY it up. But I don't see any critical things.
- Phil
On 9/22/23 15:23, Ilan Rabinovitch wrote:
Ping? Would like to wrap this up as the non-stop daily outages are no fun, Managed to mitigate it somewhat, but given none of us are on call getting to some reasonable sense of HA would be appreciated so folks can go on vacation without worrying about who will reboot the servers.
On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Thanks Davide for the initial review. I think I addressed the remaining feedback.
Anything else we want to test before merging?
On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Hi,
I took a stab at updating the cookbooks we use for the website to use centos8. Motivation was getting a new version of php so we can use some drupal plugins that dont support php5.4, but i think this was on the backlog already for a while.
https://github.com/socallinuxexpo/scale-chef/pull/283
To test this out, I spun up scale-web2 with a clone of the production scale-drupal database, and then pointed my host file at it for www.socallinuxexpo.org:
I then tested the following:
- chef runs complete end to end
- Backing up / restoring static assets (see backup scripts in
/usr/local/bin or in the scale-drupal cookbook)
- browsing legacy static website (eg
https://socallinuxexpo.org/past/2002/ https://socallinuxexpo.org/past/2003/)
- Registering as a speaker, confirming that I got the email and that
my account worked.
- publishing the submitted talk
- creating sponsors/exhibitors/blog post/events/etc
Pending things:
- Centos8 seems to default to using php-fpm instead of mod_php. Not
opposed but it's different. Seems to work fine, but I've not load tested it or anything.
- drush (https://www.drush.org/) packages no longer seemed to be
present in centos or epel. remi seems to have it but the drush docs suggest we should be installing this in our drupal code base via composer instead of as a package. I dont have strong opinions, but we should figure out the right path before merging.
Cheers, Ilan
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
I believe Ive addressed everyone's feedback.
scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production.
To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file.
54.205.247.164 www.socallinuxexpo.org
Ideally would like to merge and migrate product this week
On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Sorry I was in Mexico.
Left a review to DRY it up. But I don't see any critical things.
- Phil
On 9/22/23 15:23, Ilan Rabinovitch wrote:
Ping? Would like to wrap this up as the non-stop daily outages are no fun, Managed to mitigate it somewhat, but given none of us are on call getting to some reasonable sense of HA would be appreciated so folks can go on vacation without worrying about who will reboot the servers.
On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Thanks Davide for the initial review. I think I addressed the remaining feedback.
Anything else we want to test before merging?
On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Hi,
I took a stab at updating the cookbooks we use for the website to use centos8. Motivation was getting a new version of php so we can use some drupal plugins that dont support php5.4, but i think this was on the backlog already for a while.
https://github.com/socallinuxexpo/scale-chef/pull/283
To test this out, I spun up scale-web2 with a clone of the production scale-drupal database, and then pointed my host file at it for www.socallinuxexpo.org:
I then tested the following:
- chef runs complete end to end
- Backing up / restoring static assets (see backup scripts in
/usr/local/bin or in the scale-drupal cookbook)
- browsing legacy static website (eg
https://socallinuxexpo.org/past/2002/ https://socallinuxexpo.org/past/2003/)
- Registering as a speaker, confirming that I got the email and that
my account worked.
- publishing the submitted talk
- creating sponsors/exhibitors/blog post/events/etc
Pending things:
- Centos8 seems to default to using php-fpm instead of mod_php. Not
opposed but it's different. Seems to work fine, but I've not load tested it or anything.
- drush (https://www.drush.org/) packages no longer seemed to be
present in centos or epel. remi seems to have it but the drush docs suggest we should be installing this in our drupal code base via composer instead of as a package. I dont have strong opinions, but we should figure out the right path before merging.
Cheers, Ilan
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Thanks for doing this work. Sorry I haven't had the time to finish the migration. At least it looks like a lot of the work Davide and I did to generally support C7 and C8 seems to have done some of the heavy lifting. I'll be in Seattle this week for Chef Conf, but should be answering email.
What's the deal with the Cinc WS stuff?
Get BlueMail for Android
On Sep 24, 2023, 6:48 PM, at 6:48 PM, Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
I believe Ive addressed everyone's feedback.
scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production.
To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file.
54.205.247.164 www.socallinuxexpo.org
Ideally would like to merge and migrate product this week
On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Sorry I was in Mexico.
Left a review to DRY it up. But I don't see any critical things.
- Phil
On 9/22/23 15:23, Ilan Rabinovitch wrote:
Ping? Would like to wrap this up as the non-stop daily outages are
no
fun, Managed to mitigate it somewhat, but given none of us are on
call
getting to some reasonable sense of HA would be appreciated so
folks
can go on vacation without worrying about who will reboot the
servers.
On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch
ilan@linuxfests.org wrote:
Thanks Davide for the initial review. I think I addressed the remaining feedback.
Anything else we want to test before merging?
On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch
ilan@linuxfests.org wrote:
Hi,
I took a stab at updating the cookbooks we use for the website to
use
centos8. Motivation was getting a new version of php so we can
use
some drupal plugins that dont support php5.4, but i think this
was on
the backlog already for a while.
https://github.com/socallinuxexpo/scale-chef/pull/283
To test this out, I spun up scale-web2 with a clone of the
production
scale-drupal database, and then pointed my host file at it for www.socallinuxexpo.org:
I then tested the following:
- chef runs complete end to end
- Backing up / restoring static assets (see backup scripts in
/usr/local/bin or in the scale-drupal cookbook)
- browsing legacy static website (eg
https://socallinuxexpo.org/past/2002/ https://socallinuxexpo.org/past/2003/)
- Registering as a speaker, confirming that I got the email and
that
my account worked.
- publishing the submitted talk
- creating sponsors/exhibitors/blog post/events/etc
Pending things:
- Centos8 seems to default to using php-fpm instead of mod_php.
Not
opposed but it's different. Seems to work fine, but I've not
load
tested it or anything.
- drush (https://www.drush.org/) packages no longer seemed to be
present in centos or epel. remi seems to have it but the drush
docs
suggest we should be installing this in our drupal code base via composer instead of as a package. I dont have strong opinions,
but we
should figure out the right path before merging.
Cheers, Ilan
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of
Metallica
http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
What cinc ws stuff?
On Mon, Sep 25, 2023 at 4:12 AM Phil Dibowitz phil@ipom.com wrote:
Thanks for doing this work. Sorry I haven't had the time to finish the migration. At least it looks like a lot of the work Davide and I did to generally support C7 and C8 seems to have done some of the heavy lifting. I'll be in Seattle this week for Chef Conf, but should be answering email.
What's the deal with the Cinc WS stuff?
Get BlueMail for Android On Sep 24, 2023, at 6:48 PM, Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
I believe Ive addressed everyone's feedback.
scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production.
To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file.
54.205.247.164 www.socallinuxexpo.org
Ideally would like to merge and migrate product this week
On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Sorry I was in Mexico.
Left a review to DRY it up. But I don't see any critical things.
- Phil
On 9/22/23 15:23, Ilan Rabinovitch wrote:
Ping? Would like to wrap this up as the non-stop daily outages are no fun, Managed to mitigate it somewhat, but given none of us are on call getting to some reasonable sense of HA would be appreciated so folks can go on vacation without worrying about who will reboot the servers.
On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Thanks Davide for the initial review. I think I addressed the remaining feedback.
Anything else we want to test before merging?
On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Hi,
I took a stab at updating the cookbooks we use for the website to use centos8. Motivation was getting a new version of php so we can use some drupal plugins that dont support php5.4, but i think this was on the backlog already for a while.
https://github.com/socallinuxexpo/scale-chef/pull/283
To test this out, I spun up scale-web2 with a clone of the production scale-drupal database, and then pointed my host file at it for www.socallinuxexpo.org:
I then tested the following:
- chef runs complete end to end
- Backing up / restoring static assets (see backup scripts in
/usr/local/bin or in the scale-drupal cookbook)
- browsing legacy static website (eg
https://socallinuxexpo.org/past/2002/ https://socallinuxexpo.org/past/2003/)
- Registering as a speaker, confirming that I got the email and that
my account worked.
- publishing the submitted talk
- creating sponsors/exhibitors/blog post/events/etc
Pending things:
- Centos8 seems to default to using php-fpm instead of mod_php. Not
opposed but it's different. Seems to work fine, but I've not load tested it or anything.
- drush (https://www.drush.org/) packages no longer seemed to be
present in centos or epel. remi seems to have it but the drush docs suggest we should be installing this in our drupal code base via composer instead of as a package. I dont have strong opinions, but we should figure out the right path before merging.
Cheers, Ilan
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
You removed checking for Cinc workstation from the bootstrap script.
Get BlueMail for Android
On Sep 26, 2023, 5:02 AM, at 5:02 AM, Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
What cinc ws stuff?
On Mon, Sep 25, 2023 at 4:12 AM Phil Dibowitz phil@ipom.com wrote:
Thanks for doing this work. Sorry I haven't had the time to finish
the migration. At least it looks like a lot of the work Davide and I did to generally support C7 and C8 seems to have done some of the heavy lifting. I'll be in Seattle this week for Chef Conf, but should be answering email.
What's the deal with the Cinc WS stuff?
Get BlueMail for Android On Sep 24, 2023, at 6:48 PM, Ilan Rabinovitch
ilan@socallinuxexpo.org wrote:
I believe Ive addressed everyone's feedback.
scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting
production.
To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file.
54.205.247.164 www.socallinuxexpo.org
Ideally would like to merge and migrate product this week
On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Sorry I was in Mexico.
Left a review to DRY it up. But I don't see any critical things.
- Phil
On 9/22/23 15:23, Ilan Rabinovitch wrote:
Ping? Would like to wrap this up as the non-stop daily outages
are no
fun, Managed to mitigate it somewhat, but given none of us are on
call
getting to some reasonable sense of HA would be appreciated so
folks
can go on vacation without worrying about who will reboot the
servers.
On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch
ilan@linuxfests.org wrote:
Thanks Davide for the initial review. I think I addressed the remaining feedback.
Anything else we want to test before merging?
On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch
ilan@linuxfests.org wrote:
> > > Hi, > > I took a stab at updating the cookbooks we use for the website
to use
> centos8. Motivation was getting a new version of php so we can
use
> some drupal plugins that dont support php5.4, but i think this
was on
> the backlog already for a while. > > https://github.com/socallinuxexpo/scale-chef/pull/283 > > To test this out, I spun up scale-web2 with a clone of the
production
> scale-drupal database, and then pointed my host file at it for > www.socallinuxexpo.org: > > I then tested the following: > > - chef runs complete end to end > - Backing up / restoring static assets (see backup scripts in > /usr/local/bin or in the scale-drupal cookbook) > - browsing legacy static website (eg > https://socallinuxexpo.org/past/2002/ > https://socallinuxexpo.org/past/2003/) > - Registering as a speaker, confirming that I got the email and
that
> my account worked. > - publishing the submitted talk > - creating sponsors/exhibitors/blog post/events/etc > > Pending things: > - Centos8 seems to default to using php-fpm instead of mod_php.
Not
> opposed but it's different. Seems to work fine, but I've not
load
> tested it or anything. > > - drush (https://www.drush.org/) packages no longer seemed to
be
> present in centos or epel. remi seems to have it but the drush
docs
> suggest we should be installing this in our drupal code base
via
> composer instead of as a package. I dont have strong opinions,
but we
> should figure out the right path before merging. > > Cheers, > Ilan
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of
Metallica
http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind
don't
matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
That was unintentional. It's now removed. That part of tt script doesn't seem work on OSX or zsh or some combo of the two and I didn't have the patience to troubleshoot it, after all the fun with the actual chef stuff.
Anyways, anything else before we merge this PR? Anyone want to further test scale-web2 before we cutover?
We still have some ordering / docs issues to deal with, but I've been adding that to the issue I opened.
On Tue, Sep 26, 2023 at 9:28 AM Phil Dibowitz phil@ipom.com wrote:
You removed checking for Cinc workstation from the bootstrap script.
Get BlueMail for Android On Sep 26, 2023, at 5:02 AM, Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
What cinc ws stuff?
On Mon, Sep 25, 2023 at 4:12 AM Phil Dibowitz phil@ipom.com wrote:
Thanks for doing this work. Sorry I haven't had the time to finish the migration. At least it looks like a lot of the work Davide and I did to generally support C7 and C8 seems to have done some of the heavy lifting. I'll be in Seattle this week for Chef Conf, but should be answering email.
What's the deal with the Cinc WS stuff?
Get BlueMail for Android On Sep 24, 2023, at 6:48 PM, Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
I believe Ive addressed everyone's feedback.
scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production.
To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file.
54.205.247.164 www.socallinuxexpo.org
Ideally would like to merge and migrate product this week
On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Sorry I was in Mexico.
Left a review to DRY it up. But I don't see any critical things.
- Phil
On 9/22/23 15:23, Ilan Rabinovitch wrote:
Ping? Would like to wrap this up as the non-stop daily outages are no fun, Managed to mitigate it somewhat, but given none of us are on call getting to some reasonable sense of HA would be appreciated so folks can go on vacation without worrying about who will reboot the servers.
On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: > > > > Thanks Davide for the initial review. I think I addressed the > remaining feedback. > > Anything else we want to test before merging? > > On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >> >> >> >> Hi, >> >> I took a stab at updating the cookbooks we use for the website to use >> centos8. Motivation was getting a new version of php so we can use >> some drupal plugins that dont support php5.4, but i think this was on >> the backlog already for a while. >> >> https://github.com/socallinuxexpo/scale-chef/pull/283 >> >> To test this out, I spun up scale-web2 with a clone of the production >> scale-drupal database, and then pointed my host file at it for >> www.socallinuxexpo.org: >> >> I then tested the following: >> >> - chef runs complete end to end >> - Backing up / restoring static assets (see backup scripts in >> /usr/local/bin or in the scale-drupal cookbook) >> - browsing legacy static website (eg >> https://socallinuxexpo.org/past/2002/ >> https://socallinuxexpo.org/past/2003/) >> - Registering as a speaker, confirming that I got the email and that >> my account worked. >> - publishing the submitted talk >> - creating sponsors/exhibitors/blog post/events/etc >> >> Pending things: >> - Centos8 seems to default to using php-fpm instead of mod_php. Not >> opposed but it's different. Seems to work fine, but I've not load >> tested it or anything. >> >> - drush (https://www.drush.org/) packages no longer seemed to be >> present in centos or epel. remi seems to have it but the drush docs >> suggest we should be installing this in our drupal code base via >> composer instead of as a package. I dont have strong opinions, but we >> should figure out the right path before merging. >> >> Cheers, >> Ilan
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Yeah all development has only every happened in Linux. No idea how much of TT and friends work on other platforms. Recommend a VM or distrobox for development.
PR is good. Have no time to test until I'm back from chefconf.
Get BlueMail for Android
On Sep 26, 2023, 8:51 AM, at 8:51 AM, Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
That was unintentional. It's now removed. That part of tt script doesn't seem work on OSX or zsh or some combo of the two and I didn't have the patience to troubleshoot it, after all the fun with the actual chef stuff.
Anyways, anything else before we merge this PR? Anyone want to further test scale-web2 before we cutover?
We still have some ordering / docs issues to deal with, but I've been adding that to the issue I opened.
On Tue, Sep 26, 2023 at 9:28 AM Phil Dibowitz phil@ipom.com wrote:
You removed checking for Cinc workstation from the bootstrap script.
Get BlueMail for Android On Sep 26, 2023, at 5:02 AM, Ilan Rabinovitch
ilan@socallinuxexpo.org wrote:
What cinc ws stuff?
On Mon, Sep 25, 2023 at 4:12 AM Phil Dibowitz phil@ipom.com wrote:
Thanks for doing this work. Sorry I haven't had the time to finish
the migration. At least it looks like a lot of the work Davide and I did to generally support C7 and C8 seems to have done some of the heavy lifting. I'll be in Seattle this week for Chef Conf, but should be answering email.
What's the deal with the Cinc WS stuff?
Get BlueMail for Android On Sep 24, 2023, at 6:48 PM, Ilan Rabinovitch
ilan@socallinuxexpo.org wrote:
I believe Ive addressed everyone's feedback.
scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks
ago.
you can change it as you see fit without worrying about impacting
production.
To test you'll need to update /etc/hosts to point at the new
server.
following to your /etc/hosts file.
54.205.247.164 www.socallinuxexpo.org
Ideally would like to merge and migrate product this week
On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com
wrote:
Sorry I was in Mexico.
Left a review to DRY it up. But I don't see any critical
things.
- Phil
On 9/22/23 15:23, Ilan Rabinovitch wrote: > > > Ping? Would like to wrap this up as the non-stop daily outages
are no
> fun, Managed to mitigate it somewhat, but given none of us are
on call
> getting to some reasonable sense of HA would be appreciated so
folks
> can go on vacation without worrying about who will reboot the
servers.
> > On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch
ilan@linuxfests.org wrote:
>> >> >> >> Thanks Davide for the initial review. I think I addressed
the
>> remaining feedback. >> >> Anything else we want to test before merging? >> >> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch
ilan@linuxfests.org wrote:
>>> >>> >>> >>> Hi, >>> >>> I took a stab at updating the cookbooks we use for the
website to use
>>> centos8. Motivation was getting a new version of php so we
can use
>>> some drupal plugins that dont support php5.4, but i think
this was on
>>> the backlog already for a while. >>> >>> https://github.com/socallinuxexpo/scale-chef/pull/283 >>> >>> To test this out, I spun up scale-web2 with a clone of the
production
>>> scale-drupal database, and then pointed my host file at it
for
>>> www.socallinuxexpo.org: >>> >>> I then tested the following: >>> >>> - chef runs complete end to end >>> - Backing up / restoring static assets (see backup scripts
in
>>> /usr/local/bin or in the scale-drupal cookbook) >>> - browsing legacy static website (eg >>> https://socallinuxexpo.org/past/2002/ >>> https://socallinuxexpo.org/past/2003/) >>> - Registering as a speaker, confirming that I got the email
and that
>>> my account worked. >>> - publishing the submitted talk >>> - creating sponsors/exhibitors/blog post/events/etc >>> >>> Pending things: >>> - Centos8 seems to default to using php-fpm instead of
mod_php. Not
>>> opposed but it's different. Seems to work fine, but I've
not load
>>> tested it or anything. >>> >>> - drush (https://www.drush.org/) packages no longer seemed
to be
>>> present in centos or epel. remi seems to have it but the
drush docs
>>> suggest we should be installing this in our drupal code base
via
>>> composer instead of as a package. I dont have strong
opinions, but we
>>> should figure out the right path before merging. >>> >>> Cheers, >>> Ilan > > > ________________________________ > > > scale-infra mailing list > scale-infra@lists.linuxfests.org >
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of
Metallica
http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind
don't
matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Phil Smith, myself and a few others have banged on scale-web2 and we think we're all set to do a cutover there. However, I'm going to be out of town for a few weeks starting Thursday and do not yet know what my connectivity will be like.
Would love to cutover this week, but only want to do it if others will be available to roll back to the old server should something go awry while I'm on vacation.
On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
I believe Ive addressed everyone's feedback.
scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production.
To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file.
54.205.247.164 www.socallinuxexpo.org
Ideally would like to merge and migrate product this week
On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Sorry I was in Mexico.
Left a review to DRY it up. But I don't see any critical things.
- Phil
On 9/22/23 15:23, Ilan Rabinovitch wrote:
Ping? Would like to wrap this up as the non-stop daily outages are no fun, Managed to mitigate it somewhat, but given none of us are on call getting to some reasonable sense of HA would be appreciated so folks can go on vacation without worrying about who will reboot the servers.
On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Thanks Davide for the initial review. I think I addressed the remaining feedback.
Anything else we want to test before merging?
On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Hi,
I took a stab at updating the cookbooks we use for the website to use centos8. Motivation was getting a new version of php so we can use some drupal plugins that dont support php5.4, but i think this was on the backlog already for a while.
https://github.com/socallinuxexpo/scale-chef/pull/283
To test this out, I spun up scale-web2 with a clone of the production scale-drupal database, and then pointed my host file at it for www.socallinuxexpo.org:
I then tested the following:
- chef runs complete end to end
- Backing up / restoring static assets (see backup scripts in
/usr/local/bin or in the scale-drupal cookbook)
- browsing legacy static website (eg
https://socallinuxexpo.org/past/2002/ https://socallinuxexpo.org/past/2003/)
- Registering as a speaker, confirming that I got the email and that
my account worked.
- publishing the submitted talk
- creating sponsors/exhibitors/blog post/events/etc
Pending things:
- Centos8 seems to default to using php-fpm instead of mod_php. Not
opposed but it's different. Seems to work fine, but I've not load tested it or anything.
- drush (https://www.drush.org/) packages no longer seemed to be
present in centos or epel. remi seems to have it but the drush docs suggest we should be installing this in our drupal code base via composer instead of as a package. I dont have strong opinions, but we should figure out the right path before merging.
Cheers, Ilan
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
I'm out of town Fri-Sun, and will only be around a laptop sporadically. I'm in town all week next week though.
On 10/2/23 13:22, Ilan Rabinovitch wrote:
Phil Smith, myself and a few others have banged on scale-web2 and we think we're all set to do a cutover there. However, I'm going to be out of town for a few weeks starting Thursday and do not yet know what my connectivity will be like.
Would love to cutover this week, but only want to do it if others will be available to roll back to the old server should something go awry while I'm on vacation.
On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
I believe Ive addressed everyone's feedback.
scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production.
To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file.
54.205.247.164 www.socallinuxexpo.org
Ideally would like to merge and migrate product this week
On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Sorry I was in Mexico.
Left a review to DRY it up. But I don't see any critical things.
- Phil
On 9/22/23 15:23, Ilan Rabinovitch wrote:
Ping? Would like to wrap this up as the non-stop daily outages are no fun, Managed to mitigate it somewhat, but given none of us are on call getting to some reasonable sense of HA would be appreciated so folks can go on vacation without worrying about who will reboot the servers.
On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Thanks Davide for the initial review. I think I addressed the remaining feedback.
Anything else we want to test before merging?
On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Hi,
I took a stab at updating the cookbooks we use for the website to use centos8. Motivation was getting a new version of php so we can use some drupal plugins that dont support php5.4, but i think this was on the backlog already for a while.
https://github.com/socallinuxexpo/scale-chef/pull/283
To test this out, I spun up scale-web2 with a clone of the production scale-drupal database, and then pointed my host file at it for www.socallinuxexpo.org:
I then tested the following:
- chef runs complete end to end
- Backing up / restoring static assets (see backup scripts in
/usr/local/bin or in the scale-drupal cookbook)
- browsing legacy static website (eg
https://socallinuxexpo.org/past/2002/ https://socallinuxexpo.org/past/2003/)
- Registering as a speaker, confirming that I got the email and that
my account worked.
- publishing the submitted talk
- creating sponsors/exhibitors/blog post/events/etc
Pending things:
- Centos8 seems to default to using php-fpm instead of mod_php. Not
opposed but it's different. Seems to work fine, but I've not load tested it or anything.
- drush (https://www.drush.org/) packages no longer seemed to be
present in centos or epel. remi seems to have it but the drush docs suggest we should be installing this in our drupal code base via composer instead of as a package. I dont have strong opinions, but we should figure out the right path before merging.
Cheers, Ilan
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Sounds like we should do this today then, or you can grab next week. I'm going to be offline until November after Thursday morning.
On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote:
I'm out of town Fri-Sun, and will only be around a laptop sporadically. I'm in town all week next week though.
On 10/2/23 13:22, Ilan Rabinovitch wrote:
Phil Smith, myself and a few others have banged on scale-web2 and we think we're all set to do a cutover there. However, I'm going to be out of town for a few weeks starting Thursday and do not yet know what my connectivity will be like.
Would love to cutover this week, but only want to do it if others will be available to roll back to the old server should something go awry while I'm on vacation.
On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
I believe Ive addressed everyone's feedback.
scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production.
To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file.
54.205.247.164 www.socallinuxexpo.org
Ideally would like to merge and migrate product this week
On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Sorry I was in Mexico.
Left a review to DRY it up. But I don't see any critical things.
- Phil
On 9/22/23 15:23, Ilan Rabinovitch wrote:
Ping? Would like to wrap this up as the non-stop daily outages are no fun, Managed to mitigate it somewhat, but given none of us are on call getting to some reasonable sense of HA would be appreciated so folks can go on vacation without worrying about who will reboot the servers.
On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
Thanks Davide for the initial review. I think I addressed the remaining feedback.
Anything else we want to test before merging?
On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: > > Hi, > > I took a stab at updating the cookbooks we use for the website to use > centos8. Motivation was getting a new version of php so we can use > some drupal plugins that dont support php5.4, but i think this was on > the backlog already for a while. > > https://github.com/socallinuxexpo/scale-chef/pull/283 > > To test this out, I spun up scale-web2 with a clone of the production > scale-drupal database, and then pointed my host file at it for > www.socallinuxexpo.org: > > I then tested the following: > > - chef runs complete end to end > - Backing up / restoring static assets (see backup scripts in > /usr/local/bin or in the scale-drupal cookbook) > - browsing legacy static website (eg > https://socallinuxexpo.org/past/2002/ > https://socallinuxexpo.org/past/2003/) > - Registering as a speaker, confirming that I got the email and that > my account worked. > - publishing the submitted talk > - creating sponsors/exhibitors/blog post/events/etc > > Pending things: > - Centos8 seems to default to using php-fpm instead of mod_php. Not > opposed but it's different. Seems to work fine, but I've not load > tested it or anything. > > - drush (https://www.drush.org/) packages no longer seemed to be > present in centos or epel. remi seems to have it but the drush docs > suggest we should be installing this in our drupal code base via > composer instead of as a package. I dont have strong opinions, but we > should figure out the right path before merging. > > Cheers, > Ilan
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Feel free to do it today.
What's the migrate/failback process? It's somewhere in cloudflair?
If not I can do it next week.
On 10/3/23 11:09, Ilan Rabinovitch wrote:
Sounds like we should do this today then, or you can grab next week. I'm going to be offline until November after Thursday morning.
On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote:
I'm out of town Fri-Sun, and will only be around a laptop sporadically. I'm in town all week next week though.
On 10/2/23 13:22, Ilan Rabinovitch wrote:
Phil Smith, myself and a few others have banged on scale-web2 and we think we're all set to do a cutover there. However, I'm going to be out of town for a few weeks starting Thursday and do not yet know what my connectivity will be like.
Would love to cutover this week, but only want to do it if others will be available to roll back to the old server should something go awry while I'm on vacation.
On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
I believe Ive addressed everyone's feedback.
scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production.
To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file.
54.205.247.164 www.socallinuxexpo.org
Ideally would like to merge and migrate product this week
On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Sorry I was in Mexico.
Left a review to DRY it up. But I don't see any critical things.
- Phil
On 9/22/23 15:23, Ilan Rabinovitch wrote:
Ping? Would like to wrap this up as the non-stop daily outages are no fun, Managed to mitigate it somewhat, but given none of us are on call getting to some reasonable sense of HA would be appreciated so folks can go on vacation without worrying about who will reboot the servers.
On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: > > Thanks Davide for the initial review. I think I addressed the > remaining feedback. > > Anything else we want to test before merging? > > On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >> >> Hi, >> >> I took a stab at updating the cookbooks we use for the website to use >> centos8. Motivation was getting a new version of php so we can use >> some drupal plugins that dont support php5.4, but i think this was on >> the backlog already for a while. >> >> https://github.com/socallinuxexpo/scale-chef/pull/283 >> >> To test this out, I spun up scale-web2 with a clone of the production >> scale-drupal database, and then pointed my host file at it for >> www.socallinuxexpo.org: >> >> I then tested the following: >> >> - chef runs complete end to end >> - Backing up / restoring static assets (see backup scripts in >> /usr/local/bin or in the scale-drupal cookbook) >> - browsing legacy static website (eg >> https://socallinuxexpo.org/past/2002/ >> https://socallinuxexpo.org/past/2003/) >> - Registering as a speaker, confirming that I got the email and that >> my account worked. >> - publishing the submitted talk >> - creating sponsors/exhibitors/blog post/events/etc >> >> Pending things: >> - Centos8 seems to default to using php-fpm instead of mod_php. Not >> opposed but it's different. Seems to work fine, but I've not load >> tested it or anything. >> >> - drush (https://www.drush.org/) packages no longer seemed to be >> present in centos or epel. remi seems to have it but the drush docs >> suggest we should be installing this in our drupal code base via >> composer instead of as a package. I dont have strong opinions, but we >> should figure out the right path before merging. >> >> Cheers, >> Ilan _______________________________________________ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Cutover: - RDS Database snapshot - sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both hosts in case users have uploaded static assets - Move EIP from scale-web1 host to scale-web2
Rollback: - re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both hosts in case users have uploaded static assets - Move EIP from scale-web2 host to scale-web1
On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com wrote:
Feel free to do it today.
What's the migrate/failback process? It's somewhere in cloudflair?
If not I can do it next week.
On 10/3/23 11:09, Ilan Rabinovitch wrote:
Sounds like we should do this today then, or you can grab next week. I'm going to be offline until November after Thursday morning.
On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote:
I'm out of town Fri-Sun, and will only be around a laptop sporadically. I'm in town all week next week though.
On 10/2/23 13:22, Ilan Rabinovitch wrote:
Phil Smith, myself and a few others have banged on scale-web2 and we think we're all set to do a cutover there. However, I'm going to be out of town for a few weeks starting Thursday and do not yet know what my connectivity will be like.
Would love to cutover this week, but only want to do it if others will be available to roll back to the old server should something go awry while I'm on vacation.
On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
I believe Ive addressed everyone's feedback.
scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production.
To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file.
54.205.247.164 www.socallinuxexpo.org
Ideally would like to merge and migrate product this week
On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Sorry I was in Mexico.
Left a review to DRY it up. But I don't see any critical things.
- Phil
On 9/22/23 15:23, Ilan Rabinovitch wrote: > Ping? Would like to wrap this up as the non-stop daily outages are no > fun, Managed to mitigate it somewhat, but given none of us are on call > getting to some reasonable sense of HA would be appreciated so folks > can go on vacation without worrying about who will reboot the servers. > > On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >> >> Thanks Davide for the initial review. I think I addressed the >> remaining feedback. >> >> Anything else we want to test before merging? >> >> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>> >>> Hi, >>> >>> I took a stab at updating the cookbooks we use for the website to use >>> centos8. Motivation was getting a new version of php so we can use >>> some drupal plugins that dont support php5.4, but i think this was on >>> the backlog already for a while. >>> >>> https://github.com/socallinuxexpo/scale-chef/pull/283 >>> >>> To test this out, I spun up scale-web2 with a clone of the production >>> scale-drupal database, and then pointed my host file at it for >>> www.socallinuxexpo.org: >>> >>> I then tested the following: >>> >>> - chef runs complete end to end >>> - Backing up / restoring static assets (see backup scripts in >>> /usr/local/bin or in the scale-drupal cookbook) >>> - browsing legacy static website (eg >>> https://socallinuxexpo.org/past/2002/ >>> https://socallinuxexpo.org/past/2003/) >>> - Registering as a speaker, confirming that I got the email and that >>> my account worked. >>> - publishing the submitted talk >>> - creating sponsors/exhibitors/blog post/events/etc >>> >>> Pending things: >>> - Centos8 seems to default to using php-fpm instead of mod_php. Not >>> opposed but it's different. Seems to work fine, but I've not load >>> tested it or anything. >>> >>> - drush (https://www.drush.org/) packages no longer seemed to be >>> present in centos or epel. remi seems to have it but the drush docs >>> suggest we should be installing this in our drupal code base via >>> composer instead of as a package. I dont have strong opinions, but we >>> should figure out the right path before merging. >>> >>> Cheers, >>> Ilan > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday.
On 10/3/23 16:08, Ilan Rabinovitch wrote:
Cutover:
- RDS Database snapshot
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
hosts in case users have uploaded static assets
- Move EIP from scale-web1 host to scale-web2
Rollback:
- re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on
both hosts in case users have uploaded static assets
- Move EIP from scale-web2 host to scale-web1
On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com wrote:
Feel free to do it today.
What's the migrate/failback process? It's somewhere in cloudflair?
If not I can do it next week.
On 10/3/23 11:09, Ilan Rabinovitch wrote:
Sounds like we should do this today then, or you can grab next week. I'm going to be offline until November after Thursday morning.
On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote:
I'm out of town Fri-Sun, and will only be around a laptop sporadically. I'm in town all week next week though.
On 10/2/23 13:22, Ilan Rabinovitch wrote:
Phil Smith, myself and a few others have banged on scale-web2 and we think we're all set to do a cutover there. However, I'm going to be out of town for a few weeks starting Thursday and do not yet know what my connectivity will be like.
Would love to cutover this week, but only want to do it if others will be available to roll back to the old server should something go awry while I'm on vacation.
On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
I believe Ive addressed everyone's feedback.
scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production.
To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file.
54.205.247.164 www.socallinuxexpo.org
Ideally would like to merge and migrate product this week
On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote: > > Sorry I was in Mexico. > > Left a review to DRY it up. But I don't see any critical things. > > - Phil > > On 9/22/23 15:23, Ilan Rabinovitch wrote: >> Ping? Would like to wrap this up as the non-stop daily outages are no >> fun, Managed to mitigate it somewhat, but given none of us are on call >> getting to some reasonable sense of HA would be appreciated so folks >> can go on vacation without worrying about who will reboot the servers. >> >> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>> >>> Thanks Davide for the initial review. I think I addressed the >>> remaining feedback. >>> >>> Anything else we want to test before merging? >>> >>> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>> >>>> Hi, >>>> >>>> I took a stab at updating the cookbooks we use for the website to use >>>> centos8. Motivation was getting a new version of php so we can use >>>> some drupal plugins that dont support php5.4, but i think this was on >>>> the backlog already for a while. >>>> >>>> https://github.com/socallinuxexpo/scale-chef/pull/283 >>>> >>>> To test this out, I spun up scale-web2 with a clone of the production >>>> scale-drupal database, and then pointed my host file at it for >>>> www.socallinuxexpo.org: >>>> >>>> I then tested the following: >>>> >>>> - chef runs complete end to end >>>> - Backing up / restoring static assets (see backup scripts in >>>> /usr/local/bin or in the scale-drupal cookbook) >>>> - browsing legacy static website (eg >>>> https://socallinuxexpo.org/past/2002/ >>>> https://socallinuxexpo.org/past/2003/) >>>> - Registering as a speaker, confirming that I got the email and that >>>> my account worked. >>>> - publishing the submitted talk >>>> - creating sponsors/exhibitors/blog post/events/etc >>>> >>>> Pending things: >>>> - Centos8 seems to default to using php-fpm instead of mod_php. Not >>>> opposed but it's different. Seems to work fine, but I've not load >>>> tested it or anything. >>>> >>>> - drush (https://www.drush.org/) packages no longer seemed to be >>>> present in centos or epel. remi seems to have it but the drush docs >>>> suggest we should be installing this in our drupal code base via >>>> composer instead of as a package. I dont have strong opinions, but we >>>> should figure out the right path before merging. >>>> >>>> Cheers, >>>> Ilan >> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org >> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra > > -- > Phil Dibowitz phil@ipom.com > Open Source software and tech docs Insanity Palace of Metallica > http://www.phildev.net/ http://www.ipom.com/ > > "Be who you are and say what you feel, because those who mind don't > matter and those who matter don't mind." > - Dr. Seuss > > > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
This is done. Here's the specific ids for any roll backs:
1) start scale-web1 up again (i-017df11978f41d9ae) 2) sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae) 3) move EIP (23.21.71.118) from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.
Assuming no issues the next change to think about is: - enable the s3fs plugin work Phil Smith did recently so static files aren't stored on box anymore. - move scale-web2 behind an ELB and auto scaling group. - remove the cron jobs that back up static files
This will enable some reasonable failover that doesn't require human intervention.
On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday.
On 10/3/23 16:08, Ilan Rabinovitch wrote:
Cutover:
- RDS Database snapshot
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
hosts in case users have uploaded static assets
- Move EIP from scale-web1 host to scale-web2
Rollback:
- re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on
both hosts in case users have uploaded static assets
- Move EIP from scale-web2 host to scale-web1
On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com wrote:
Feel free to do it today.
What's the migrate/failback process? It's somewhere in cloudflair?
If not I can do it next week.
On 10/3/23 11:09, Ilan Rabinovitch wrote:
Sounds like we should do this today then, or you can grab next week. I'm going to be offline until November after Thursday morning.
On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote:
I'm out of town Fri-Sun, and will only be around a laptop sporadically. I'm in town all week next week though.
On 10/2/23 13:22, Ilan Rabinovitch wrote:
Phil Smith, myself and a few others have banged on scale-web2 and we think we're all set to do a cutover there. However, I'm going to be out of town for a few weeks starting Thursday and do not yet know what my connectivity will be like.
Would love to cutover this week, but only want to do it if others will be available to roll back to the old server should something go awry while I'm on vacation.
On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch ilan@socallinuxexpo.org wrote: > > I believe Ive addressed everyone's feedback. > > scale-web2 is online on running off the scale_web_centos8 branch. > it's using a test db running from a snapshot of prod 2~ weeks ago. > you can change it as you see fit without worrying about impacting production. > > To test you'll need to update /etc/hosts to point at the new server. > following to your /etc/hosts file. > > 54.205.247.164 www.socallinuxexpo.org > > Ideally would like to merge and migrate product this week > > > On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote: >> >> Sorry I was in Mexico. >> >> Left a review to DRY it up. But I don't see any critical things. >> >> - Phil >> >> On 9/22/23 15:23, Ilan Rabinovitch wrote: >>> Ping? Would like to wrap this up as the non-stop daily outages are no >>> fun, Managed to mitigate it somewhat, but given none of us are on call >>> getting to some reasonable sense of HA would be appreciated so folks >>> can go on vacation without worrying about who will reboot the servers. >>> >>> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>> >>>> Thanks Davide for the initial review. I think I addressed the >>>> remaining feedback. >>>> >>>> Anything else we want to test before merging? >>>> >>>> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>> >>>>> Hi, >>>>> >>>>> I took a stab at updating the cookbooks we use for the website to use >>>>> centos8. Motivation was getting a new version of php so we can use >>>>> some drupal plugins that dont support php5.4, but i think this was on >>>>> the backlog already for a while. >>>>> >>>>> https://github.com/socallinuxexpo/scale-chef/pull/283 >>>>> >>>>> To test this out, I spun up scale-web2 with a clone of the production >>>>> scale-drupal database, and then pointed my host file at it for >>>>> www.socallinuxexpo.org: >>>>> >>>>> I then tested the following: >>>>> >>>>> - chef runs complete end to end >>>>> - Backing up / restoring static assets (see backup scripts in >>>>> /usr/local/bin or in the scale-drupal cookbook) >>>>> - browsing legacy static website (eg >>>>> https://socallinuxexpo.org/past/2002/ >>>>> https://socallinuxexpo.org/past/2003/) >>>>> - Registering as a speaker, confirming that I got the email and that >>>>> my account worked. >>>>> - publishing the submitted talk >>>>> - creating sponsors/exhibitors/blog post/events/etc >>>>> >>>>> Pending things: >>>>> - Centos8 seems to default to using php-fpm instead of mod_php. Not >>>>> opposed but it's different. Seems to work fine, but I've not load >>>>> tested it or anything. >>>>> >>>>> - drush (https://www.drush.org/) packages no longer seemed to be >>>>> present in centos or epel. remi seems to have it but the drush docs >>>>> suggest we should be installing this in our drupal code base via >>>>> composer instead of as a package. I dont have strong opinions, but we >>>>> should figure out the right path before merging. >>>>> >>>>> Cheers, >>>>> Ilan >>> _______________________________________________ >>> scale-infra mailing list >>> scale-infra@lists.linuxfests.org >>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >> >> -- >> Phil Dibowitz phil@ipom.com >> Open Source software and tech docs Insanity Palace of Metallica >> http://www.phildev.net/ http://www.ipom.com/ >> >> "Be who you are and say what you feel, because those who mind don't >> matter and those who matter don't mind." >> - Dr. Seuss >> >> >> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org >> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra _______________________________________________ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Copy that, I'll keep an eye out. Nice work, and thanks!
On 10/3/23 18:06, Ilan Rabinovitch wrote:
This is done. Here's the specific ids for any roll backs:
- start scale-web1 up again (i-017df11978f41d9ae)
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ from
scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae) 3) move EIP (23.21.71.118) from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.
Assuming no issues the next change to think about is:
- enable the s3fs plugin work Phil Smith did recently so static files
aren't stored on box anymore.
- move scale-web2 behind an ELB and auto scaling group.
- remove the cron jobs that back up static files
This will enable some reasonable failover that doesn't require human intervention.
On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday.
On 10/3/23 16:08, Ilan Rabinovitch wrote:
Cutover:
- RDS Database snapshot
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
hosts in case users have uploaded static assets
- Move EIP from scale-web1 host to scale-web2
Rollback:
- re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on
both hosts in case users have uploaded static assets
- Move EIP from scale-web2 host to scale-web1
On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com wrote:
Feel free to do it today.
What's the migrate/failback process? It's somewhere in cloudflair?
If not I can do it next week.
On 10/3/23 11:09, Ilan Rabinovitch wrote:
Sounds like we should do this today then, or you can grab next week. I'm going to be offline until November after Thursday morning.
On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote:
I'm out of town Fri-Sun, and will only be around a laptop sporadically. I'm in town all week next week though.
On 10/2/23 13:22, Ilan Rabinovitch wrote: > Phil Smith, myself and a few others have banged on scale-web2 and we > think we're all set to do a cutover there. > However, I'm going to be out of town for a few weeks starting Thursday > and do not yet know what my connectivity will be like. > > Would love to cutover this week, but only want to do it if others will > be available to roll back to the old server should something go awry > while I'm on vacation. > > On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch > ilan@socallinuxexpo.org wrote: >> >> I believe Ive addressed everyone's feedback. >> >> scale-web2 is online on running off the scale_web_centos8 branch. >> it's using a test db running from a snapshot of prod 2~ weeks ago. >> you can change it as you see fit without worrying about impacting production. >> >> To test you'll need to update /etc/hosts to point at the new server. >> following to your /etc/hosts file. >> >> 54.205.247.164 www.socallinuxexpo.org >> >> Ideally would like to merge and migrate product this week >> >> >> On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote: >>> >>> Sorry I was in Mexico. >>> >>> Left a review to DRY it up. But I don't see any critical things. >>> >>> - Phil >>> >>> On 9/22/23 15:23, Ilan Rabinovitch wrote: >>>> Ping? Would like to wrap this up as the non-stop daily outages are no >>>> fun, Managed to mitigate it somewhat, but given none of us are on call >>>> getting to some reasonable sense of HA would be appreciated so folks >>>> can go on vacation without worrying about who will reboot the servers. >>>> >>>> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>> >>>>> Thanks Davide for the initial review. I think I addressed the >>>>> remaining feedback. >>>>> >>>>> Anything else we want to test before merging? >>>>> >>>>> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> I took a stab at updating the cookbooks we use for the website to use >>>>>> centos8. Motivation was getting a new version of php so we can use >>>>>> some drupal plugins that dont support php5.4, but i think this was on >>>>>> the backlog already for a while. >>>>>> >>>>>> https://github.com/socallinuxexpo/scale-chef/pull/283 >>>>>> >>>>>> To test this out, I spun up scale-web2 with a clone of the production >>>>>> scale-drupal database, and then pointed my host file at it for >>>>>> www.socallinuxexpo.org: >>>>>> >>>>>> I then tested the following: >>>>>> >>>>>> - chef runs complete end to end >>>>>> - Backing up / restoring static assets (see backup scripts in >>>>>> /usr/local/bin or in the scale-drupal cookbook) >>>>>> - browsing legacy static website (eg >>>>>> https://socallinuxexpo.org/past/2002/ >>>>>> https://socallinuxexpo.org/past/2003/) >>>>>> - Registering as a speaker, confirming that I got the email and that >>>>>> my account worked. >>>>>> - publishing the submitted talk >>>>>> - creating sponsors/exhibitors/blog post/events/etc >>>>>> >>>>>> Pending things: >>>>>> - Centos8 seems to default to using php-fpm instead of mod_php. Not >>>>>> opposed but it's different. Seems to work fine, but I've not load >>>>>> tested it or anything. >>>>>> >>>>>> - drush (https://www.drush.org/) packages no longer seemed to be >>>>>> present in centos or epel. remi seems to have it but the drush docs >>>>>> suggest we should be installing this in our drupal code base via >>>>>> composer instead of as a package. I dont have strong opinions, but we >>>>>> should figure out the right path before merging. >>>>>> >>>>>> Cheers, >>>>>> Ilan >>>> _______________________________________________ >>>> scale-infra mailing list >>>> scale-infra@lists.linuxfests.org >>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>> >>> -- >>> Phil Dibowitz phil@ipom.com >>> Open Source software and tech docs Insanity Palace of Metallica >>> http://www.phildev.net/ http://www.ipom.com/ >>> >>> "Be who you are and say what you feel, because those who mind don't >>> matter and those who matter don't mind." >>> - Dr. Seuss >>> >>> >>> _______________________________________________ >>> scale-infra mailing list >>> scale-infra@lists.linuxfests.org >>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Theres definitely something continuing to go on with drupal and/or httpd. Something continues to eat up all the worker threads from httpd. Before we set maxworkers to something reasonable it would eat up all the memory, now it just consumes all the workers and the site becomes unresponsive. I dont think this relates to the upgrade, as it was happening before.
We've had the site down three times in the past 2 days as a result. Monitoring detected it and I was able to respond to it fairly quickly, but I'm about to board a 30 hour flight, so I'm likely not going to be able to respond moving forward.
Given the importance of the CFP this month I hope someone has a chance to dig in.
On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz phil@ipom.com wrote:
Copy that, I'll keep an eye out. Nice work, and thanks!
On 10/3/23 18:06, Ilan Rabinovitch wrote:
This is done. Here's the specific ids for any roll backs:
- start scale-web1 up again (i-017df11978f41d9ae)
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ from
scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae) 3) move EIP (23.21.71.118) from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.
Assuming no issues the next change to think about is:
- enable the s3fs plugin work Phil Smith did recently so static files
aren't stored on box anymore.
- move scale-web2 behind an ELB and auto scaling group.
- remove the cron jobs that back up static files
This will enable some reasonable failover that doesn't require human intervention.
On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday.
On 10/3/23 16:08, Ilan Rabinovitch wrote:
Cutover:
- RDS Database snapshot
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
hosts in case users have uploaded static assets
- Move EIP from scale-web1 host to scale-web2
Rollback:
- re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on
both hosts in case users have uploaded static assets
- Move EIP from scale-web2 host to scale-web1
On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com wrote:
Feel free to do it today.
What's the migrate/failback process? It's somewhere in cloudflair?
If not I can do it next week.
On 10/3/23 11:09, Ilan Rabinovitch wrote:
Sounds like we should do this today then, or you can grab next week. I'm going to be offline until November after Thursday morning.
On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote: > > I'm out of town Fri-Sun, and will only be around a laptop sporadically. > I'm in town all week next week though. > > > > On 10/2/23 13:22, Ilan Rabinovitch wrote: >> Phil Smith, myself and a few others have banged on scale-web2 and we >> think we're all set to do a cutover there. >> However, I'm going to be out of town for a few weeks starting Thursday >> and do not yet know what my connectivity will be like. >> >> Would love to cutover this week, but only want to do it if others will >> be available to roll back to the old server should something go awry >> while I'm on vacation. >> >> On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch >> ilan@socallinuxexpo.org wrote: >>> >>> I believe Ive addressed everyone's feedback. >>> >>> scale-web2 is online on running off the scale_web_centos8 branch. >>> it's using a test db running from a snapshot of prod 2~ weeks ago. >>> you can change it as you see fit without worrying about impacting production. >>> >>> To test you'll need to update /etc/hosts to point at the new server. >>> following to your /etc/hosts file. >>> >>> 54.205.247.164 www.socallinuxexpo.org >>> >>> Ideally would like to merge and migrate product this week >>> >>> >>> On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote: >>>> >>>> Sorry I was in Mexico. >>>> >>>> Left a review to DRY it up. But I don't see any critical things. >>>> >>>> - Phil >>>> >>>> On 9/22/23 15:23, Ilan Rabinovitch wrote: >>>>> Ping? Would like to wrap this up as the non-stop daily outages are no >>>>> fun, Managed to mitigate it somewhat, but given none of us are on call >>>>> getting to some reasonable sense of HA would be appreciated so folks >>>>> can go on vacation without worrying about who will reboot the servers. >>>>> >>>>> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>> >>>>>> Thanks Davide for the initial review. I think I addressed the >>>>>> remaining feedback. >>>>>> >>>>>> Anything else we want to test before merging? >>>>>> >>>>>> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I took a stab at updating the cookbooks we use for the website to use >>>>>>> centos8. Motivation was getting a new version of php so we can use >>>>>>> some drupal plugins that dont support php5.4, but i think this was on >>>>>>> the backlog already for a while. >>>>>>> >>>>>>> https://github.com/socallinuxexpo/scale-chef/pull/283 >>>>>>> >>>>>>> To test this out, I spun up scale-web2 with a clone of the production >>>>>>> scale-drupal database, and then pointed my host file at it for >>>>>>> www.socallinuxexpo.org: >>>>>>> >>>>>>> I then tested the following: >>>>>>> >>>>>>> - chef runs complete end to end >>>>>>> - Backing up / restoring static assets (see backup scripts in >>>>>>> /usr/local/bin or in the scale-drupal cookbook) >>>>>>> - browsing legacy static website (eg >>>>>>> https://socallinuxexpo.org/past/2002/ >>>>>>> https://socallinuxexpo.org/past/2003/) >>>>>>> - Registering as a speaker, confirming that I got the email and that >>>>>>> my account worked. >>>>>>> - publishing the submitted talk >>>>>>> - creating sponsors/exhibitors/blog post/events/etc >>>>>>> >>>>>>> Pending things: >>>>>>> - Centos8 seems to default to using php-fpm instead of mod_php. Not >>>>>>> opposed but it's different. Seems to work fine, but I've not load >>>>>>> tested it or anything. >>>>>>> >>>>>>> - drush (https://www.drush.org/) packages no longer seemed to be >>>>>>> present in centos or epel. remi seems to have it but the drush docs >>>>>>> suggest we should be installing this in our drupal code base via >>>>>>> composer instead of as a package. I dont have strong opinions, but we >>>>>>> should figure out the right path before merging. >>>>>>> >>>>>>> Cheers, >>>>>>> Ilan >>>>> _______________________________________________ >>>>> scale-infra mailing list >>>>> scale-infra@lists.linuxfests.org >>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>>> >>>> -- >>>> Phil Dibowitz phil@ipom.com >>>> Open Source software and tech docs Insanity Palace of Metallica >>>> http://www.phildev.net/ http://www.ipom.com/ >>>> >>>> "Be who you are and say what you feel, because those who mind don't >>>> matter and those who matter don't mind." >>>> - Dr. Seuss >>>> >>>> >>>> _______________________________________________ >>>> scale-infra mailing list >>>> scale-infra@lists.linuxfests.org >>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org >> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra > > -- > Phil Dibowitz phil@ipom.com > Open Source software and tech docs Insanity Palace of Metallica > http://www.phildev.net/ http://www.ipom.com/ > > "Be who you are and say what you feel, because those who mind don't > matter and those who matter don't mind." > - Dr. Seuss > > > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra _______________________________________________ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Phil Smith is probably the best person since it seems to be a Drupal issue, best I can tell (I scoured logs a few months back and found nothing other than what you just described: apache threads just stay around, busy, until they eat up all the memory).
I'm not sure he's on this list so I'm CC'ing hom.
Phil - I'm happy to get you root access (send me an SSH pubkey), or anything else you may not already have to help you debug. I spent quite a while last SCALE trying to figure this out, Ilan has spent time on this, and it seems to be drupal-specific, best I can tell.
On 10/5/23 14:49, Ilan Rabinovitch wrote:
Theres definitely something continuing to go on with drupal and/or httpd. Something continues to eat up all the worker threads from httpd. Before we set maxworkers to something reasonable it would eat up all the memory, now it just consumes all the workers and the site becomes unresponsive. I dont think this relates to the upgrade, as it was happening before.
We've had the site down three times in the past 2 days as a result. Monitoring detected it and I was able to respond to it fairly quickly, but I'm about to board a 30 hour flight, so I'm likely not going to be able to respond moving forward.
Given the importance of the CFP this month I hope someone has a chance to dig in.
On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz phil@ipom.com wrote:
Copy that, I'll keep an eye out. Nice work, and thanks!
On 10/3/23 18:06, Ilan Rabinovitch wrote:
This is done. Here's the specific ids for any roll backs:
- start scale-web1 up again (i-017df11978f41d9ae)
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ from
scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae) 3) move EIP (23.21.71.118) from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.
Assuming no issues the next change to think about is:
- enable the s3fs plugin work Phil Smith did recently so static files
aren't stored on box anymore.
- move scale-web2 behind an ELB and auto scaling group.
- remove the cron jobs that back up static files
This will enable some reasonable failover that doesn't require human intervention.
On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday.
On 10/3/23 16:08, Ilan Rabinovitch wrote:
Cutover:
- RDS Database snapshot
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
hosts in case users have uploaded static assets
- Move EIP from scale-web1 host to scale-web2
Rollback:
- re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on
both hosts in case users have uploaded static assets
- Move EIP from scale-web2 host to scale-web1
On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com wrote:
Feel free to do it today.
What's the migrate/failback process? It's somewhere in cloudflair?
If not I can do it next week.
On 10/3/23 11:09, Ilan Rabinovitch wrote: > Sounds like we should do this today then, or you can grab next week. > I'm going to be offline until November after Thursday morning. > > On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote: >> >> I'm out of town Fri-Sun, and will only be around a laptop sporadically. >> I'm in town all week next week though. >> >> >> >> On 10/2/23 13:22, Ilan Rabinovitch wrote: >>> Phil Smith, myself and a few others have banged on scale-web2 and we >>> think we're all set to do a cutover there. >>> However, I'm going to be out of town for a few weeks starting Thursday >>> and do not yet know what my connectivity will be like. >>> >>> Would love to cutover this week, but only want to do it if others will >>> be available to roll back to the old server should something go awry >>> while I'm on vacation. >>> >>> On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch >>> ilan@socallinuxexpo.org wrote: >>>> >>>> I believe Ive addressed everyone's feedback. >>>> >>>> scale-web2 is online on running off the scale_web_centos8 branch. >>>> it's using a test db running from a snapshot of prod 2~ weeks ago. >>>> you can change it as you see fit without worrying about impacting production. >>>> >>>> To test you'll need to update /etc/hosts to point at the new server. >>>> following to your /etc/hosts file. >>>> >>>> 54.205.247.164 www.socallinuxexpo.org >>>> >>>> Ideally would like to merge and migrate product this week >>>> >>>> >>>> On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote: >>>>> >>>>> Sorry I was in Mexico. >>>>> >>>>> Left a review to DRY it up. But I don't see any critical things. >>>>> >>>>> - Phil >>>>> >>>>> On 9/22/23 15:23, Ilan Rabinovitch wrote: >>>>>> Ping? Would like to wrap this up as the non-stop daily outages are no >>>>>> fun, Managed to mitigate it somewhat, but given none of us are on call >>>>>> getting to some reasonable sense of HA would be appreciated so folks >>>>>> can go on vacation without worrying about who will reboot the servers. >>>>>> >>>>>> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>>> >>>>>>> Thanks Davide for the initial review. I think I addressed the >>>>>>> remaining feedback. >>>>>>> >>>>>>> Anything else we want to test before merging? >>>>>>> >>>>>>> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I took a stab at updating the cookbooks we use for the website to use >>>>>>>> centos8. Motivation was getting a new version of php so we can use >>>>>>>> some drupal plugins that dont support php5.4, but i think this was on >>>>>>>> the backlog already for a while. >>>>>>>> >>>>>>>> https://github.com/socallinuxexpo/scale-chef/pull/283 >>>>>>>> >>>>>>>> To test this out, I spun up scale-web2 with a clone of the production >>>>>>>> scale-drupal database, and then pointed my host file at it for >>>>>>>> www.socallinuxexpo.org: >>>>>>>> >>>>>>>> I then tested the following: >>>>>>>> >>>>>>>> - chef runs complete end to end >>>>>>>> - Backing up / restoring static assets (see backup scripts in >>>>>>>> /usr/local/bin or in the scale-drupal cookbook) >>>>>>>> - browsing legacy static website (eg >>>>>>>> https://socallinuxexpo.org/past/2002/ >>>>>>>> https://socallinuxexpo.org/past/2003/) >>>>>>>> - Registering as a speaker, confirming that I got the email and that >>>>>>>> my account worked. >>>>>>>> - publishing the submitted talk >>>>>>>> - creating sponsors/exhibitors/blog post/events/etc >>>>>>>> >>>>>>>> Pending things: >>>>>>>> - Centos8 seems to default to using php-fpm instead of mod_php. Not >>>>>>>> opposed but it's different. Seems to work fine, but I've not load >>>>>>>> tested it or anything. >>>>>>>> >>>>>>>> - drush (https://www.drush.org/) packages no longer seemed to be >>>>>>>> present in centos or epel. remi seems to have it but the drush docs >>>>>>>> suggest we should be installing this in our drupal code base via >>>>>>>> composer instead of as a package. I dont have strong opinions, but we >>>>>>>> should figure out the right path before merging. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Ilan >>>>>> _______________________________________________ >>>>>> scale-infra mailing list >>>>>> scale-infra@lists.linuxfests.org >>>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>>>> >>>>> -- >>>>> Phil Dibowitz phil@ipom.com >>>>> Open Source software and tech docs Insanity Palace of Metallica >>>>> http://www.phildev.net/ http://www.ipom.com/ >>>>> >>>>> "Be who you are and say what you feel, because those who mind don't >>>>> matter and those who matter don't mind." >>>>> - Dr. Seuss >>>>> >>>>> >>>>> _______________________________________________ >>>>> scale-infra mailing list >>>>> scale-infra@lists.linuxfests.org >>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>> _______________________________________________ >>> scale-infra mailing list >>> scale-infra@lists.linuxfests.org >>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >> >> -- >> Phil Dibowitz phil@ipom.com >> Open Source software and tech docs Insanity Palace of Metallica >> http://www.phildev.net/ http://www.ipom.com/ >> >> "Be who you are and say what you feel, because those who mind don't >> matter and those who matter don't mind." >> - Dr. Seuss >> >> >> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org >> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Phil already has ssh on systems, but I dont think we have any reason to believe its drupal at this point more than we believe its httpd configs or something else . I'm also not sure Phil has enough knowledge of our chef setup to push any changes out should he find a reason.
On Thu, Oct 5, 2023 at 6:11 PM Phil Dibowitz phil@ipom.com wrote:
Phil Smith is probably the best person since it seems to be a Drupal issue, best I can tell (I scoured logs a few months back and found nothing other than what you just described: apache threads just stay around, busy, until they eat up all the memory).
I'm not sure he's on this list so I'm CC'ing hom.
Phil - I'm happy to get you root access (send me an SSH pubkey), or anything else you may not already have to help you debug. I spent quite a while last SCALE trying to figure this out, Ilan has spent time on this, and it seems to be drupal-specific, best I can tell.
On 10/5/23 14:49, Ilan Rabinovitch wrote:
Theres definitely something continuing to go on with drupal and/or httpd. Something continues to eat up all the worker threads from httpd. Before we set maxworkers to something reasonable it would eat up all the memory, now it just consumes all the workers and the site becomes unresponsive. I dont think this relates to the upgrade, as it was happening before.
We've had the site down three times in the past 2 days as a result. Monitoring detected it and I was able to respond to it fairly quickly, but I'm about to board a 30 hour flight, so I'm likely not going to be able to respond moving forward.
Given the importance of the CFP this month I hope someone has a chance to dig in.
On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz phil@ipom.com wrote:
Copy that, I'll keep an eye out. Nice work, and thanks!
On 10/3/23 18:06, Ilan Rabinovitch wrote:
This is done. Here's the specific ids for any roll backs:
- start scale-web1 up again (i-017df11978f41d9ae)
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ from
scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae) 3) move EIP (23.21.71.118) from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.
Assuming no issues the next change to think about is:
- enable the s3fs plugin work Phil Smith did recently so static files
aren't stored on box anymore.
- move scale-web2 behind an ELB and auto scaling group.
- remove the cron jobs that back up static files
This will enable some reasonable failover that doesn't require human intervention.
On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday.
On 10/3/23 16:08, Ilan Rabinovitch wrote:
Cutover:
- RDS Database snapshot
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
hosts in case users have uploaded static assets
- Move EIP from scale-web1 host to scale-web2
Rollback:
- re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on
both hosts in case users have uploaded static assets
- Move EIP from scale-web2 host to scale-web1
On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com wrote: > > Feel free to do it today. > > What's the migrate/failback process? It's somewhere in cloudflair? > > If not I can do it next week. > > > On 10/3/23 11:09, Ilan Rabinovitch wrote: >> Sounds like we should do this today then, or you can grab next week. >> I'm going to be offline until November after Thursday morning. >> >> On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote: >>> >>> I'm out of town Fri-Sun, and will only be around a laptop sporadically. >>> I'm in town all week next week though. >>> >>> >>> >>> On 10/2/23 13:22, Ilan Rabinovitch wrote: >>>> Phil Smith, myself and a few others have banged on scale-web2 and we >>>> think we're all set to do a cutover there. >>>> However, I'm going to be out of town for a few weeks starting Thursday >>>> and do not yet know what my connectivity will be like. >>>> >>>> Would love to cutover this week, but only want to do it if others will >>>> be available to roll back to the old server should something go awry >>>> while I'm on vacation. >>>> >>>> On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch >>>> ilan@socallinuxexpo.org wrote: >>>>> >>>>> I believe Ive addressed everyone's feedback. >>>>> >>>>> scale-web2 is online on running off the scale_web_centos8 branch. >>>>> it's using a test db running from a snapshot of prod 2~ weeks ago. >>>>> you can change it as you see fit without worrying about impacting production. >>>>> >>>>> To test you'll need to update /etc/hosts to point at the new server. >>>>> following to your /etc/hosts file. >>>>> >>>>> 54.205.247.164 www.socallinuxexpo.org >>>>> >>>>> Ideally would like to merge and migrate product this week >>>>> >>>>> >>>>> On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote: >>>>>> >>>>>> Sorry I was in Mexico. >>>>>> >>>>>> Left a review to DRY it up. But I don't see any critical things. >>>>>> >>>>>> - Phil >>>>>> >>>>>> On 9/22/23 15:23, Ilan Rabinovitch wrote: >>>>>>> Ping? Would like to wrap this up as the non-stop daily outages are no >>>>>>> fun, Managed to mitigate it somewhat, but given none of us are on call >>>>>>> getting to some reasonable sense of HA would be appreciated so folks >>>>>>> can go on vacation without worrying about who will reboot the servers. >>>>>>> >>>>>>> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>>>> >>>>>>>> Thanks Davide for the initial review. I think I addressed the >>>>>>>> remaining feedback. >>>>>>>> >>>>>>>> Anything else we want to test before merging? >>>>>>>> >>>>>>>> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I took a stab at updating the cookbooks we use for the website to use >>>>>>>>> centos8. Motivation was getting a new version of php so we can use >>>>>>>>> some drupal plugins that dont support php5.4, but i think this was on >>>>>>>>> the backlog already for a while. >>>>>>>>> >>>>>>>>> https://github.com/socallinuxexpo/scale-chef/pull/283 >>>>>>>>> >>>>>>>>> To test this out, I spun up scale-web2 with a clone of the production >>>>>>>>> scale-drupal database, and then pointed my host file at it for >>>>>>>>> www.socallinuxexpo.org: >>>>>>>>> >>>>>>>>> I then tested the following: >>>>>>>>> >>>>>>>>> - chef runs complete end to end >>>>>>>>> - Backing up / restoring static assets (see backup scripts in >>>>>>>>> /usr/local/bin or in the scale-drupal cookbook) >>>>>>>>> - browsing legacy static website (eg >>>>>>>>> https://socallinuxexpo.org/past/2002/ >>>>>>>>> https://socallinuxexpo.org/past/2003/) >>>>>>>>> - Registering as a speaker, confirming that I got the email and that >>>>>>>>> my account worked. >>>>>>>>> - publishing the submitted talk >>>>>>>>> - creating sponsors/exhibitors/blog post/events/etc >>>>>>>>> >>>>>>>>> Pending things: >>>>>>>>> - Centos8 seems to default to using php-fpm instead of mod_php. Not >>>>>>>>> opposed but it's different. Seems to work fine, but I've not load >>>>>>>>> tested it or anything. >>>>>>>>> >>>>>>>>> - drush (https://www.drush.org/) packages no longer seemed to be >>>>>>>>> present in centos or epel. remi seems to have it but the drush docs >>>>>>>>> suggest we should be installing this in our drupal code base via >>>>>>>>> composer instead of as a package. I dont have strong opinions, but we >>>>>>>>> should figure out the right path before merging. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Ilan >>>>>>> _______________________________________________ >>>>>>> scale-infra mailing list >>>>>>> scale-infra@lists.linuxfests.org >>>>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>>>>> >>>>>> -- >>>>>> Phil Dibowitz phil@ipom.com >>>>>> Open Source software and tech docs Insanity Palace of Metallica >>>>>> http://www.phildev.net/ http://www.ipom.com/ >>>>>> >>>>>> "Be who you are and say what you feel, because those who mind don't >>>>>> matter and those who matter don't mind." >>>>>> - Dr. Seuss >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> scale-infra mailing list >>>>>> scale-infra@lists.linuxfests.org >>>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>>> _______________________________________________ >>>> scale-infra mailing list >>>> scale-infra@lists.linuxfests.org >>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>> >>> -- >>> Phil Dibowitz phil@ipom.com >>> Open Source software and tech docs Insanity Palace of Metallica >>> http://www.phildev.net/ http://www.ipom.com/ >>> >>> "Be who you are and say what you feel, because those who mind don't >>> matter and those who matter don't mind." >>> - Dr. Seuss >>> >>> >>> _______________________________________________ >>> scale-infra mailing list >>> scale-infra@lists.linuxfests.org >>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org >> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra > > -- > Phil Dibowitz phil@ipom.com > Open Source software and tech docs Insanity Palace of Metallica > http://www.phildev.net/ http://www.ipom.com/ > > "Be who you are and say what you feel, because those who mind don't > matter and those who matter don't mind." > - Dr. Seuss > > > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra _______________________________________________ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
On 10/5/23 15:17, Ilan Rabinovitch wrote:
Phil already has ssh on systems, but I dont think we have any reason to believe its drupal at this point more than we believe its httpd configs or something else . I'm also not sure Phil has enough knowledge of our chef setup to push any changes out should he find a reason.
Our Apache configs are pretty stock. And they haven't changed in years and years and this only started happening 2 years ago or so.
I'm happy to walk him through pausing chef to test manual changes, and/or make Chef PRs for him, if that's what needs doing.
My guess is that it's Drupal itself is holding a connection somewhere (to a DB? to a client it thinks is still around? unclear...), holding threads open forever. But it's totally a guess.
(resending with offlist folks)
Looks like we had another outage this morning. I'm online for the next 45-60 min if someone wants to chat about it. Otherwise, I would greatly appreciate it if someone on the infra team could keep an eye on monitoring and at least restart things.
On Fri, Oct 6, 2023 at 1:11 AM Phil Dibowitz phil@ipom.com wrote:
Phil Smith is probably the best person since it seems to be a Drupal issue, best I can tell (I scoured logs a few months back and found nothing other than what you just described: apache threads just stay around, busy, until they eat up all the memory).
I'm not sure he's on this list so I'm CC'ing hom.
Phil - I'm happy to get you root access (send me an SSH pubkey), or anything else you may not already have to help you debug. I spent quite a while last SCALE trying to figure this out, Ilan has spent time on this, and it seems to be drupal-specific, best I can tell.
On 10/5/23 14:49, Ilan Rabinovitch wrote:
Theres definitely something continuing to go on with drupal and/or httpd. Something continues to eat up all the worker threads from httpd. Before we set maxworkers to something reasonable it would eat up all the memory, now it just consumes all the workers and the site becomes unresponsive. I dont think this relates to the upgrade, as it was happening before.
We've had the site down three times in the past 2 days as a result. Monitoring detected it and I was able to respond to it fairly quickly, but I'm about to board a 30 hour flight, so I'm likely not going to be able to respond moving forward.
Given the importance of the CFP this month I hope someone has a chance to dig in.
On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz phil@ipom.com wrote:
Copy that, I'll keep an eye out. Nice work, and thanks!
On 10/3/23 18:06, Ilan Rabinovitch wrote:
This is done. Here's the specific ids for any roll backs:
- start scale-web1 up again (i-017df11978f41d9ae)
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ from
scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae) 3) move EIP (23.21.71.118) from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.
Assuming no issues the next change to think about is:
- enable the s3fs plugin work Phil Smith did recently so static files
aren't stored on box anymore.
- move scale-web2 behind an ELB and auto scaling group.
- remove the cron jobs that back up static files
This will enable some reasonable failover that doesn't require human intervention.
On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday.
On 10/3/23 16:08, Ilan Rabinovitch wrote:
Cutover:
- RDS Database snapshot
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
hosts in case users have uploaded static assets
- Move EIP from scale-web1 host to scale-web2
Rollback:
- re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on
both hosts in case users have uploaded static assets
- Move EIP from scale-web2 host to scale-web1
On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com wrote: > > Feel free to do it today. > > What's the migrate/failback process? It's somewhere in cloudflair? > > If not I can do it next week. > > > On 10/3/23 11:09, Ilan Rabinovitch wrote: >> Sounds like we should do this today then, or you can grab next week. >> I'm going to be offline until November after Thursday morning. >> >> On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote: >>> >>> I'm out of town Fri-Sun, and will only be around a laptop sporadically. >>> I'm in town all week next week though. >>> >>> >>> >>> On 10/2/23 13:22, Ilan Rabinovitch wrote: >>>> Phil Smith, myself and a few others have banged on scale-web2 and we >>>> think we're all set to do a cutover there. >>>> However, I'm going to be out of town for a few weeks starting Thursday >>>> and do not yet know what my connectivity will be like. >>>> >>>> Would love to cutover this week, but only want to do it if others will >>>> be available to roll back to the old server should something go awry >>>> while I'm on vacation. >>>> >>>> On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch >>>> ilan@socallinuxexpo.org wrote: >>>>> >>>>> I believe Ive addressed everyone's feedback. >>>>> >>>>> scale-web2 is online on running off the scale_web_centos8 branch. >>>>> it's using a test db running from a snapshot of prod 2~ weeks ago. >>>>> you can change it as you see fit without worrying about impacting production. >>>>> >>>>> To test you'll need to update /etc/hosts to point at the new server. >>>>> following to your /etc/hosts file. >>>>> >>>>> 54.205.247.164 www.socallinuxexpo.org >>>>> >>>>> Ideally would like to merge and migrate product this week >>>>> >>>>> >>>>> On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote: >>>>>> >>>>>> Sorry I was in Mexico. >>>>>> >>>>>> Left a review to DRY it up. But I don't see any critical things. >>>>>> >>>>>> - Phil >>>>>> >>>>>> On 9/22/23 15:23, Ilan Rabinovitch wrote: >>>>>>> Ping? Would like to wrap this up as the non-stop daily outages are no >>>>>>> fun, Managed to mitigate it somewhat, but given none of us are on call >>>>>>> getting to some reasonable sense of HA would be appreciated so folks >>>>>>> can go on vacation without worrying about who will reboot the servers. >>>>>>> >>>>>>> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>>>> >>>>>>>> Thanks Davide for the initial review. I think I addressed the >>>>>>>> remaining feedback. >>>>>>>> >>>>>>>> Anything else we want to test before merging? >>>>>>>> >>>>>>>> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I took a stab at updating the cookbooks we use for the website to use >>>>>>>>> centos8. Motivation was getting a new version of php so we can use >>>>>>>>> some drupal plugins that dont support php5.4, but i think this was on >>>>>>>>> the backlog already for a while. >>>>>>>>> >>>>>>>>> https://github.com/socallinuxexpo/scale-chef/pull/283 >>>>>>>>> >>>>>>>>> To test this out, I spun up scale-web2 with a clone of the production >>>>>>>>> scale-drupal database, and then pointed my host file at it for >>>>>>>>> www.socallinuxexpo.org: >>>>>>>>> >>>>>>>>> I then tested the following: >>>>>>>>> >>>>>>>>> - chef runs complete end to end >>>>>>>>> - Backing up / restoring static assets (see backup scripts in >>>>>>>>> /usr/local/bin or in the scale-drupal cookbook) >>>>>>>>> - browsing legacy static website (eg >>>>>>>>> https://socallinuxexpo.org/past/2002/ >>>>>>>>> https://socallinuxexpo.org/past/2003/) >>>>>>>>> - Registering as a speaker, confirming that I got the email and that >>>>>>>>> my account worked. >>>>>>>>> - publishing the submitted talk >>>>>>>>> - creating sponsors/exhibitors/blog post/events/etc >>>>>>>>> >>>>>>>>> Pending things: >>>>>>>>> - Centos8 seems to default to using php-fpm instead of mod_php. Not >>>>>>>>> opposed but it's different. Seems to work fine, but I've not load >>>>>>>>> tested it or anything. >>>>>>>>> >>>>>>>>> - drush (https://www.drush.org/) packages no longer seemed to be >>>>>>>>> present in centos or epel. remi seems to have it but the drush docs >>>>>>>>> suggest we should be installing this in our drupal code base via >>>>>>>>> composer instead of as a package. I dont have strong opinions, but we >>>>>>>>> should figure out the right path before merging. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> Ilan >>>>>>> _______________________________________________ >>>>>>> scale-infra mailing list >>>>>>> scale-infra@lists.linuxfests.org >>>>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>>>>> >>>>>> -- >>>>>> Phil Dibowitz phil@ipom.com >>>>>> Open Source software and tech docs Insanity Palace of Metallica >>>>>> http://www.phildev.net/ http://www.ipom.com/ >>>>>> >>>>>> "Be who you are and say what you feel, because those who mind don't >>>>>> matter and those who matter don't mind." >>>>>> - Dr. Seuss >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> scale-infra mailing list >>>>>> scale-infra@lists.linuxfests.org >>>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>>> _______________________________________________ >>>> scale-infra mailing list >>>> scale-infra@lists.linuxfests.org >>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>> >>> -- >>> Phil Dibowitz phil@ipom.com >>> Open Source software and tech docs Insanity Palace of Metallica >>> http://www.phildev.net/ http://www.ipom.com/ >>> >>> "Be who you are and say what you feel, because those who mind don't >>> matter and those who matter don't mind." >>> - Dr. Seuss >>> >>> >>> _______________________________________________ >>> scale-infra mailing list >>> scale-infra@lists.linuxfests.org >>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org >> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra > > -- > Phil Dibowitz phil@ipom.com > Open Source software and tech docs Insanity Palace of Metallica > http://www.phildev.net/ http://www.ipom.com/ > > "Be who you are and say what you feel, because those who mind don't > matter and those who matter don't mind." > - Dr. Seuss > > > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra _______________________________________________ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Looks like we were down again for a few hours this morning.
On Fri, Oct 6, 2023 at 10:03 PM Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
(resending with offlist folks)
Looks like we had another outage this morning. I'm online for the next 45-60 min if someone wants to chat about it. Otherwise, I would greatly appreciate it if someone on the infra team could keep an eye on monitoring and at least restart things.
On Fri, Oct 6, 2023 at 1:11 AM Phil Dibowitz phil@ipom.com wrote:
Phil Smith is probably the best person since it seems to be a Drupal issue, best I can tell (I scoured logs a few months back and found nothing other than what you just described: apache threads just stay around, busy, until they eat up all the memory).
I'm not sure he's on this list so I'm CC'ing hom.
Phil - I'm happy to get you root access (send me an SSH pubkey), or anything else you may not already have to help you debug. I spent quite a while last SCALE trying to figure this out, Ilan has spent time on this, and it seems to be drupal-specific, best I can tell.
On 10/5/23 14:49, Ilan Rabinovitch wrote:
Theres definitely something continuing to go on with drupal and/or httpd. Something continues to eat up all the worker threads from httpd. Before we set maxworkers to something reasonable it would eat up all the memory, now it just consumes all the workers and the site becomes unresponsive. I dont think this relates to the upgrade, as it was happening before.
We've had the site down three times in the past 2 days as a result. Monitoring detected it and I was able to respond to it fairly quickly, but I'm about to board a 30 hour flight, so I'm likely not going to be able to respond moving forward.
Given the importance of the CFP this month I hope someone has a chance to dig in.
On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz phil@ipom.com wrote:
Copy that, I'll keep an eye out. Nice work, and thanks!
On 10/3/23 18:06, Ilan Rabinovitch wrote:
This is done. Here's the specific ids for any roll backs:
- start scale-web1 up again (i-017df11978f41d9ae)
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ from
scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae) 3) move EIP (23.21.71.118) from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.
Assuming no issues the next change to think about is:
- enable the s3fs plugin work Phil Smith did recently so static files
aren't stored on box anymore.
- move scale-web2 behind an ELB and auto scaling group.
- remove the cron jobs that back up static files
This will enable some reasonable failover that doesn't require human intervention.
On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday.
On 10/3/23 16:08, Ilan Rabinovitch wrote: > Cutover: > - RDS Database snapshot > - sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both > hosts in case users have uploaded static assets > - Move EIP from scale-web1 host to scale-web2 > > Rollback: > - re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on > both hosts in case users have uploaded static assets > - Move EIP from scale-web2 host to scale-web1 > > > > > > On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com wrote: >> >> Feel free to do it today. >> >> What's the migrate/failback process? It's somewhere in cloudflair? >> >> If not I can do it next week. >> >> >> On 10/3/23 11:09, Ilan Rabinovitch wrote: >>> Sounds like we should do this today then, or you can grab next week. >>> I'm going to be offline until November after Thursday morning. >>> >>> On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote: >>>> >>>> I'm out of town Fri-Sun, and will only be around a laptop sporadically. >>>> I'm in town all week next week though. >>>> >>>> >>>> >>>> On 10/2/23 13:22, Ilan Rabinovitch wrote: >>>>> Phil Smith, myself and a few others have banged on scale-web2 and we >>>>> think we're all set to do a cutover there. >>>>> However, I'm going to be out of town for a few weeks starting Thursday >>>>> and do not yet know what my connectivity will be like. >>>>> >>>>> Would love to cutover this week, but only want to do it if others will >>>>> be available to roll back to the old server should something go awry >>>>> while I'm on vacation. >>>>> >>>>> On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch >>>>> ilan@socallinuxexpo.org wrote: >>>>>> >>>>>> I believe Ive addressed everyone's feedback. >>>>>> >>>>>> scale-web2 is online on running off the scale_web_centos8 branch. >>>>>> it's using a test db running from a snapshot of prod 2~ weeks ago. >>>>>> you can change it as you see fit without worrying about impacting production. >>>>>> >>>>>> To test you'll need to update /etc/hosts to point at the new server. >>>>>> following to your /etc/hosts file. >>>>>> >>>>>> 54.205.247.164 www.socallinuxexpo.org >>>>>> >>>>>> Ideally would like to merge and migrate product this week >>>>>> >>>>>> >>>>>> On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote: >>>>>>> >>>>>>> Sorry I was in Mexico. >>>>>>> >>>>>>> Left a review to DRY it up. But I don't see any critical things. >>>>>>> >>>>>>> - Phil >>>>>>> >>>>>>> On 9/22/23 15:23, Ilan Rabinovitch wrote: >>>>>>>> Ping? Would like to wrap this up as the non-stop daily outages are no >>>>>>>> fun, Managed to mitigate it somewhat, but given none of us are on call >>>>>>>> getting to some reasonable sense of HA would be appreciated so folks >>>>>>>> can go on vacation without worrying about who will reboot the servers. >>>>>>>> >>>>>>>> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>>>>> >>>>>>>>> Thanks Davide for the initial review. I think I addressed the >>>>>>>>> remaining feedback. >>>>>>>>> >>>>>>>>> Anything else we want to test before merging? >>>>>>>>> >>>>>>>>> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I took a stab at updating the cookbooks we use for the website to use >>>>>>>>>> centos8. Motivation was getting a new version of php so we can use >>>>>>>>>> some drupal plugins that dont support php5.4, but i think this was on >>>>>>>>>> the backlog already for a while. >>>>>>>>>> >>>>>>>>>> https://github.com/socallinuxexpo/scale-chef/pull/283 >>>>>>>>>> >>>>>>>>>> To test this out, I spun up scale-web2 with a clone of the production >>>>>>>>>> scale-drupal database, and then pointed my host file at it for >>>>>>>>>> www.socallinuxexpo.org: >>>>>>>>>> >>>>>>>>>> I then tested the following: >>>>>>>>>> >>>>>>>>>> - chef runs complete end to end >>>>>>>>>> - Backing up / restoring static assets (see backup scripts in >>>>>>>>>> /usr/local/bin or in the scale-drupal cookbook) >>>>>>>>>> - browsing legacy static website (eg >>>>>>>>>> https://socallinuxexpo.org/past/2002/ >>>>>>>>>> https://socallinuxexpo.org/past/2003/) >>>>>>>>>> - Registering as a speaker, confirming that I got the email and that >>>>>>>>>> my account worked. >>>>>>>>>> - publishing the submitted talk >>>>>>>>>> - creating sponsors/exhibitors/blog post/events/etc >>>>>>>>>> >>>>>>>>>> Pending things: >>>>>>>>>> - Centos8 seems to default to using php-fpm instead of mod_php. Not >>>>>>>>>> opposed but it's different. Seems to work fine, but I've not load >>>>>>>>>> tested it or anything. >>>>>>>>>> >>>>>>>>>> - drush (https://www.drush.org/) packages no longer seemed to be >>>>>>>>>> present in centos or epel. remi seems to have it but the drush docs >>>>>>>>>> suggest we should be installing this in our drupal code base via >>>>>>>>>> composer instead of as a package. I dont have strong opinions, but we >>>>>>>>>> should figure out the right path before merging. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Ilan >>>>>>>> _______________________________________________ >>>>>>>> scale-infra mailing list >>>>>>>> scale-infra@lists.linuxfests.org >>>>>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>>>>>> >>>>>>> -- >>>>>>> Phil Dibowitz phil@ipom.com >>>>>>> Open Source software and tech docs Insanity Palace of Metallica >>>>>>> http://www.phildev.net/ http://www.ipom.com/ >>>>>>> >>>>>>> "Be who you are and say what you feel, because those who mind don't >>>>>>> matter and those who matter don't mind." >>>>>>> - Dr. Seuss >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> scale-infra mailing list >>>>>>> scale-infra@lists.linuxfests.org >>>>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>>>> _______________________________________________ >>>>> scale-infra mailing list >>>>> scale-infra@lists.linuxfests.org >>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>>> >>>> -- >>>> Phil Dibowitz phil@ipom.com >>>> Open Source software and tech docs Insanity Palace of Metallica >>>> http://www.phildev.net/ http://www.ipom.com/ >>>> >>>> "Be who you are and say what you feel, because those who mind don't >>>> matter and those who matter don't mind." >>>> - Dr. Seuss >>>> >>>> >>>> _______________________________________________ >>>> scale-infra mailing list >>>> scale-infra@lists.linuxfests.org >>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>> _______________________________________________ >>> scale-infra mailing list >>> scale-infra@lists.linuxfests.org >>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >> >> -- >> Phil Dibowitz phil@ipom.com >> Open Source software and tech docs Insanity Palace of Metallica >> http://www.phildev.net/ http://www.ipom.com/ >> >> "Be who you are and say what you feel, because those who mind don't >> matter and those who matter don't mind." >> - Dr. Seuss >> >> >> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org >> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Restarted. Added cron to restart. Will find time to pair debug with Smith this week.
Get BlueMail for Android
On Oct 7, 2023, 1:50 AM, at 1:50 AM, Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
Looks like we were down again for a few hours this morning.
On Fri, Oct 6, 2023 at 10:03 PM Ilan Rabinovitch ilan@socallinuxexpo.org wrote:
(resending with offlist folks)
Looks like we had another outage this morning. I'm online for the
next
45-60 min if someone wants to chat about it. Otherwise, I would greatly appreciate it if someone on the infra team could keep an eye on monitoring and at least restart things.
On Fri, Oct 6, 2023 at 1:11 AM Phil Dibowitz phil@ipom.com wrote:
Phil Smith is probably the best person since it seems to be a
Drupal
issue, best I can tell (I scoured logs a few months back and found nothing other than what you just described: apache threads just
stay
around, busy, until they eat up all the memory).
I'm not sure he's on this list so I'm CC'ing hom.
Phil - I'm happy to get you root access (send me an SSH pubkey), or anything else you may not already have to help you debug. I spent
quite
a while last SCALE trying to figure this out, Ilan has spent time
on
this, and it seems to be drupal-specific, best I can tell.
On 10/5/23 14:49, Ilan Rabinovitch wrote:
Theres definitely something continuing to go on with drupal
and/or
httpd. Something continues to eat up all the worker threads from httpd. Before we set maxworkers to something reasonable it would
eat
up all the memory, now it just consumes all the workers and the
site
becomes unresponsive. I dont think this relates to the upgrade, as it was happening
before.
We've had the site down three times in the past 2 days as a
result.
Monitoring detected it and I was able to respond to it fairly
quickly,
but I'm about to board a 30 hour flight, so I'm likely not going
to be
able to respond moving forward.
Given the importance of the CFP this month I hope someone has a
chance
to dig in.
On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz phil@ipom.com
wrote:
Copy that, I'll keep an eye out. Nice work, and thanks!
On 10/3/23 18:06, Ilan Rabinovitch wrote:
This is done. Here's the specific ids for any roll backs:
- start scale-web1 up again (i-017df11978f41d9ae)
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/
from
scale-web2 (i-0bf1dd453774f2c3b) to scale-web1
(i-017df11978f41d9ae)
- move EIP (23.21.71.118) from scale-web2
(i-0bf1dd453774f2c3b) to scale-web1.
Assuming no issues the next change to think about is:
- enable the s3fs plugin work Phil Smith did recently so static
files
aren't stored on box anymore.
- move scale-web2 behind an ELB and auto scaling group.
- remove the cron jobs that back up static files
This will enable some reasonable failover that doesn't require
human
intervention.
On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com
wrote:
> > Easy-peasey. Feel free to do it as your leisure today. I have
a few
> errands to run, but will never be more than 30m from home and
will be
> able to roll back today, and will be around all day tomorrow
and Thursday.
> > > On 10/3/23 16:08, Ilan Rabinovitch wrote: >> Cutover: >> - RDS Database snapshot >> - sync
/home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
>> hosts in case users have uploaded static assets >> - Move EIP from scale-web1 host to scale-web2 >> >> Rollback: >> - re-sync
/home/drupal/scale-drupal/httpdocs/sites/default/files/ on
>> both hosts in case users have uploaded static assets >> - Move EIP from scale-web2 host to scale-web1 >> >> >> >> >> >> On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com
wrote:
>>> >>> Feel free to do it today. >>> >>> What's the migrate/failback process? It's somewhere in
cloudflair?
>>> >>> If not I can do it next week. >>> >>> >>> On 10/3/23 11:09, Ilan Rabinovitch wrote: >>>> Sounds like we should do this today then, or you can grab
next week.
>>>> I'm going to be offline until November after Thursday
morning.
>>>> >>>> On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz
phil@ipom.com wrote:
>>>>> >>>>> I'm out of town Fri-Sun, and will only be around a laptop
sporadically.
>>>>> I'm in town all week next week though. >>>>> >>>>> >>>>> >>>>> On 10/2/23 13:22, Ilan Rabinovitch wrote: >>>>>> Phil Smith, myself and a few others have banged on
scale-web2 and we
>>>>>> think we're all set to do a cutover there. >>>>>> However, I'm going to be out of town for a few weeks
starting Thursday
>>>>>> and do not yet know what my connectivity will be like. >>>>>> >>>>>> Would love to cutover this week, but only want to do it
if others will
>>>>>> be available to roll back to the old server should
something go awry
>>>>>> while I'm on vacation. >>>>>> >>>>>> On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch >>>>>> ilan@socallinuxexpo.org wrote: >>>>>>> >>>>>>> I believe Ive addressed everyone's feedback. >>>>>>> >>>>>>> scale-web2 is online on running off the
scale_web_centos8 branch.
>>>>>>> it's using a test db running from a snapshot of prod 2~
weeks ago.
>>>>>>> you can change it as you see fit without worrying about
impacting production.
>>>>>>> >>>>>>> To test you'll need to update /etc/hosts to point at the
new server.
>>>>>>> following to your /etc/hosts file. >>>>>>> >>>>>>> 54.205.247.164 www.socallinuxexpo.org >>>>>>> >>>>>>> Ideally would like to merge and migrate product this
week
>>>>>>> >>>>>>> >>>>>>> On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz
phil@ipom.com wrote:
>>>>>>>> >>>>>>>> Sorry I was in Mexico. >>>>>>>> >>>>>>>> Left a review to DRY it up. But I don't see any
critical things.
>>>>>>>> >>>>>>>> - Phil >>>>>>>> >>>>>>>> On 9/22/23 15:23, Ilan Rabinovitch wrote: >>>>>>>>> Ping? Would like to wrap this up as the non-stop daily
outages are no
>>>>>>>>> fun, Managed to mitigate it somewhat, but given none
of us are on call
>>>>>>>>> getting to some reasonable sense of HA would be
appreciated so folks
>>>>>>>>> can go on vacation without worrying about who will
reboot the servers.
>>>>>>>>> >>>>>>>>> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch
ilan@linuxfests.org wrote:
>>>>>>>>>> >>>>>>>>>> Thanks Davide for the initial review. I think I
addressed the
>>>>>>>>>> remaining feedback. >>>>>>>>>> >>>>>>>>>> Anything else we want to test before merging? >>>>>>>>>> >>>>>>>>>> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch
ilan@linuxfests.org wrote:
>>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I took a stab at updating the cookbooks we use for
the website to use
>>>>>>>>>>> centos8. Motivation was getting a new version of php
so we can use
>>>>>>>>>>> some drupal plugins that dont support php5.4, but i
think this was on
>>>>>>>>>>> the backlog already for a while. >>>>>>>>>>> >>>>>>>>>>>
https://github.com/socallinuxexpo/scale-chef/pull/283
>>>>>>>>>>> >>>>>>>>>>> To test this out, I spun up scale-web2 with a clone
of the production
>>>>>>>>>>> scale-drupal database, and then pointed my host file
at it for
>>>>>>>>>>> www.socallinuxexpo.org: >>>>>>>>>>> >>>>>>>>>>> I then tested the following: >>>>>>>>>>> >>>>>>>>>>> - chef runs complete end to end >>>>>>>>>>> - Backing up / restoring static assets (see backup
scripts in
>>>>>>>>>>> /usr/local/bin or in the scale-drupal cookbook) >>>>>>>>>>> - browsing legacy static website (eg >>>>>>>>>>> https://socallinuxexpo.org/past/2002/ >>>>>>>>>>> https://socallinuxexpo.org/past/2003/) >>>>>>>>>>> - Registering as a speaker, confirming that I got
the email and that
>>>>>>>>>>> my account worked. >>>>>>>>>>> - publishing the submitted talk >>>>>>>>>>> - creating sponsors/exhibitors/blog post/events/etc >>>>>>>>>>> >>>>>>>>>>> Pending things: >>>>>>>>>>> - Centos8 seems to default to using php-fpm instead
of mod_php. Not
>>>>>>>>>>> opposed but it's different. Seems to work fine, but
I've not load
>>>>>>>>>>> tested it or anything. >>>>>>>>>>> >>>>>>>>>>> - drush (https://www.drush.org/) packages no longer
seemed to be
>>>>>>>>>>> present in centos or epel. remi seems to have it but
the drush docs
>>>>>>>>>>> suggest we should be installing this in our drupal
code base via
>>>>>>>>>>> composer instead of as a package. I dont have
strong opinions, but we
>>>>>>>>>>> should figure out the right path before merging. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> Ilan >>>>>>>>> _______________________________________________ >>>>>>>>> scale-infra mailing list >>>>>>>>> scale-infra@lists.linuxfests.org >>>>>>>>>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
>>>>>>>> >>>>>>>> -- >>>>>>>> Phil Dibowitz phil@ipom.com >>>>>>>> Open Source software and tech docs Insanity
Palace of Metallica
>>>>>>>> http://www.phildev.net/
>>>>>>>> >>>>>>>> "Be who you are and say what you feel, because those
who mind don't
>>>>>>>> matter and those who matter don't mind." >>>>>>>> - Dr. Seuss >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> scale-infra mailing list >>>>>>>> scale-infra@lists.linuxfests.org >>>>>>>>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
>>>>>> _______________________________________________ >>>>>> scale-infra mailing list >>>>>> scale-infra@lists.linuxfests.org >>>>>>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
>>>>> >>>>> -- >>>>> Phil Dibowitz phil@ipom.com >>>>> Open Source software and tech docs Insanity Palace
of Metallica
>>>>> http://www.phildev.net/
>>>>> >>>>> "Be who you are and say what you feel, because those who
mind don't
>>>>> matter and those who matter don't mind." >>>>> - Dr. Seuss >>>>> >>>>> >>>>> _______________________________________________ >>>>> scale-infra mailing list >>>>> scale-infra@lists.linuxfests.org >>>>>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
>>>> _______________________________________________ >>>> scale-infra mailing list >>>> scale-infra@lists.linuxfests.org >>>>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
>>> >>> -- >>> Phil Dibowitz phil@ipom.com >>> Open Source software and tech docs Insanity Palace of
Metallica
>>> >>> "Be who you are and say what you feel, because those who
mind don't
>>> matter and those who matter don't mind." >>> - Dr. Seuss >>> >>> >>> _______________________________________________ >>> scale-infra mailing list >>> scale-infra@lists.linuxfests.org >>>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
>> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org >>
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
> > -- > Phil Dibowitz phil@ipom.com > Open Source software and tech docs Insanity Palace of
Metallica
> http://www.phildev.net/ http://www.ipom.com/ > > "Be who you are and say what you feel, because those who mind
don't
> matter and those who matter don't mind." > - Dr. Seuss > > > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org >
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of
Metallica
http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind
don't
matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of
Metallica
http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
re-sending with Smith on CC.
On 10/7/23 11:38, Phil Dibowitz wrote:
Restarted. Added cron to restart. Will find time to pair debug with Smith this week.
Get BlueMail for Android https://bluemail.me On Oct 7, 2023, at 1:50 AM, Ilan Rabinovitch <ilan@socallinuxexpo.org mailto:ilan@socallinuxexpo.org> wrote:
Looks like we were down again for a few hours this morning. On Fri, Oct 6, 2023 at 10:03 PM Ilan Rabinovitch <ilan@socallinuxexpo.org> wrote: (resending with offlist folks) Looks like we had another outage this morning. I'm online for the next 45-60 min if someone wants to chat about it. Otherwise, I would greatly appreciate it if someone on the infra team could keep an eye on monitoring and at least restart things. On Fri, Oct 6, 2023 at 1:11 AM Phil Dibowitz <phil@ipom.com> wrote: Phil Smith is probably the best person since it seems to be a Drupal issue, best I can tell (I scoured logs a few months back and found nothing other than what you just described: apache threads just stay around, busy, until they eat up all the memory). I'm not sure he's on this list so I'm CC'ing hom. Phil - I'm happy to get you root access (send me an SSH pubkey), or anything else you may not already have to help you debug. I spent quite a while last SCALE trying to figure this out, Ilan has spent time on this, and it seems to be drupal-specific, best I can tell. On 10/5/23 14:49, Ilan Rabinovitch wrote: Theres definitely something continuing to go on with drupal and/or httpd. Something continues to eat up all the worker threads from httpd. Before we set maxworkers to something reasonable it would eat up all the memory, now it just consumes all the workers and the site becomes unresponsive. I dont think this relates to the upgrade, as it was happening before. We've had the site down three times in the past 2 days as a result. Monitoring detected it and I was able to respond to it fairly quickly, but I'm about to board a 30 hour flight, so I'm likely not going to be able to respond moving forward. Given the importance of the CFP this month I hope someone has a chance to dig in. On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz <phil@ipom.com> wrote: Copy that, I'll keep an eye out. Nice work, and thanks! On 10/3/23 18:06, Ilan Rabinovitch wrote: This is done. Here's the specific ids for any roll backs: 1) start scale-web1 up again (i-017df11978f41d9ae) 2) sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae) 3) move EIP (23.21.71.118 <http://23.21.71.118>) from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1. Assuming no issues the next change to think about is: - enable the s3fs plugin work Phil Smith did recently so static files aren't stored on box anymore. - move scale-web2 behind an ELB and auto scaling group. - remove the cron jobs that back up static files This will enable some reasonable failover that doesn't require human intervention. On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz <phil@ipom.com> wrote: Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday. On 10/3/23 16:08, Ilan Rabinovitch wrote: Cutover: - RDS Database snapshot - sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both hosts in case users have uploaded static assets - Move EIP from scale-web1 host to scale-web2 Rollback: - re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both hosts in case users have uploaded static assets - Move EIP from scale-web2 host to scale-web1 On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz <phil@ipom.com> wrote: Feel free to do it today. What's the migrate/failback process? It's somewhere in cloudflair? If not I can do it next week. On 10/3/23 11:09, Ilan Rabinovitch wrote: Sounds like we should do this today then, or you can grab next week. I'm going to be offline until November after Thursday morning. On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz <phil@ipom.com> wrote: I'm out of town Fri-Sun, and will only be around a laptop sporadically. I'm in town all week next week though. On 10/2/23 13:22, Ilan Rabinovitch wrote: Phil Smith, myself and a few others have banged on scale-web2 and we think we're all set to do a cutover there. However, I'm going to be out of town for a few weeks starting Thursday and do not yet know what my connectivity will be like. Would love to cutover this week, but only want to do it if others will be available to roll back to the old server should something go awry while I'm on vacation. On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch <ilan@socallinuxexpo.org> wrote: I believe Ive addressed everyone's feedback. scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production. To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file. 54.205.247.164 <http://54.205.247.164> www.socallinuxexpo.org <http://www.socallinuxexpo.org> Ideally would like to merge and migrate product this week On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz <phil@ipom.com> wrote: Sorry I was in Mexico. Left a review to DRY it up. But I don't see any critical things. - Phil On 9/22/23 15:23, Ilan Rabinovitch wrote: Ping? Would like to wrap this up as the non-stop daily outages are no fun, Managed to mitigate it somewhat, but given none of us are on call getting to some reasonable sense of HA would be appreciated so folks can go on vacation without worrying about who will reboot the servers. On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch <ilan@linuxfests.org> wrote: Thanks Davide for the initial review. I think I addressed the remaining feedback. Anything else we want to test before merging? On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch <ilan@linuxfests.org> wrote: Hi, I took a stab at updating the cookbooks we use for the website to use centos8. Motivation was getting a new version of php so we can use some drupal plugins that dont support php5.4, but i think this was on the backlog already for a while. https://github.com/socallinuxexpo/scale-chef/pull/283 <https://github.com/socallinuxexpo/scale-chef/pull/283> To test this out, I spun up scale-web2 with a clone of the production scale-drupal database, and then pointed my host file at it for www.socallinuxexpo.org <http://www.socallinuxexpo.org>: I then tested the following: - chef runs complete end to end - Backing up / restoring static assets (see backup scripts in /usr/local/bin or in the scale-drupal cookbook) - browsing legacy static website (eg https://socallinuxexpo.org/past/2002 <https://socallinuxexpo.org/past/2002>/ https://socallinuxexpo.org/past/2003 <https://socallinuxexpo.org/past/2003>/) - Registering as a speaker, confirming that I got the email and that my account worked. - publishing the submitted talk - creating sponsors/exhibitors/blog post/events/etc Pending things: - Centos8 seems to default to using php-fpm instead of mod_php. Not opposed but it's different. Seems to work fine, but I've not load tested it or anything. - drush (https://www.drush.org <https://www.drush.org>/) packages no longer seemed to be present in centos or epel. remi seems to have it but the drush docs suggest we should be installing this in our drupal code base via composer instead of as a package. I dont have strong opinions, but we should figure out the right path before merging. Cheers, Ilan ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> -- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> -- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> -- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> -- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> -- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> -- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> ------------------------------------------------------------------------ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
Yes, let's schedule something. I'm available tomorrow morning and Tuesday evening.
Phillip Smith 954-558-5022
On Sat, Oct 7, 2023, 5:27 PM Phil Dibowitz phil@ipom.com wrote:
re-sending with Smith on CC.
On 10/7/23 11:38, Phil Dibowitz wrote:
Restarted. Added cron to restart. Will find time to pair debug with Smith this week.
Get BlueMail for Android https://bluemail.me On Oct 7, 2023, at 1:50 AM, Ilan Rabinovitch <ilan@socallinuxexpo.org mailto:ilan@socallinuxexpo.org> wrote:
Looks like we were down again for a few hours this morning. On Fri, Oct 6, 2023 at 10:03 PM Ilan Rabinovitch <ilan@socallinuxexpo.org> wrote: (resending with offlist folks) Looks like we had another outage this morning. I'm online for the next 45-60 min if someone wants to chat about it. Otherwise, I would greatly appreciate it if someone on the infra team could keep aneye
on monitoring and at least restart things. On Fri, Oct 6, 2023 at 1:11 AM Phil Dibowitz <phil@ipom.com>wrote:
Phil Smith is probably the best person since it seems to be a Drupal issue, best I can tell (I scoured logs a few months back and found nothing other than what you just described: apache threads just stay around, busy, until they eat up all the memory). I'm not sure he's on this list so I'm CC'ing hom. Phil - I'm happy to get you root access (send me an SSH pubkey), or anything else you may not already have to help you debug. I spent quite a while last SCALE trying to figure this out, Ilan has spent time on this, and it seems to be drupal-specific, best I can tell. On 10/5/23 14:49, Ilan Rabinovitch wrote: Theres definitely something continuing to go on with drupal and/or httpd. Something continues to eat up all the worker threads from httpd. Before we set maxworkers to something reasonable it would eat up all the memory, now it just consumes all the workers and the site becomes unresponsive. I dont think this relates to the upgrade, as it was happening before. We've had the site down three times in the past 2 days as a result. Monitoring detected it and I was able to respond to it fairly quickly, but I'm about to board a 30 hour flight, so I'm likely not going to be able to respond moving forward. Given the importance of the CFP this month I hope someone has a chance to dig in. On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz <phil@ipom.com> wrote: Copy that, I'll keep an eye out. Nice work, andthanks!
On 10/3/23 18:06, Ilan Rabinovitch wrote: This is done. Here's the specific ids for any roll backs: 1) start scale-web1 up again(i-017df11978f41d9ae)
2) sync/home/drupal/scale-drupal/httpdocs/sites/default/files/ from
scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae) 3) move EIP (23.21.71.118 <http://23.21.71.118>) from scale-web2 (i-0bf1dd453774f2c3b) toscale-web1.
Assuming no issues the next change to think about is: - enable the s3fs plugin work Phil Smith did recently so static files aren't stored on box anymore. - move scale-web2 behind an ELB and auto scaling group. - remove the cron jobs that back up static files This will enable some reasonable failover that doesn't require human intervention. On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz <phil@ipom.com> wrote: Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday. On 10/3/23 16:08, Ilan Rabinovitch wrote: Cutover: - RDS Database snapshot - sync/home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
hosts in case users have uploaded static assets - Move EIP from scale-web1 host to scale-web2 Rollback: - re-sync/home/drupal/scale-drupal/httpdocs/sites/default/files/ on
both hosts in case users have uploaded static assets - Move EIP from scale-web2 host to scale-web1 On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz <phil@ipom.com> wrote: Feel free to do it today. What's the migrate/failback process? It's somewhere in cloudflair? If not I can do it next week. On 10/3/23 11:09, Ilan Rabinovitch wrote: Sounds like we should do this today then, or you can grab next week. I'm going to be offline until November after Thursday morning. On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz <phil@ipom.com>wrote:
I'm out of town Fri-Sun, and will only be around a laptop sporadically. I'm in town all week next week though. On 10/2/23 13:22, Ilan Rabinovitch wrote: Phil Smith, myself and a few others have banged on scale-web2 and we think we're all set to do a cutover there. However, I'm going to be out of town for a few weeks starting Thursday and do not yet know what my connectivity will be like. Would love to cutover this week, but only want to do it if others will be available to roll back to the old server should something go awry while I'm on vacation. On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch <ilan@socallinuxexpo.org>wrote:
I believe Ive addressed everyone's feedback. scale-web2 is online on running off the scale_web_centos8 branch. it's using a test db running from a snapshot of prod 2~ weeks ago. you can change it as you see fit without worrying about impacting production. To test you'll need to update /etc/hosts to point at the new server. following to your /etc/hosts file. 54.205.247.164 <http://54.205.247.164%3E www.socallinuxexpo.org < http://www.socallinuxexpo.org%3E
Ideally would like to merge and migrate product this week On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz <phil@ipom.com>wrote:
Sorry I was in Mexico. Left a review to DRY it up. But I don't see any critical things. - Phil On 9/22/23 15:23, Ilan Rabinovitchwrote:
Ping? Would like to wrap this up as the non-stop daily outages areno
fun, Managed to mitigate it somewhat, but given none of us are on call getting to some reasonable sense of HA would be appreciated so folks can go on vacation without worrying about who will reboot the servers. On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch <ilan@linuxfests.org> wrote:
Thanks Davide for the initial review. I think Iaddressed the
remaining feedback.
Anything else we want to test before merging? On Fri, Sep 15, 2023 at 7:52 PM IlanRabinovitch ilan@linuxfests.org wrote:
Hi, I took a stab atupdating the cookbooks we use for the website to use
centos8. Motivation was getting a new version of php so we can use
somedrupal plugins that dont support php5.4, but i think this was on
thebacklog already for a while.
https://github.com/socallinuxexpo/scale-chef/pull/283 < https://github.com/socallinuxexpo/scale-chef/pull/283%3E
To test this out, I spun upscale-web2 with a clone of the production
scale-drupal database, and then pointed my host file at it for
www.socallinuxexpo.org http://www.socallinuxexpo.org:
I thentested the following:
- chef runscomplete end to end
-Backing up / restoring static assets (see backup scripts in
/usr/local/bin or in the scale-drupal cookbook)
-browsing legacy static website (eg
https://socallinuxexpo.org/past/2002 <https://socallinuxexpo.org/past/2002
/
https://socallinuxexpo.org/past/2003 <https://socallinuxexpo.org/past/2003
/) -
Registering as a speaker, confirming that I got the email and that
myaccount worked.
-publishing the submitted talk
-creating sponsors/exhibitors/blog post/events/etc
Pending things:
-Centos8 seems to default to using php-fpm instead of mod_php. Not
opposed but it's different. Seems to work fine, but I've not load
tested it or anything.
-drush (https://www.drush.org https://www.drush.org/) packages no longer seemed to be
present in centos or epel. remi seems to have it but the drush docs
suggest we should be installing this in our drupal code base via
composer instead of as a package. I dont have strong opinions, but we
should figure out the right path before merging.
Cheers,
Ilan
scale-infra mailing listscale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallicahttp://www.phildev.net http://www.phildev.net/ http://www.ipom.com < http://www.ipom.com%3E/
"Be who you are and say what you feel, because those who minddon't
matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing listscale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing listscale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing listscale-infra@lists.linuxfests.org
https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don'tmind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace ofMetallica
http://www.phildev.net <http://www.phildev.net>/ http://www.ipom.com <http://www.ipom.com>/ "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.orghttps://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
On 10/7/23 14:30, Phillip Smith wrote:
Yes, let's schedule something. I'm available tomorrow morning and Tuesday evening.
What can I provide you in the mean time? I thought I saw an email from you saying you wanted Datadog access, but I can't seem to find it.
Davide may have time Tuesday, I will have time Wed evening. I'm out of town until Monday and have plans Mon/Tues evening.
- Phil
Are we able to remove the cron job that's restarting httpd?
On Sun, Oct 8, 2023 at 12:48 AM Phil Dibowitz phil@ipom.com wrote:
On 10/7/23 14:30, Phillip Smith wrote:
Yes, let's schedule something. I'm available tomorrow morning and Tuesday evening.
What can I provide you in the mean time? I thought I saw an email from you saying you wanted Datadog access, but I can't seem to find it.
Davide may have time Tuesday, I will have time Wed evening. I'm out of town until Monday and have plans Mon/Tues evening.
- Phil
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
I dropped it to less often a few days ago, I haven't yet looked at the logs to see what the status of httpd is at those times. My plan is to do that, then drop it to less often, and rinse and repeat.
Sorry new job is keeping me super busy.
On 10/21/23 08:25, Ilan Rabinovitch wrote:
Are we able to remove the cron job that's restarting httpd?
On Sun, Oct 8, 2023 at 12:48 AM Phil Dibowitz phil@ipom.com wrote:
On 10/7/23 14:30, Phillip Smith wrote:
Yes, let's schedule something. I'm available tomorrow morning and Tuesday evening.
What can I provide you in the mean time? I thought I saw an email from you saying you wanted Datadog access, but I can't seem to find it.
Davide may have time Tuesday, I will have time Wed evening. I'm out of town until Monday and have plans Mon/Tues evening.
- Phil
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Logs look good.
https://github.com/socallinuxexpo/scale-chef/pull/302 drops restarts to every 2 hours. Merged.
On 10/21/23 13:47, Phil Dibowitz wrote:
I dropped it to less often a few days ago, I haven't yet looked at the logs to see what the status of httpd is at those times. My plan is to do that, then drop it to less often, and rinse and repeat.
Sorry new job is keeping me super busy.
On 10/21/23 08:25, Ilan Rabinovitch wrote:
Are we able to remove the cron job that's restarting httpd?
On Sun, Oct 8, 2023 at 12:48 AM Phil Dibowitz phil@ipom.com wrote:
On 10/7/23 14:30, Phillip Smith wrote:
Yes, let's schedule something. I'm available tomorrow morning and Tuesday evening.
What can I provide you in the mean time? I thought I saw an email from you saying you wanted Datadog access, but I can't seem to find it.
Davide may have time Tuesday, I will have time Wed evening. I'm out of town until Monday and have plans Mon/Tues evening.
- Phil
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
website was down again.. looks like for about an hour. ive just restarted httpd.
On Sat, Oct 21, 2023 at 5:00 PM Phil Dibowitz phil@ipom.com wrote:
Logs look good.
https://github.com/socallinuxexpo/scale-chef/pull/302 drops restarts to every 2 hours. Merged.
On 10/21/23 13:47, Phil Dibowitz wrote:
I dropped it to less often a few days ago, I haven't yet looked at the logs to see what the status of httpd is at those times. My plan is to do that, then drop it to less often, and rinse and repeat.
Sorry new job is keeping me super busy.
On 10/21/23 08:25, Ilan Rabinovitch wrote:
Are we able to remove the cron job that's restarting httpd?
On Sun, Oct 8, 2023 at 12:48 AM Phil Dibowitz phil@ipom.com wrote:
On 10/7/23 14:30, Phillip Smith wrote:
Yes, let's schedule something. I'm available tomorrow morning and Tuesday evening.
What can I provide you in the mean time? I thought I saw an email from you saying you wanted Datadog access, but I can't seem to find it.
Davide may have time Tuesday, I will have time Wed evening. I'm out of town until Monday and have plans Mon/Tues evening.
- Phil
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Looks like another outage about 2 hours ago.
On Sun, Dec 24, 2023 at 8:45 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
website was down again.. looks like for about an hour. ive just restarted httpd.
On Sat, Oct 21, 2023 at 5:00 PM Phil Dibowitz phil@ipom.com wrote:
Logs look good.
https://github.com/socallinuxexpo/scale-chef/pull/302 drops restarts to every 2 hours. Merged.
On 10/21/23 13:47, Phil Dibowitz wrote:
I dropped it to less often a few days ago, I haven't yet looked at the logs to see what the status of httpd is at those times. My plan is to
do
that, then drop it to less often, and rinse and repeat.
Sorry new job is keeping me super busy.
On 10/21/23 08:25, Ilan Rabinovitch wrote:
Are we able to remove the cron job that's restarting httpd?
On Sun, Oct 8, 2023 at 12:48 AM Phil Dibowitz phil@ipom.com wrote:
On 10/7/23 14:30, Phillip Smith wrote:
Yes, let's schedule something. I'm available tomorrow morning and Tuesday evening.
What can I provide you in the mean time? I thought I saw an email from you saying you wanted Datadog access, but I can't seem to find it.
Davide may have time Tuesday, I will have time Wed evening. I'm out of town until Monday and have plans Mon/Tues evening.
- Phil
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Looks like we had another outage today. Any idea whats up?
On Mon, Dec 25, 2023 at 1:17 AM Ilan Rabinovitch ilan@linuxfests.org wrote:
Looks like another outage about 2 hours ago.
On Sun, Dec 24, 2023 at 8:45 PM Ilan Rabinovitch ilan@linuxfests.org wrote:
website was down again.. looks like for about an hour. ive just restarted httpd.
On Sat, Oct 21, 2023 at 5:00 PM Phil Dibowitz phil@ipom.com wrote:
Logs look good.
https://github.com/socallinuxexpo/scale-chef/pull/302 drops restarts to every 2 hours. Merged.
On 10/21/23 13:47, Phil Dibowitz wrote:
I dropped it to less often a few days ago, I haven't yet looked at the logs to see what the status of httpd is at those times. My plan is to
do
that, then drop it to less often, and rinse and repeat.
Sorry new job is keeping me super busy.
On 10/21/23 08:25, Ilan Rabinovitch wrote:
Are we able to remove the cron job that's restarting httpd?
On Sun, Oct 8, 2023 at 12:48 AM Phil Dibowitz phil@ipom.com wrote:
On 10/7/23 14:30, Phillip Smith wrote: > Yes, let's schedule something. I'm available tomorrow morning and > Tuesday evening.
What can I provide you in the mean time? I thought I saw an email
from
you saying you wanted Datadog access, but I can't seem to find it.
Davide may have time Tuesday, I will have time Wed evening. I'm out
of
town until Monday and have plans Mon/Tues evening.
- Phil
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
No, I woke up, saw the alert, bounced httpd, and had to run out the door.
It's a reasonable guess that it's the same memory issue as before. We're currently bouncing apache in cron every-other-hour, which seems to mostly keep us up, but occasionally not, so my guess is that's RIGHT at the threshold.
I don't have the bandwidth the go spelunking - the httpd.conf adjustments I made in... November or whatever that was seemed to help, but obviously they didn't solve everything. I still suspect that Drupal is causing connections to be held open somewhere, but that's pure speculation.
We have the data, between the apache_status.log and DD to figure it out, I'd just need time to do it, and I don't have the cycles right now.
On 1/7/24 17:11, Ilan Rabinovitch wrote:
Looks like we had another outage today. Any idea whats up?
On Mon, Dec 25, 2023 at 1:17 AM Ilan Rabinovitch <ilan@linuxfests.org mailto:ilan@linuxfests.org> wrote:
Looks like another outage about 2 hours ago. On Sun, Dec 24, 2023 at 8:45 PM Ilan Rabinovitch <ilan@linuxfests.org <mailto:ilan@linuxfests.org>> wrote: website was down again.. looks like for about an hour. ive just restarted httpd. On Sat, Oct 21, 2023 at 5:00 PM Phil Dibowitz <phil@ipom.com <mailto:phil@ipom.com>> wrote: Logs look good. https://github.com/socallinuxexpo/scale-chef/pull/302 <https://github.com/socallinuxexpo/scale-chef/pull/302> drops restarts to every 2 hours. Merged. On 10/21/23 13:47, Phil Dibowitz wrote: > I dropped it to less often a few days ago, I haven't yet looked at the > logs to see what the status of httpd is at those times. My plan is to do > that, then drop it to less often, and rinse and repeat. > > Sorry new job is keeping me super busy. > > On 10/21/23 08:25, Ilan Rabinovitch wrote: >> Are we able to remove the cron job that's restarting httpd? >> >> On Sun, Oct 8, 2023 at 12:48 AM Phil Dibowitz <phil@ipom.com <mailto:phil@ipom.com>> wrote: >>> >>> On 10/7/23 14:30, Phillip Smith wrote: >>>> Yes, let's schedule something. I'm available tomorrow morning and >>>> Tuesday evening. >>> >>> What can I provide you in the mean time? I thought I saw an email from >>> you saying you wanted Datadog access, but I can't seem to find it. >>> >>> Davide may have time Tuesday, I will have time Wed evening. I'm out of >>> town until Monday and have plans Mon/Tues evening. >>> >>> - Phil >>> >>> >>> _______________________________________________ >>> scale-infra mailing list >>> scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org> >>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> >> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org> >> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> > -- Phil Dibowitz phil@ipom.com <mailto:phil@ipom.com> Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ <http://www.phildev.net/> http://www.ipom.com/ <http://www.ipom.com/> "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss _______________________________________________ scale-infra mailing list scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Looks like we had another outage for about an hour today. 10am - 11:36am PT.
On Mon, Jan 8, 2024 at 9:53 AM Phil Dibowitz phil@ipom.com wrote:
No, I woke up, saw the alert, bounced httpd, and had to run out the door.
It's a reasonable guess that it's the same memory issue as before. We're currently bouncing apache in cron every-other-hour, which seems to mostly keep us up, but occasionally not, so my guess is that's RIGHT at the threshold.
I don't have the bandwidth the go spelunking - the httpd.conf adjustments I made in... November or whatever that was seemed to help, but obviously they didn't solve everything. I still suspect that Drupal is causing connections to be held open somewhere, but that's pure speculation.
We have the data, between the apache_status.log and DD to figure it out, I'd just need time to do it, and I don't have the cycles right now.
On 1/7/24 17:11, Ilan Rabinovitch wrote:
Looks like we had another outage today. Any idea whats up?
On Mon, Dec 25, 2023 at 1:17 AM Ilan Rabinovitch <ilan@linuxfests.org mailto:ilan@linuxfests.org> wrote:
Looks like another outage about 2 hours ago. On Sun, Dec 24, 2023 at 8:45 PM Ilan Rabinovitch <ilan@linuxfests.org <mailto:ilan@linuxfests.org>> wrote: website was down again.. looks like for about an hour. ive just restarted httpd. On Sat, Oct 21, 2023 at 5:00 PM Phil Dibowitz <phil@ipom.com <mailto:phil@ipom.com>> wrote: Logs look good. https://github.com/socallinuxexpo/scale-chef/pull/302 <https://github.com/socallinuxexpo/scale-chef/pull/302> drops restarts to every 2 hours. Merged. On 10/21/23 13:47, Phil Dibowitz wrote: > I dropped it to less often a few days ago, I haven't yet looked at the > logs to see what the status of httpd is at those times. My plan is to do > that, then drop it to less often, and rinse and repeat. > > Sorry new job is keeping me super busy. > > On 10/21/23 08:25, Ilan Rabinovitch wrote: >> Are we able to remove the cron job that's restartinghttpd?
>> >> On Sun, Oct 8, 2023 at 12:48 AM Phil Dibowitz <phil@ipom.com <mailto:phil@ipom.com>> wrote: >>> >>> On 10/7/23 14:30, Phillip Smith wrote: >>>> Yes, let's schedule something. I'm available tomorrow morning and >>>> Tuesday evening. >>> >>> What can I provide you in the mean time? I thought I saw an email from >>> you saying you wanted Datadog access, but I can't seem to find it. >>> >>> Davide may have time Tuesday, I will have time Wed evening. I'm out of >>> town until Monday and have plans Mon/Tues evening. >>> >>> - Phil >>> >>> >>> _______________________________________________ >>> scale-infra mailing list >>> scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org> >>>https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
>> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org> >>https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
> -- Phil Dibowitz phil@ipom.com <mailto:phil@ipom.com> Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ <http://www.phildev.net/> http://www.ipom.com/ <http://www.ipom.com/> "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss _______________________________________________ scale-infra mailing list scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org>https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra < https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra%3E
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Er, you're off by two hours 8:10am- 9:39am PT, best I can tell.
I just got home from work, but I did some digging around.
I don't think it's apache using too much memory. I don't think it's memory related at all.
Here's useful things I've found:
* Load average SPIKED a little while before the outage at around 7:52. It came back down before 8 though
* We had a spike in network traffic at exactly the same time. Only 7MiB/s, but way bigger than our usual traffic.
* There was a spike in memory usage at the same time, but the system was no wear near out of memory.
* The last datapoint DD got from Apache was at 7:54
* Odd thing that I don't think is related but is funny. The amount of memory journald is using drops significantly when apache gets killed from 215MB to 160MB. At the beginning the outage it ~188. I just restarted it and it's at 22MB. It's the top users of RSS memory on the box, or was until I restarted it.
Based on the fact that every OTHER process on the system was reporting into DD, but apache was not, it seems pretty likely that every thread in apache is tied up on _something_ - either talking to a DB, a bad client (seems unlikely since we have a cloudflare in front of us), or something else.
That's all I got for now.
On 1/30/24 12:04, Ilan Rabinovitch wrote:
Looks like we had another outage for about an hour today. 10am - 11:36am PT.
On Mon, Jan 8, 2024 at 9:53 AM Phil Dibowitz <phil@ipom.com mailto:phil@ipom.com> wrote:
No, I woke up, saw the alert, bounced httpd, and had to run out the door. It's a reasonable guess that it's the same memory issue as before. We're currently bouncing apache in cron every-other-hour, which seems to mostly keep us up, but occasionally not, so my guess is that's RIGHT at the threshold. I don't have the bandwidth the go spelunking - the httpd.conf adjustments I made in... November or whatever that was seemed to help, but obviously they didn't solve everything. I still suspect that Drupal is causing connections to be held open somewhere, but that's pure speculation. We have the data, between the apache_status.log and DD to figure it out, I'd just need time to do it, and I don't have the cycles right now. On 1/7/24 17:11, Ilan Rabinovitch wrote: > Looks like we had another outage today. Any idea whats up? > > On Mon, Dec 25, 2023 at 1:17 AM Ilan Rabinovitch <ilan@linuxfests.org <mailto:ilan@linuxfests.org> > <mailto:ilan@linuxfests.org <mailto:ilan@linuxfests.org>>> wrote: > > Looks like another outage about 2 hours ago. > > On Sun, Dec 24, 2023 at 8:45 PM Ilan Rabinovitch > <ilan@linuxfests.org <mailto:ilan@linuxfests.org> <mailto:ilan@linuxfests.org <mailto:ilan@linuxfests.org>>> wrote: > > > > website was down again.. looks like for about an hour. ive just > restarted httpd. > > On Sat, Oct 21, 2023 at 5:00 PM Phil Dibowitz <phil@ipom.com <mailto:phil@ipom.com> > <mailto:phil@ipom.com <mailto:phil@ipom.com>>> wrote: > > Logs look good. > > https://github.com/socallinuxexpo/scale-chef/pull/302 <https://github.com/socallinuxexpo/scale-chef/pull/302> > <https://github.com/socallinuxexpo/scale-chef/pull/302 <https://github.com/socallinuxexpo/scale-chef/pull/302>> > drops restarts to > every 2 hours. Merged. > > On 10/21/23 13:47, Phil Dibowitz wrote: > > I dropped it to less often a few days ago, I haven't yet > looked at the > > logs to see what the status of httpd is at those times. > My plan is to do > > that, then drop it to less often, and rinse and repeat. > > > > Sorry new job is keeping me super busy. > > > > On 10/21/23 08:25, Ilan Rabinovitch wrote: > >> Are we able to remove the cron job that's restarting httpd? > >> > >> On Sun, Oct 8, 2023 at 12:48 AM Phil Dibowitz > <phil@ipom.com <mailto:phil@ipom.com> <mailto:phil@ipom.com <mailto:phil@ipom.com>>> wrote: > >>> > >>> On 10/7/23 14:30, Phillip Smith wrote: > >>>> Yes, let's schedule something. I'm available tomorrow > morning and > >>>> Tuesday evening. > >>> > >>> What can I provide you in the mean time? I thought I > saw an email from > >>> you saying you wanted Datadog access, but I can't seem > to find it. > >>> > >>> Davide may have time Tuesday, I will have time Wed > evening. I'm out of > >>> town until Monday and have plans Mon/Tues evening. > >>> > >>> - Phil > >>> > >>> > >>> _______________________________________________ > >>> scale-infra mailing list > >>> scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org> > <mailto:scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org>> > >>> > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>> > >> _______________________________________________ > >> scale-infra mailing list > >> scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org> > <mailto:scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org>> > >> > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>> > > > > -- > Phil Dibowitz phil@ipom.com <mailto:phil@ipom.com> <mailto:phil@ipom.com <mailto:phil@ipom.com>> > Open Source software and tech docs Insanity Palace of > Metallica > http://www.phildev.net/ <http://www.phildev.net/> <http://www.phildev.net/ <http://www.phildev.net/>> > http://www.ipom.com/ <http://www.ipom.com/> <http://www.ipom.com/ <http://www.ipom.com/>> > > "Be who you are and say what you feel, because those who > mind don't > matter and those who matter don't mind." > - Dr. Seuss > > > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org> > <mailto:scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org>> > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>> > > > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org> > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> -- Phil Dibowitz phil@ipom.com <mailto:phil@ipom.com> Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ <http://www.phildev.net/> http://www.ipom.com/ <http://www.ipom.com/> "Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss _______________________________________________ scale-infra mailing list scale-infra@lists.linuxfests.org <mailto:scale-infra@lists.linuxfests.org> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra>
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Oh and I missed the obvious one, the huge spike in user CPU at the same time.
Anyway, this dashboard is locked at a timeframe that shows the spike, the outage, and the recovery for anyone wanting to see:
https://app.datadoghq.com/dash/host/14232198117?refresh_mode=paused&view...
On 1/30/24 21:58, Phil Dibowitz wrote:
Er, you're off by two hours 8:10am- 9:39am PT, best I can tell.
I just got home from work, but I did some digging around.
I don't think it's apache using too much memory. I don't think it's memory related at all.
Here's useful things I've found:
- Load average SPIKED a little while before the outage at around 7:52.
It came back down before 8 though
- We had a spike in network traffic at exactly the same time. Only
7MiB/s, but way bigger than our usual traffic.
- There was a spike in memory usage at the same time, but the system was
no wear near out of memory.
The last datapoint DD got from Apache was at 7:54
Odd thing that I don't think is related but is funny. The amount of
memory journald is using drops significantly when apache gets killed from 215MB to 160MB. At the beginning the outage it ~188. I just restarted it and it's at 22MB. It's the top users of RSS memory on the box, or was until I restarted it.
Based on the fact that every OTHER process on the system was reporting into DD, but apache was not, it seems pretty likely that every thread in apache is tied up on _something_ - either talking to a DB, a bad client (seems unlikely since we have a cloudflare in front of us), or something else.
That's all I got for now.
On 1/30/24 12:04, Ilan Rabinovitch wrote:
Looks like we had another outage for about an hour today. 10am - 11:36am PT.
On Mon, Jan 8, 2024 at 9:53 AM Phil Dibowitz <phil@ipom.com mailto:phil@ipom.com> wrote:
No, I woke up, saw the alert, bounced httpd, and had to run out the door.
It's a reasonable guess that it's the same memory issue as before. We're currently bouncing apache in cron every-other-hour, which seems to mostly keep us up, but occasionally not, so my guess is that's RIGHT at the threshold.
I don't have the bandwidth the go spelunking - the httpd.conf adjustments I made in... November or whatever that was seemed to help, but obviously they didn't solve everything. I still suspect that Drupal is causing connections to be held open somewhere, but that's pure speculation.
We have the data, between the apache_status.log and DD to figure it out, I'd just need time to do it, and I don't have the cycles right now.
On 1/7/24 17:11, Ilan Rabinovitch wrote: > Looks like we had another outage today. Any idea whats up? > > On Mon, Dec 25, 2023 at 1:17 AM Ilan Rabinovitch <ilan@linuxfests.org mailto:ilan@linuxfests.org > <mailto:ilan@linuxfests.org mailto:ilan@linuxfests.org>> wrote: > > Looks like another outage about 2 hours ago. > > On Sun, Dec 24, 2023 at 8:45 PM Ilan Rabinovitch > <ilan@linuxfests.org mailto:ilan@linuxfests.org <mailto:ilan@linuxfests.org mailto:ilan@linuxfests.org>> wrote: > > > > website was down again.. looks like for about an hour. ive just > restarted httpd. > > On Sat, Oct 21, 2023 at 5:00 PM Phil Dibowitz <phil@ipom.com mailto:phil@ipom.com > <mailto:phil@ipom.com mailto:phil@ipom.com>> wrote: > > Logs look good. > > https://github.com/socallinuxexpo/scale-chef/pull/302 https://github.com/socallinuxexpo/scale-chef/pull/302 > <https://github.com/socallinuxexpo/scale-chef/pull/302 https://github.com/socallinuxexpo/scale-chef/pull/302> > drops restarts to > every 2 hours. Merged. > > On 10/21/23 13:47, Phil Dibowitz wrote: > > I dropped it to less often a few days ago, I haven't yet > looked at the > > logs to see what the status of httpd is at those times. > My plan is to do > > that, then drop it to less often, and rinse and repeat. > > > > Sorry new job is keeping me super busy. > > > > On 10/21/23 08:25, Ilan Rabinovitch wrote: > >> Are we able to remove the cron job that's restarting httpd? > >> > >> On Sun, Oct 8, 2023 at 12:48 AM Phil Dibowitz > <phil@ipom.com mailto:phil@ipom.com <mailto:phil@ipom.com mailto:phil@ipom.com>> wrote: > >>> > >>> On 10/7/23 14:30, Phillip Smith wrote: > >>>> Yes, let's schedule something. I'm available tomorrow > morning and > >>>> Tuesday evening. > >>> > >>> What can I provide you in the mean time? I thought I > saw an email from > >>> you saying you wanted Datadog access, but I can't seem > to find it. > >>> > >>> Davide may have time Tuesday, I will have time Wed > evening. I'm out of > >>> town until Monday and have plans Mon/Tues evening. > >>> > >>> - Phil > >>> > >>> > >>> _______________________________________________ > >>> scale-infra mailing list > >>> scale-infra@lists.linuxfests.org mailto:scale-infra@lists.linuxfests.org > <mailto:scale-infra@lists.linuxfests.org mailto:scale-infra@lists.linuxfests.org> > >>> > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> > >> _______________________________________________ > >> scale-infra mailing list > >> scale-infra@lists.linuxfests.org mailto:scale-infra@lists.linuxfests.org > <mailto:scale-infra@lists.linuxfests.org mailto:scale-infra@lists.linuxfests.org> > >> > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> > > > > -- > Phil Dibowitz phil@ipom.com mailto:phil@ipom.com <mailto:phil@ipom.com mailto:phil@ipom.com> > Open Source software and tech docs Insanity Palace of > Metallica > http://www.phildev.net/ http://www.phildev.net/ <http://www.phildev.net/ http://www.phildev.net/> > http://www.ipom.com/ http://www.ipom.com/ <http://www.ipom.com/ http://www.ipom.com/> > > "Be who you are and say what you feel, because those who > mind don't > matter and those who matter don't mind." > - Dr. Seuss > > > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org mailto:scale-infra@lists.linuxfests.org > <mailto:scale-infra@lists.linuxfests.org mailto:scale-infra@lists.linuxfests.org> > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra <https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra> > > > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org mailto:scale-infra@lists.linuxfests.org > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com mailto:phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.phildev.net/ http://www.ipom.com/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
_______________________________________________ scale-infra mailing list scale-infra@lists.linuxfests.org mailto:scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
On Wed, Jan 31, 2024 at 12:59 AM Phil Dibowitz phil@ipom.com wrote:
Er, you're off by two hours 8:10am- 9:39am PT, best I can tell.
Was traveling and guessing at timezones from slack. Hope we can get to the bottom of it all.
In the meantime...
https://github.com/socallinuxexpo/scale-chef/pull/305
On 2/2/24 15:43, Ilan Rabinovitch wrote:
On Wed, Jan 31, 2024 at 12:59 AM Phil Dibowitz phil@ipom.com wrote:
Er, you're off by two hours 8:10am- 9:39am PT, best I can tell.
Was traveling and guessing at timezones from slack. Hope we can get to the bottom of it all. _______________________________________________ scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Logs on the host from just before the last outage. Seems like we need some mpm tuning. We set worker limit and max connections, but haven't adjusted server limit from defaults yet. I think that defaults so something low with mpm_event.
error_log:[Sat Oct 07 18:35:35.108228 2023] [mpm_event:error] [pid 598185:tid 139630331418944] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. error_log:[Sat Oct 07 18:35:36.109222 2023] [mpm_event:error] [pid 598185:tid 139630331418944] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. error_log:[Sat Oct 07 18:35:40.123947 2023] [mpm_event:error] [pid 598185:tid 139630331418944] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
On Sat, Oct 7, 2023 at 9:39 PM Phil Dibowitz phil@ipom.com wrote:
Restarted. Added cron to restart. Will find time to pair debug with Smith this week.
Get BlueMail for Android
This is a confirmed bug in Apache with no configuration workaround:
https://bz.apache.org/bugzilla/show_bug.cgi?id=53555
There's a pretty good description of the details, which match our behavior here:
https://serverfault.com/questions/516373/what-is-the-meaning-of-ah00485-scor...
One thing that some people seem to have had success with is turning keepalive off.
ServerLimit, for the event MPM (what we use), the recommendation is:
With event, increase this directive if the process number defined by your MaxRequestWorkers and ThreadsPerChild settings, plus the number of gracefully shutting down processes, is more than 16 server processes (default).
Our MaxRequestWorkers is 50. We don't define ThreadsPerChild which defaults to 25. So if I read that correctly (and I haven't tuned Apache for a living in a looonnnggg time), we want something like 50+25=75 plus some more for shutting down processes, so like... 80? I can prep a diff for that plus keepalive and see how that goes.
On 10/7/23 22:00, Ilan Rabinovitch wrote:
Logs on the host from just before the last outage. Seems like we need some mpm tuning. We set worker limit and max connections, but haven't adjusted server limit from defaults yet. I think that defaults so something low with mpm_event.
error_log:[Sat Oct 07 18:35:35.108228 2023] [mpm_event:error] [pid 598185:tid 139630331418944] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. error_log:[Sat Oct 07 18:35:36.109222 2023] [mpm_event:error] [pid 598185:tid 139630331418944] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. error_log:[Sat Oct 07 18:35:40.123947 2023] [mpm_event:error] [pid 598185:tid 139630331418944] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
On Sat, Oct 7, 2023 at 9:39 PM Phil Dibowitz phil@ipom.com wrote:
Restarted. Added cron to restart. Will find time to pair debug with Smith this week.
Get BlueMail for Android
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
OK I wrote up https://github.com/socallinuxexpo/scale-chef/pull/293 which does several things:
* log state of apache before we restart * restart hourly instead of every 30 minutes * bump requestsperchild a bit * bump serverlimit * turn off keep alive
I'll wait to merge it until there's a review.
On 10/8/23 13:17, Phil Dibowitz wrote:
This is a confirmed bug in Apache with no configuration workaround:
https://bz.apache.org/bugzilla/show_bug.cgi?id=53555
There's a pretty good description of the details, which match our behavior here:
https://serverfault.com/questions/516373/what-is-the-meaning-of-ah00485-scor...
One thing that some people seem to have had success with is turning keepalive off.
ServerLimit, for the event MPM (what we use), the recommendation is:
With event, increase this directive if the process number defined by your MaxRequestWorkers and ThreadsPerChild settings, plus the number of gracefully shutting down processes, is more than 16 server processes (default).
Our MaxRequestWorkers is 50. We don't define ThreadsPerChild which defaults to 25. So if I read that correctly (and I haven't tuned Apache for a living in a looonnnggg time), we want something like 50+25=75 plus some more for shutting down processes, so like... 80? I can prep a diff for that plus keepalive and see how that goes.
On 10/7/23 22:00, Ilan Rabinovitch wrote:
Logs on the host from just before the last outage. Seems like we need some mpm tuning. We set worker limit and max connections, but haven't adjusted server limit from defaults yet. I think that defaults so something low with mpm_event.
error_log:[Sat Oct 07 18:35:35.108228 2023] [mpm_event:error] [pid 598185:tid 139630331418944] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. error_log:[Sat Oct 07 18:35:36.109222 2023] [mpm_event:error] [pid 598185:tid 139630331418944] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. error_log:[Sat Oct 07 18:35:40.123947 2023] [mpm_event:error] [pid 598185:tid 139630331418944] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit.
On Sat, Oct 7, 2023 at 9:39 PM Phil Dibowitz phil@ipom.com wrote:
Restarted. Added cron to restart. Will find time to pair debug with Smith this week.
Get BlueMail for Android
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
Looks like we had another outage this morning.
On Fri, Oct 6, 2023 at 12:49 AM Ilan Rabinovitch ilan@linuxfests.org wrote:
Theres definitely something continuing to go on with drupal and/or httpd. Something continues to eat up all the worker threads from httpd. Before we set maxworkers to something reasonable it would eat up all the memory, now it just consumes all the workers and the site becomes unresponsive. I dont think this relates to the upgrade, as it was happening before.
We've had the site down three times in the past 2 days as a result. Monitoring detected it and I was able to respond to it fairly quickly, but I'm about to board a 30 hour flight, so I'm likely not going to be able to respond moving forward.
Given the importance of the CFP this month I hope someone has a chance to dig in.
On Tue, Oct 3, 2023 at 10:29 PM Phil Dibowitz phil@ipom.com wrote:
Copy that, I'll keep an eye out. Nice work, and thanks!
On 10/3/23 18:06, Ilan Rabinovitch wrote:
This is done. Here's the specific ids for any roll backs:
- start scale-web1 up again (i-017df11978f41d9ae)
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ from
scale-web2 (i-0bf1dd453774f2c3b) to scale-web1 (i-017df11978f41d9ae) 3) move EIP (23.21.71.118) from scale-web2 (i-0bf1dd453774f2c3b) to scale-web1.
Assuming no issues the next change to think about is:
- enable the s3fs plugin work Phil Smith did recently so static files
aren't stored on box anymore.
- move scale-web2 behind an ELB and auto scaling group.
- remove the cron jobs that back up static files
This will enable some reasonable failover that doesn't require human intervention.
On Tue, Oct 3, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote:
Easy-peasey. Feel free to do it as your leisure today. I have a few errands to run, but will never be more than 30m from home and will be able to roll back today, and will be around all day tomorrow and Thursday.
On 10/3/23 16:08, Ilan Rabinovitch wrote:
Cutover:
- RDS Database snapshot
- sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on both
hosts in case users have uploaded static assets
- Move EIP from scale-web1 host to scale-web2
Rollback:
- re-sync /home/drupal/scale-drupal/httpdocs/sites/default/files/ on
both hosts in case users have uploaded static assets
- Move EIP from scale-web2 host to scale-web1
On Tue, Oct 3, 2023 at 3:41 PM Phil Dibowitz phil@ipom.com wrote:
Feel free to do it today.
What's the migrate/failback process? It's somewhere in cloudflair?
If not I can do it next week.
On 10/3/23 11:09, Ilan Rabinovitch wrote: > Sounds like we should do this today then, or you can grab next week. > I'm going to be offline until November after Thursday morning. > > On Mon, Oct 2, 2023 at 9:35 PM Phil Dibowitz phil@ipom.com wrote: >> >> I'm out of town Fri-Sun, and will only be around a laptop sporadically. >> I'm in town all week next week though. >> >> >> >> On 10/2/23 13:22, Ilan Rabinovitch wrote: >>> Phil Smith, myself and a few others have banged on scale-web2 and we >>> think we're all set to do a cutover there. >>> However, I'm going to be out of town for a few weeks starting Thursday >>> and do not yet know what my connectivity will be like. >>> >>> Would love to cutover this week, but only want to do it if others will >>> be available to roll back to the old server should something go awry >>> while I'm on vacation. >>> >>> On Sun, Sep 24, 2023 at 9:46 PM Ilan Rabinovitch >>> ilan@socallinuxexpo.org wrote: >>>> >>>> I believe Ive addressed everyone's feedback. >>>> >>>> scale-web2 is online on running off the scale_web_centos8 branch. >>>> it's using a test db running from a snapshot of prod 2~ weeks ago. >>>> you can change it as you see fit without worrying about impacting production. >>>> >>>> To test you'll need to update /etc/hosts to point at the new server. >>>> following to your /etc/hosts file. >>>> >>>> 54.205.247.164 www.socallinuxexpo.org >>>> >>>> Ideally would like to merge and migrate product this week >>>> >>>> >>>> On Fri, Sep 22, 2023 at 7:19 PM Phil Dibowitz phil@ipom.com wrote: >>>>> >>>>> Sorry I was in Mexico. >>>>> >>>>> Left a review to DRY it up. But I don't see any critical things. >>>>> >>>>> - Phil >>>>> >>>>> On 9/22/23 15:23, Ilan Rabinovitch wrote: >>>>>> Ping? Would like to wrap this up as the non-stop daily outages are no >>>>>> fun, Managed to mitigate it somewhat, but given none of us are on call >>>>>> getting to some reasonable sense of HA would be appreciated so folks >>>>>> can go on vacation without worrying about who will reboot the servers. >>>>>> >>>>>> On Tue, Sep 19, 2023 at 8:59 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>>> >>>>>>> Thanks Davide for the initial review. I think I addressed the >>>>>>> remaining feedback. >>>>>>> >>>>>>> Anything else we want to test before merging? >>>>>>> >>>>>>> On Fri, Sep 15, 2023 at 7:52 PM Ilan Rabinovitch ilan@linuxfests.org wrote: >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I took a stab at updating the cookbooks we use for the website to use >>>>>>>> centos8. Motivation was getting a new version of php so we can use >>>>>>>> some drupal plugins that dont support php5.4, but i think this was on >>>>>>>> the backlog already for a while. >>>>>>>> >>>>>>>> https://github.com/socallinuxexpo/scale-chef/pull/283 >>>>>>>> >>>>>>>> To test this out, I spun up scale-web2 with a clone of the production >>>>>>>> scale-drupal database, and then pointed my host file at it for >>>>>>>> www.socallinuxexpo.org: >>>>>>>> >>>>>>>> I then tested the following: >>>>>>>> >>>>>>>> - chef runs complete end to end >>>>>>>> - Backing up / restoring static assets (see backup scripts in >>>>>>>> /usr/local/bin or in the scale-drupal cookbook) >>>>>>>> - browsing legacy static website (eg >>>>>>>> https://socallinuxexpo.org/past/2002/ >>>>>>>> https://socallinuxexpo.org/past/2003/) >>>>>>>> - Registering as a speaker, confirming that I got the email and that >>>>>>>> my account worked. >>>>>>>> - publishing the submitted talk >>>>>>>> - creating sponsors/exhibitors/blog post/events/etc >>>>>>>> >>>>>>>> Pending things: >>>>>>>> - Centos8 seems to default to using php-fpm instead of mod_php. Not >>>>>>>> opposed but it's different. Seems to work fine, but I've not load >>>>>>>> tested it or anything. >>>>>>>> >>>>>>>> - drush (https://www.drush.org/) packages no longer seemed to be >>>>>>>> present in centos or epel. remi seems to have it but the drush docs >>>>>>>> suggest we should be installing this in our drupal code base via >>>>>>>> composer instead of as a package. I dont have strong opinions, but we >>>>>>>> should figure out the right path before merging. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Ilan >>>>>> _______________________________________________ >>>>>> scale-infra mailing list >>>>>> scale-infra@lists.linuxfests.org >>>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>>>> >>>>> -- >>>>> Phil Dibowitz phil@ipom.com >>>>> Open Source software and tech docs Insanity Palace of Metallica >>>>> http://www.phildev.net/ http://www.ipom.com/ >>>>> >>>>> "Be who you are and say what you feel, because those who mind don't >>>>> matter and those who matter don't mind." >>>>> - Dr. Seuss >>>>> >>>>> >>>>> _______________________________________________ >>>>> scale-infra mailing list >>>>> scale-infra@lists.linuxfests.org >>>>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >>> _______________________________________________ >>> scale-infra mailing list >>> scale-infra@lists.linuxfests.org >>> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra >> >> -- >> Phil Dibowitz phil@ipom.com >> Open Source software and tech docs Insanity Palace of Metallica >> http://www.phildev.net/ http://www.ipom.com/ >> >> "Be who you are and say what you feel, because those who mind don't >> matter and those who matter don't mind." >> - Dr. Seuss >> >> >> _______________________________________________ >> scale-infra mailing list >> scale-infra@lists.linuxfests.org >> https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra > _______________________________________________ > scale-infra mailing list > scale-infra@lists.linuxfests.org > https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind." - Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
-- Phil Dibowitz phil@ipom.com Open Source software and tech docs Insanity Palace of Metallica http://www.phildev.net/ http://www.ipom.com/
"Be who you are and say what you feel, because those who mind don't matter and those who matter don't mind."
- Dr. Seuss
scale-infra mailing list scale-infra@lists.linuxfests.org https://lists.linuxfests.org/cgi-bin/mailman/listinfo/scale-infra
scale-infra@lists.linuxfests.org
-
Ilan Rabinovitch -
Ilan Rabinovitch -
Phil Dibowitz -
Phillip Smith