2020-08-21 12-44-59 PDTManuel Recena <recena@gmail.com>When I rebooted the system everything around Rocket Chat was working fine.
2018-12-26 15-38-25 PSTManuel Recena <recena@gmail.com>Let me check it
2018-09-27 13-05-58 PDTManuel Recena <recena@gmail.com>By the way, it would be great to know who is part of the infrastructure team ;)
2019-02-03 11-38-07 PSTManuel Recena <recena@gmail.com>Let me back to the mailman3 subject later.
2018-09-27 12-22-26 PDTManuel Recena <recena@gmail.com>Is there any place where I can see the services that we want to set up?
2018-12-27 01-50-53 PSTManuel Recena <recena@gmail.com>By the way, Weblate is ready to deploy on production
2018-09-27 12-05-01 PDTManuel Recena <recena@gmail.com>Ahh perfect
2018-09-27 13-02-32 PDTManuel Recena <recena@gmail.com>I am here for helping
2020-11-08 04-04-53 PSTManuel Recena <recena@gmail.com>Do you know if someone has manipulated this VM(5.102.146.198)?
2019-02-03 11-46-21 PSTManuel Recena <recena@gmail.com>ok
2020-11-06 13-10-55 PSTManuel Recena <recena@gmail.com>I will spend time tomorrow. I hope to get good news.
2018-12-27 01-57-56 PSTManuel Recena <recena@gmail.com>In fact, we should deploy the solution over the Christmas
2020-11-08 04-04-16 PSTManuel Recena <recena@gmail.com>FYI https://gitlab.com/inkscape/infra/services/-/issues/43#note_443680262
2018-12-27 01-44-48 PSTManuel Recena <recena@gmail.com>Ok, we can do it but a backup would be recommended
2018-09-27 12-05-54 PDTManuel Recena <recena@gmail.com>It is fine. I just wanted to confirm it
2018-12-27 01-49-14 PSTManuel Recena <recena@gmail.com>My main concern is about mongo
2018-09-27 12-05-26 PDTManuel Recena <recena@gmail.com>I am assuming that each service will be deployed into a dedicated machine
2018-09-27 12-16-57 PDTManuel Recena <recena@gmail.com>For that reason, it is better small machines (2Gb) with only one service
2019-02-03 11-53-22 PSTManuel Recena <recena@gmail.com>My concern is with credentials related with mongo, mailjet, etc...
2019-02-03 11-41-33 PSTManuel Recena <recena@gmail.com>That is great
2018-12-27 01-50-33 PSTManuel Recena <recena@gmail.com>Let me investigate a bit, ok?
2018-12-27 01-58-46 PSTManuel Recena <recena@gmail.com>It is just running the playbook
2018-12-27 01-45-17 PSTManuel Recena <recena@gmail.com>A snapshot of the server
2019-02-03 11-42-55 PSTManuel Recena <recena@gmail.com>Thank to this re-deployment we have updated some components ;)
2020-11-08 10-38-21 PSTManuel Recena <recena@gmail.com>But the Ansible Playbook wasn't designed to be run having two production environment. I have to perform some changes
2018-09-27 11-43-46 PDTManuel Recena <recena@gmail.com>Hello!
2018-09-27 12-15-33 PDTManuel Recena <recena@gmail.com>I can rework on it, but we need to define the organization firstly
2019-02-03 11-39-37 PSTManuel Recena <recena@gmail.com>https://gitlab.com/inkscape/infra/services/blob/master/ansible/inventories/inkscape/hosts.yml
2018-12-27 01-49-36 PSTManuel Recena <recena@gmail.com>If we upgrade the OS many packages will be upgraded as well
2019-02-03 11-51-20 PSTManuel Recena <recena@gmail.com>ok
2019-02-03 11-38-28 PSTManuel Recena <recena@gmail.com>I had to fix the SMTP configuration
2020-11-08 10-58-24 PSTManuel Recena <recena@gmail.com>My user does not have a password and it is not a sudoers
2019-02-03 11-36-04 PSTManuel Recena <recena@gmail.com>ok
2020-09-10 23-42-06 PDTManuel Recena <recena@gmail.com>Are you and family safe?
2019-02-03 11-42-08 PSTManuel Recena <recena@gmail.com>ok
2019-02-03 11-44-17 PSTManuel Recena <recena@gmail.com>But the new version are all of them compatible for Rocket.Chat
2020-11-08 10-36-53 PSTManuel Recena <recena@gmail.com>I am going to check the new VM
2019-02-03 11-44-01 PSTManuel Recena <recena@gmail.com>Node is update, Mongo is updated, Nginx is updated
2019-02-03 11-39-08 PSTManuel Recena <recena@gmail.com>the password was not configurared
2019-02-03 12-11-26 PSTManuel Recena <recena@gmail.com>letsencrypt is working fine
2020-08-21 12-46-16 PDTManuel Recena <recena@gmail.com>I would like to perform a full refesh of the environment
2020-08-21 12-46-42 PDTManuel Recena <recena@gmail.com>But it requires time and planning
2019-02-03 11-39-17 PSTManuel Recena <recena@gmail.com>because it is not in the configuration file
2019-02-03 11-44-55 PSTManuel Recena <recena@gmail.com>thanks.
2019-02-03 11-43-25 PSTManuel Recena <recena@gmail.com>1) My plan is to define a backup for Rocket.Chat
2018-12-27 02-00-29 PSTManuel Recena <recena@gmail.com>I will keep you in touch
2018-09-27 11-43-59 PDTManuel Recena <recena@gmail.com>I've just read your email
2018-09-27 12-14-44 PDTManuel Recena <recena@gmail.com>One service, one machine
2018-09-27 12-20-03 PDTManuel Recena <recena@gmail.com>I am going to finish first the playbook for Weblate, deploy it and them, we can discuss the next step
2018-09-27 11-44-18 PDTManuel Recena <recena@gmail.com>I hope to reply to you later
2020-08-21 12-45-43 PDTManuel Recena <recena@gmail.com>IMHO, that dist-upgrade was the root cause of the outage
2019-02-03 11-40-35 PSTManuel Recena <recena@gmail.com>when they are consumed from Ansible
2020-08-21 09-53-12 PDTManuel Recena <recena@gmail.com>Hello
2020-11-08 10-39-40 PSTManuel Recena <recena@gmail.com>When we are able to shutdown the current production environment?
2019-02-03 11-43-34 PSTManuel Recena <recena@gmail.com>2) Finish Weblate
2019-02-03 11-40-29 PSTManuel Recena <recena@gmail.com>I would like to improve how we are managing the passwords and tokens
2019-02-03 11-34-34 PSTManuel Recena <recena@gmail.com>Which IP?
2019-02-03 11-46-33 PSTManuel Recena <recena@gmail.com>I want to write it and automate it
2019-02-03 11-34-02 PSTManuel Recena <recena@gmail.com>Hello Bryce
2019-02-03 11-51-36 PSTManuel Recena <recena@gmail.com>I'm checking if everyting is fine on Rocket.Chat
2020-09-06 23-37-39 PDTManuel Recena <recena@gmail.com>Hello! We should move forward the issue on Gitlab about upgrading RocketChat
2019-02-03 11-43-05 PSTManuel Recena <recena@gmail.com>I can do it.
2020-08-21 12-47-44 PDTManuel Recena <recena@gmail.com>I will create a proper issue to address this topic
2019-02-03 11-44-32 PSTManuel Recena <recena@gmail.com>I think the Ansible Playbook worked very well.
2019-02-03 11-34-22 PSTManuel Recena <recena@gmail.com>Is the previous env avaialble?
2020-08-21 09-54-31 PDTManuel Recena <recena@gmail.com>But in advance, I didn't touch anything, just restarting the server after facing the issue
2020-08-21 09-53-56 PDTManuel Recena <recena@gmail.com>I didn't have a chance for participating on the gitlab issue regarding to Rocket.chat service...
2019-02-03 11-36-22 PSTManuel Recena <recena@gmail.com>that server is the previous Rocket.Chat, right?
2019-02-03 11-48-35 PSTManuel Recena <recena@gmail.com>Have you configured something related to letsencrypt?
2019-02-03 11-38-21 PSTManuel Recena <recena@gmail.com>I want to make sure that everything is working fine with Rocket.Chat
2018-12-27 01-57-35 PSTManuel Recena <recena@gmail.com>We needed to define the procedure to use the system
2018-09-27 12-28-38 PDTManuel Recena <recena@gmail.com>I don't have any problem about maintaining all the infrastructure, but I need to know the requirements and which thinks I can decide
2018-09-27 12-02-45 PDTManuel Recena <recena@gmail.com>I don't know if I understood correctly from your email
2018-12-27 02-03-07 PSTManuel Recena <recena@gmail.com>By the way, Happy Christmas!
2018-12-27 01-46-40 PSTManuel Recena <recena@gmail.com>But IMHO the base OS is enough updated
2018-09-27 12-03-14 PDTManuel Recena <recena@gmail.com>But do we plan to have each service on a separated machine, right?
2020-11-08 10-57-50 PSTManuel Recena <recena@gmail.com>I'm blocked again.
2018-12-27 01-50-23 PSTManuel Recena <recena@gmail.com>And I am not sure a about the compatibility with Rocket.Chat
2018-12-27 01-43-28 PSTManuel Recena <recena@gmail.com>What is the motivation?
2018-09-27 12-13-33 PDTManuel Recena <recena@gmail.com>The playbook are being designed in a different manner
2018-09-27 12-16-19 PDTManuel Recena <recena@gmail.com>In any way, if a service grows up, we will need to scale the machine
2018-12-27 01-56-56 PSTManuel Recena <recena@gmail.com>From my side, it is done
2019-02-03 17-08-00 PSTBryce Harrington <bryce@bryceharrington.org>to avoid bus factor issues, it would probably make sense at some point for us to establish a team email that can be used for service registrations and such
2020-09-13 21-01-25 PDTBryce Harrington <bryce@bryceharrington.org>Hi recena, yes we're safe. The county adjacent to us had to evacuate and we were on the cusp, but they got the fire near us under control yesterday, and we've not had to actually evacuate. We did pack everything up though. Hard to go through all your possessions and prioritize what small bit you want to preserve. I feel really bad for people around here who have lost their houses, and thankful the fire crews are working so hard.
2018-09-27 12-04-28 PDTBryce Harrington <bryce@bryceharrington.org>the sponsored hosting provides us with up to 32G total memory for VMs. We can divide the memory up by 2G increments, thus we have an effective maximum of 16 vms
2018-09-27 12-09-33 PDTBryce Harrington <bryce@bryceharrington.org>services/cloudscale_nodes.txt is where my current thoughts on purposes are
2018-12-26 15-09-03 PSTBryce Harrington <bryce@bryceharrington.org>heya
2020-11-08 08-36-47 PSTBryce Harrington <bryce@bryceharrington.org>It's at 5.102.146.198 I've set up ssh root access for you and I
2018-12-27 01-58-15 PSTBryce Harrington <bryce@bryceharrington.org>how?
2020-12-30 10-09-43 PSTBryce Harrington <bryce@bryceharrington.org>hi recena, I am guessing your time hasn't freed up much. I hope everything is ok. I've got a bit of time over the holidays but it's going to tighten up again after new years. On https://gitlab.com/inkscape/infra/services/-/issues/43 Mc had expressed interest in handling the rocketchat upgrade. Later today I'm going to give Mc root access to the machines so he can do the task, unless you tell me otherwise.
2018-12-27 02-06-45 PSTBryce Harrington <bryce@bryceharrington.org>Happy Christmas to you too!
2018-09-27 12-26-11 PDTBryce Harrington <bryce@bryceharrington.org>the most definitive list of services we need to host, was in the original call for hosting, which I think is posted on the main website. Should be googleable I think.
2019-02-03 17-11-18 PSTBryce Harrington <bryce@bryceharrington.org>another idea might be to set up a mailing list in the new mailman3 and have the incoming emails from infra@inkscape.org go there for handling.
2019-02-03 11-35-03 PSTBryce Harrington <bryce@bryceharrington.org>5.102.147.13
2020-11-08 08-25-09 PSTBryce Harrington <bryce@bryceharrington.org>I will create a new node for you, and leave him off ssh
2020-11-08 08-45-00 PSTBryce Harrington <bryce@bryceharrington.org>I don't usually leave chat.inkscape.org up and running, plus if we're in the midst of an upgrade we may not be able to chat that way anyway
2019-02-01 23-16-49 PSTBryce Harrington <bryce@bryceharrington.org>error.rocketchat.2019-02-01.txt (https://chat.inkscape.org/file-upload/kz47F6WuL6NxFkYjE/error.rocketchat.2019-02-01.txt)
2018-12-27 01-50-53 PSTBryce Harrington <bryce@bryceharrington.org>yes, no problem
2019-02-03 11-39-13 PSTBryce Harrington <bryce@bryceharrington.org>ah
2018-09-27 12-13-19 PDTBryce Harrington <bryce@bryceharrington.org>similarly, we have our main web server at OSUOSL, but figure keeping another 4G in reserve in case we had to some day self-host that, might also be smart
2018-09-27 12-14-11 PDTBryce Harrington <bryce@bryceharrington.org>oh, how are they being designed differently?
2018-09-27 12-13-43 PDTBryce Harrington <bryce@bryceharrington.org>however I haven't penciled in either of those in the file, since they're just speculative for now
2020-09-09 15-18-06 PDTBryce Harrington <bryce@bryceharrington.org>I thought of two options, one would be to set up a new VM entirely with all new stuff, and migrate the data to it. The other option would be to shut down the chat service, back it up really well, and then attempt to upgrade in place
2018-09-27 12-11-03 PDTBryce Harrington <bryce@bryceharrington.org>ian's mailman3 work has been against ubuntu 16.04, so best to keep that isolated for now, at least until it can be moved to ubuntu 18.04
2019-02-03 11-42-46 PSTBryce Harrington <bryce@bryceharrington.org>do you want to walk through it's creation to learn, or shall I set it up?
2020-09-09 15-15-26 PDTBryce Harrington <bryce@bryceharrington.org>my area of Oregon is currently on fire, and I may need to evacuate (we're packing in any case) so I likely will be unavailable at least for a week
2018-09-27 12-09-46 PDTBryce Harrington <bryce@bryceharrington.org>however I feel we need a better thought out strategy on how to divide things
2020-11-08 11-29-55 PSTBryce Harrington <bryce@bryceharrington.org>I thought my provisioning script automatically configured /etc/sudoers for passwordless logins but I see it's still a manual config step
2020-11-08 11-30-10 PSTBryce Harrington <bryce@bryceharrington.org>which I've done now.
2020-08-21 09-57-50 PDTBryce Harrington <bryce@bryceharrington.org>I also installed logwatch, but it pulled in postfix so I purged both. Hopefully that doesn't disrupt anything mail-wise
2019-02-03 17-09-06 PSTBryce Harrington <bryce@bryceharrington.org>I can establish email redirects via gandi, which would be better than using our own personal email accounts, but it still would just go to one person.
2018-12-27 02-00-01 PSTBryce Harrington <bryce@bryceharrington.org>ah ok
2019-02-03 11-34-11 PSTBryce Harrington <bryce@bryceharrington.org>hi
2020-08-21 09-56-32 PDTBryce Harrington <bryce@bryceharrington.org>ok
2018-12-27 01-48-07 PSTBryce Harrington <bryce@bryceharrington.org>okay. Also fwiw, in doing the update to alpha, and running the meltdownchecker script before and after, the difference is modest
2019-02-03 11-41-26 PSTBryce Harrington <bryce@bryceharrington.org>https://gitlab.com/inkscape/infra/credentials
2018-09-27 12-23-56 PDTBryce Harrington <bryce@bryceharrington.org>credentials/services.txt has a comprehensive listing of services we currently provide, or that are provided to us, and includes some run externally by 3rd parties which we need to be aware of.
2020-09-09 15-18-45 PDTBryce Harrington <bryce@bryceharrington.org>your call on what strategy to take. If I'm not evacuated tonight then I can try to at least get things set up for you
2020-11-08 08-46-43 PSTBryce Harrington <bryce@bryceharrington.org>comments on the gitlab ticket also reach me reasonably quickly
2020-08-21 09-58-44 PDTBryce Harrington <bryce@bryceharrington.org>for the security fix, are you planning on patching, or do a full update of rocket.chat to a more current version?
2018-09-27 11-59-01 PDTBryce Harrington <bryce@bryceharrington.org>hiya
2019-02-03 11-49-51 PSTBryce Harrington <bryce@bryceharrington.org>checking...
2019-02-03 11-40-02 PSTBryce Harrington <bryce@bryceharrington.org>ok I wondered about that
2019-02-03 11-36-59 PSTBryce Harrington <bryce@bryceharrington.org>correct
2019-02-03 11-48-49 PSTBryce Harrington <bryce@bryceharrington.org>it sounds like yaron knows ansible and would be interested in helping, and he's been contributing to the project for a long time. it may be beneficial to involve him a little
2019-02-03 11-34-28 PSTBryce Harrington <bryce@bryceharrington.org>yes
2018-12-27 01-57-48 PSTBryce Harrington <bryce@bryceharrington.org>yes
2020-11-08 08-24-03 PSTBryce Harrington <bryce@bryceharrington.org>he had requested ssh access to the machine "as backup". I did not give him permission to go ahead and start the work, certainly not without coordinating with you.
2018-09-27 12-06-31 PDTBryce Harrington <bryce@bryceharrington.org>I'm thinking for production machines we may wish some to have more than 2G memory, so that reduces the total. So I'm figuring maybe 6-7 production machines
2018-09-27 12-05-10 PDTBryce Harrington <bryce@bryceharrington.org>some of those we want to provide for folks to use for development, a couple we need to reserve for staging purposes, and so we can install services across the remaining ones
2020-08-21 14-44-42 PDTBryce Harrington <bryce@bryceharrington.org>no, whatever the problem was had happened prior to the dist-upgrade
2020-11-06 13-12-10 PSTBryce Harrington <bryce@bryceharrington.org>cool sounds good
2019-02-03 11-38-46 PSTBryce Harrington <bryce@bryceharrington.org>what was wrong with it?
2019-02-03 17-09-54 PSTBryce Harrington <bryce@bryceharrington.org>I noticed that gitlab allows email-based bug submissions, so maybe one idea would be to set up a redirect that points at a private gitlab subgroup issue tracker
2019-02-03 11-51-40 PSTBryce Harrington <bryce@bryceharrington.org>btw, the mailman3 ansible includes a gen_credentials.py script to make random passwords for testing purposes
2019-02-03 17-10-33 PSTBryce Harrington <bryce@bryceharrington.org>only problem might be that it opens a new bug for every email to that address, and that could be a spam annoyance.
2018-12-26 15-13-25 PSTBryce Harrington <bryce@bryceharrington.org>I dist-upgraded alpha.inkscape.org. Any concerns if I do the same on chat.inkscape.org?
2018-09-27 12-08-40 PDTBryce Harrington <bryce@bryceharrington.org>I figure one production machine should be dedicated to backup, and only backup
2018-09-27 12-16-26 PDTBryce Harrington <bryce@bryceharrington.org>yes
2019-02-03 17-11-35 PSTBryce Harrington <bryce@bryceharrington.org>maybe you have some other ideas to consider?
2018-09-27 12-06-52 PDTBryce Harrington <bryce@bryceharrington.org>I'm betting we'll end up with more than 6-7 individual services, so we'll need to have some lesser services piggyback on the same vm
2020-08-21 09-57-11 PDTBryce Harrington <bryce@bryceharrington.org>I did a dist-upgrade (since it's not been done in a while) but noticed some error with mongodb when you rebooted it today so didn't want to reboot and break things
2018-09-27 12-12-45 PDTBryce Harrington <bryce@bryceharrington.org>I've thought we should keep at least 4G reserved for in case we need to start self-hosting gitlab. That's not planned for now; we're doing fine with their existing service. But we may need something a bit beefy if we have to go that route. I don't know
2018-12-27 01-51-09 PSTBryce Harrington <bryce@bryceharrington.org>ahh ok I was wondering about that status
2020-11-06 13-12-23 PSTBryce Harrington <bryce@bryceharrington.org>I should be around
2019-02-03 11-42-02 PSTBryce Harrington <bryce@bryceharrington.org>there is a README for how to create new service credentials, so me or you can create one for rocketchat admin
2020-11-08 08-46-19 PSTBryce Harrington <bryce@bryceharrington.org>I for synchronous chat, I'm on IRC freenode as #bryce (I hang out on #ubuntu-devel) however I'm not always watching that (work hours only mostly).
2018-09-27 12-15-42 PDTBryce Harrington <bryce@bryceharrington.org>yes
2018-09-27 12-05-23 PDTBryce Harrington <bryce@bryceharrington.org>you can see how I split things out in the file in the services repo
2019-02-03 17-08-19 PSTBryce Harrington <bryce@bryceharrington.org>like maybe infra@inkscape.org or something
2019-02-03 11-50-44 PSTBryce Harrington <bryce@bryceharrington.org>the mailman3 ansible scripts do set up letsencrypt
2020-09-09 15-14-54 PDTBryce Harrington <bryce@bryceharrington.org>hi recena, yes you're right
2018-12-27 01-44-08 PSTBryce Harrington <bryce@bryceharrington.org>none, just an item on my todo list to keep the systems upgraded.
2019-02-03 11-43-37 PSTBryce Harrington <bryce@bryceharrington.org>ok, I also did an apt-get upgrade on the host, so it's got system updates too. I did not do a dist-upgrade so it's not 100% up to date
2020-11-08 08-23-14 PSTBryce Harrington <bryce@bryceharrington.org>https://gitlab.com/inkscape/infra/services/-/issues/43#note_437576404
2018-09-27 12-17-02 PDTBryce Harrington <bryce@bryceharrington.org>cloudscale provides a way to scale the resources, so starting with 2G and moving up should be feasible. I haven't tried that out yet though.
2019-02-03 11-44-22 PSTBryce Harrington <bryce@bryceharrington.org>Ian has been a lot more active with mailman3 work, and we're nearly done
2020-11-18 00-22-04 PSTBryce Harrington <bryce@bryceharrington.org>hi recena, sorry again for being a blockage when you were working on the rocketchat upgrade previously. Let me know how I can help you.
2018-09-27 12-29-24 PDTBryce Harrington <bryce@bryceharrington.org>the main priorities have been: mailman3, forums, wiki. And the supporting services for those - backup, dns
2018-09-27 12-07-52 PDTBryce Harrington <bryce@bryceharrington.org>thus why I suggest putting rocketchat and weblate together one one machine, but if there is a better arrangement I'm open to thoughts
2019-02-03 11-37-47 PSTBryce Harrington <bryce@bryceharrington.org>but it has mailman3 overlaid on it, so the nginx config and probably other things are incorrect
2019-02-03 11-45-30 PSTBryce Harrington <bryce@bryceharrington.org>some of the remaining goofs could be addressed via documentation, so the backup info you're writing will help there.
2019-02-03 11-49-32 PSTBryce Harrington <bryce@bryceharrington.org>no, is that from mailman3?
2020-11-08 08-44-27 PSTBryce Harrington <bryce@bryceharrington.org>btw, my direct email is bryce@bryceharrington.org; I check this regularly when I'm online so that's the fastest way to contact me
2019-02-03 11-40-56 PSTBryce Harrington <bryce@bryceharrington.org>we have a GPG encrypted store for project credentials with controlled access for who can see it
2018-12-27 01-45-56 PSTBryce Harrington <bryce@bryceharrington.org>agreed. we don't have a mechanism to create a snapshot though
2020-11-06 13-06-53 PSTBryce Harrington <bryce@bryceharrington.org>hi recena, sorry again it took me a while to get the machine set up, just wanted to check in if you've had a chance to start work on the node, and if you ran into any other troubles I can help with
2019-02-03 11-44-41 PSTBryce Harrington <bryce@bryceharrington.org>yes it did, it saved my bacon. Good work
2018-09-27 12-26-57 PDTBryce Harrington <bryce@bryceharrington.org>however, I can tell there are going to be a lot of unexpected services needed. Both rocketchat and weblate came up as requests only just recently, and are in neither of those lists.
2020-08-21 09-56-36 PDTBryce Harrington <bryce@bryceharrington.org>heya :-)