2020-08-21 12-44-59 PDT Manuel Recena When I rebooted the system everything around Rocket Chat was working fine. 2018-12-26 15-38-25 PST Manuel Recena Let me check it 2018-09-27 13-05-58 PDT Manuel Recena By the way, it would be great to know who is part of the infrastructure team ;) 2019-02-03 11-38-07 PST Manuel Recena Let me back to the mailman3 subject later. 2018-09-27 12-22-26 PDT Manuel Recena Is there any place where I can see the services that we want to set up? 2018-12-27 01-50-53 PST Manuel Recena By the way, Weblate is ready to deploy on production 2018-09-27 12-05-01 PDT Manuel Recena Ahh perfect 2018-09-27 13-02-32 PDT Manuel Recena I am here for helping 2020-11-08 04-04-53 PST Manuel Recena Do you know if someone has manipulated this VM(5.102.146.198)? 2019-02-03 11-46-21 PST Manuel Recena ok 2020-11-06 13-10-55 PST Manuel Recena I will spend time tomorrow. I hope to get good news. 2018-12-27 01-57-56 PST Manuel Recena In fact, we should deploy the solution over the Christmas 2020-11-08 04-04-16 PST Manuel Recena FYI https://gitlab.com/inkscape/infra/services/-/issues/43#note_443680262 2018-12-27 01-44-48 PST Manuel Recena Ok, we can do it but a backup would be recommended 2018-09-27 12-05-54 PDT Manuel Recena It is fine. I just wanted to confirm it 2018-12-27 01-49-14 PST Manuel Recena My main concern is about mongo 2018-09-27 12-05-26 PDT Manuel Recena I am assuming that each service will be deployed into a dedicated machine 2018-09-27 12-16-57 PDT Manuel Recena For that reason, it is better small machines (2Gb) with only one service 2019-02-03 11-53-22 PST Manuel Recena My concern is with credentials related with mongo, mailjet, etc... 2019-02-03 11-41-33 PST Manuel Recena That is great 2018-12-27 01-50-33 PST Manuel Recena Let me investigate a bit, ok? 2018-12-27 01-58-46 PST Manuel Recena It is just running the playbook 2018-12-27 01-45-17 PST Manuel Recena A snapshot of the server 2019-02-03 11-42-55 PST Manuel Recena Thank to this re-deployment we have updated some components ;) 2020-11-08 10-38-21 PST Manuel Recena But the Ansible Playbook wasn't designed to be run having two production environment. I have to perform some changes 2018-09-27 11-43-46 PDT Manuel Recena Hello! 2018-09-27 12-15-33 PDT Manuel Recena I can rework on it, but we need to define the organization firstly 2019-02-03 11-39-37 PST Manuel Recena https://gitlab.com/inkscape/infra/services/blob/master/ansible/inventories/i... 2018-12-27 01-49-36 PST Manuel Recena If we upgrade the OS many packages will be upgraded as well 2019-02-03 11-51-20 PST Manuel Recena ok 2019-02-03 11-38-28 PST Manuel Recena I had to fix the SMTP configuration 2020-11-08 10-58-24 PST Manuel Recena My user does not have a password and it is not a sudoers 2019-02-03 11-36-04 PST Manuel Recena ok 2020-09-10 23-42-06 PDT Manuel Recena Are you and family safe? 2019-02-03 11-42-08 PST Manuel Recena ok 2019-02-03 11-44-17 PST Manuel Recena But the new version are all of them compatible for Rocket.Chat 2020-11-08 10-36-53 PST Manuel Recena I am going to check the new VM 2019-02-03 11-44-01 PST Manuel Recena Node is update, Mongo is updated, Nginx is updated 2019-02-03 11-39-08 PST Manuel Recena the password was not configurared 2019-02-03 12-11-26 PST Manuel Recena letsencrypt is working fine 2020-08-21 12-46-16 PDT Manuel Recena I would like to perform a full refesh of the environment 2020-08-21 12-46-42 PDT Manuel Recena But it requires time and planning 2019-02-03 11-39-17 PST Manuel Recena because it is not in the configuration file 2019-02-03 11-44-55 PST Manuel Recena thanks. 2019-02-03 11-43-25 PST Manuel Recena 1) My plan is to define a backup for Rocket.Chat 2018-12-27 02-00-29 PST Manuel Recena I will keep you in touch 2018-09-27 11-43-59 PDT Manuel Recena I've just read your email 2018-09-27 12-14-44 PDT Manuel Recena One service, one machine 2018-09-27 12-20-03 PDT Manuel Recena I am going to finish first the playbook for Weblate, deploy it and them, we can discuss the next step 2018-09-27 11-44-18 PDT Manuel Recena I hope to reply to you later 2020-08-21 12-45-43 PDT Manuel Recena IMHO, that dist-upgrade was the root cause of the outage 2019-02-03 11-40-35 PST Manuel Recena when they are consumed from Ansible 2020-08-21 09-53-12 PDT Manuel Recena Hello 2020-11-08 10-39-40 PST Manuel Recena When we are able to shutdown the current production environment? 2019-02-03 11-43-34 PST Manuel Recena 2) Finish Weblate 2019-02-03 11-40-29 PST Manuel Recena I would like to improve how we are managing the passwords and tokens 2019-02-03 11-34-34 PST Manuel Recena Which IP? 2019-02-03 11-46-33 PST Manuel Recena I want to write it and automate it 2019-02-03 11-34-02 PST Manuel Recena Hello Bryce 2019-02-03 11-51-36 PST Manuel Recena I'm checking if everyting is fine on Rocket.Chat 2020-09-06 23-37-39 PDT Manuel Recena Hello! We should move forward the issue on Gitlab about upgrading RocketChat 2019-02-03 11-43-05 PST Manuel Recena I can do it. 2020-08-21 12-47-44 PDT Manuel Recena I will create a proper issue to address this topic 2019-02-03 11-44-32 PST Manuel Recena I think the Ansible Playbook worked very well. 2019-02-03 11-34-22 PST Manuel Recena Is the previous env avaialble? 2020-08-21 09-54-31 PDT Manuel Recena But in advance, I didn't touch anything, just restarting the server after facing the issue 2020-08-21 09-53-56 PDT Manuel Recena I didn't have a chance for participating on the gitlab issue regarding to Rocket.chat service... 2019-02-03 11-36-22 PST Manuel Recena that server is the previous Rocket.Chat, right? 2019-02-03 11-48-35 PST Manuel Recena Have you configured something related to letsencrypt? 2019-02-03 11-38-21 PST Manuel Recena I want to make sure that everything is working fine with Rocket.Chat 2018-12-27 01-57-35 PST Manuel Recena We needed to define the procedure to use the system 2018-09-27 12-28-38 PDT Manuel Recena I don't have any problem about maintaining all the infrastructure, but I need to know the requirements and which thinks I can decide 2018-09-27 12-02-45 PDT Manuel Recena I don't know if I understood correctly from your email 2018-12-27 02-03-07 PST Manuel Recena By the way, Happy Christmas! 2018-12-27 01-46-40 PST Manuel Recena But IMHO the base OS is enough updated 2018-09-27 12-03-14 PDT Manuel Recena But do we plan to have each service on a separated machine, right? 2020-11-08 10-57-50 PST Manuel Recena I'm blocked again. 2018-12-27 01-50-23 PST Manuel Recena And I am not sure a about the compatibility with Rocket.Chat 2018-12-27 01-43-28 PST Manuel Recena What is the motivation? 2018-09-27 12-13-33 PDT Manuel Recena The playbook are being designed in a different manner 2018-09-27 12-16-19 PDT Manuel Recena In any way, if a service grows up, we will need to scale the machine 2018-12-27 01-56-56 PST Manuel Recena From my side, it is done 2019-02-03 17-08-00 PST Bryce Harrington to avoid bus factor issues, it would probably make sense at some point for us to establish a team email that can be used for service registrations and such 2020-09-13 21-01-25 PDT Bryce Harrington Hi recena, yes we're safe. The county adjacent to us had to evacuate and we were on the cusp, but they got the fire near us under control yesterday, and we've not had to actually evacuate. We did pack everything up though. Hard to go through all your possessions and prioritize what small bit you want to preserve. I feel really bad for people around here who have lost their houses, and thankful the fire crews are working so hard. 2018-09-27 12-04-28 PDT Bryce Harrington the sponsored hosting provides us with up to 32G total memory for VMs. We can divide the memory up by 2G increments, thus we have an effective maximum of 16 vms 2018-09-27 12-09-33 PDT Bryce Harrington services/cloudscale_nodes.txt is where my current thoughts on purposes are 2018-12-26 15-09-03 PST Bryce Harrington heya 2020-11-08 08-36-47 PST Bryce Harrington It's at 5.102.146.198 I've set up ssh root access for you and I 2018-12-27 01-58-15 PST Bryce Harrington how? 2020-12-30 10-09-43 PST Bryce Harrington hi recena, I am guessing your time hasn't freed up much. I hope everything is ok. I've got a bit of time over the holidays but it's going to tighten up again after new years. On https://gitlab.com/inkscape/infra/services/-/issues/43 Mc had expressed interest in handling the rocketchat upgrade. Later today I'm going to give Mc root access to the machines so he can do the task, unless you tell me otherwise. 2018-12-27 02-06-45 PST Bryce Harrington Happy Christmas to you too! 2018-09-27 12-26-11 PDT Bryce Harrington the most definitive list of services we need to host, was in the original call for hosting, which I think is posted on the main website. Should be googleable I think. 2019-02-03 17-11-18 PST Bryce Harrington another idea might be to set up a mailing list in the new mailman3 and have the incoming emails from infra@inkscape.org go there for handling. 2019-02-03 11-35-03 PST Bryce Harrington 5.102.147.13 2020-11-08 08-25-09 PST Bryce Harrington I will create a new node for you, and leave him off ssh 2020-11-08 08-45-00 PST Bryce Harrington I don't usually leave chat.inkscape.org up and running, plus if we're in the midst of an upgrade we may not be able to chat that way anyway 2019-02-01 23-16-49 PST Bryce Harrington error.rocketchat.2019-02-01.txt (https://chat.inkscape.org/file-upload/kz47F6WuL6NxFkYjE/error.rocketchat.201...) 2018-12-27 01-50-53 PST Bryce Harrington yes, no problem 2019-02-03 11-39-13 PST Bryce Harrington ah 2018-09-27 12-13-19 PDT Bryce Harrington similarly, we have our main web server at OSUOSL, but figure keeping another 4G in reserve in case we had to some day self-host that, might also be smart 2018-09-27 12-14-11 PDT Bryce Harrington oh, how are they being designed differently? 2018-09-27 12-13-43 PDT Bryce Harrington however I haven't penciled in either of those in the file, since they're just speculative for now 2020-09-09 15-18-06 PDT Bryce Harrington I thought of two options, one would be to set up a new VM entirely with all new stuff, and migrate the data to it. The other option would be to shut down the chat service, back it up really well, and then attempt to upgrade in place 2018-09-27 12-11-03 PDT Bryce Harrington ian's mailman3 work has been against ubuntu 16.04, so best to keep that isolated for now, at least until it can be moved to ubuntu 18.04 2019-02-03 11-42-46 PST Bryce Harrington do you want to walk through it's creation to learn, or shall I set it up? 2020-09-09 15-15-26 PDT Bryce Harrington my area of Oregon is currently on fire, and I may need to evacuate (we're packing in any case) so I likely will be unavailable at least for a week 2018-09-27 12-09-46 PDT Bryce Harrington however I feel we need a better thought out strategy on how to divide things 2020-11-08 11-29-55 PST Bryce Harrington I thought my provisioning script automatically configured /etc/sudoers for passwordless logins but I see it's still a manual config step 2020-11-08 11-30-10 PST Bryce Harrington which I've done now. 2020-08-21 09-57-50 PDT Bryce Harrington I also installed logwatch, but it pulled in postfix so I purged both. Hopefully that doesn't disrupt anything mail-wise 2019-02-03 17-09-06 PST Bryce Harrington I can establish email redirects via gandi, which would be better than using our own personal email accounts, but it still would just go to one person. 2018-12-27 02-00-01 PST Bryce Harrington ah ok 2019-02-03 11-34-11 PST Bryce Harrington hi 2020-08-21 09-56-32 PDT Bryce Harrington ok 2018-12-27 01-48-07 PST Bryce Harrington okay. Also fwiw, in doing the update to alpha, and running the meltdownchecker script before and after, the difference is modest 2019-02-03 11-41-26 PST Bryce Harrington https://gitlab.com/inkscape/infra/credentials 2018-09-27 12-23-56 PDT Bryce Harrington credentials/services.txt has a comprehensive listing of services we currently provide, or that are provided to us, and includes some run externally by 3rd parties which we need to be aware of. 2020-09-09 15-18-45 PDT Bryce Harrington your call on what strategy to take. If I'm not evacuated tonight then I can try to at least get things set up for you 2020-11-08 08-46-43 PST Bryce Harrington comments on the gitlab ticket also reach me reasonably quickly 2020-08-21 09-58-44 PDT Bryce Harrington for the security fix, are you planning on patching, or do a full update of rocket.chat to a more current version? 2018-09-27 11-59-01 PDT Bryce Harrington hiya 2019-02-03 11-49-51 PST Bryce Harrington checking... 2019-02-03 11-40-02 PST Bryce Harrington ok I wondered about that 2019-02-03 11-36-59 PST Bryce Harrington correct 2019-02-03 11-48-49 PST Bryce Harrington it sounds like yaron knows ansible and would be interested in helping, and he's been contributing to the project for a long time. it may be beneficial to involve him a little 2019-02-03 11-34-28 PST Bryce Harrington yes 2018-12-27 01-57-48 PST Bryce Harrington yes 2020-11-08 08-24-03 PST Bryce Harrington he had requested ssh access to the machine "as backup". I did not give him permission to go ahead and start the work, certainly not without coordinating with you. 2018-09-27 12-06-31 PDT Bryce Harrington I'm thinking for production machines we may wish some to have more than 2G memory, so that reduces the total. So I'm figuring maybe 6-7 production machines 2018-09-27 12-05-10 PDT Bryce Harrington some of those we want to provide for folks to use for development, a couple we need to reserve for staging purposes, and so we can install services across the remaining ones 2020-08-21 14-44-42 PDT Bryce Harrington no, whatever the problem was had happened prior to the dist-upgrade 2020-11-06 13-12-10 PST Bryce Harrington cool sounds good 2019-02-03 11-38-46 PST Bryce Harrington what was wrong with it? 2019-02-03 17-09-54 PST Bryce Harrington I noticed that gitlab allows email-based bug submissions, so maybe one idea would be to set up a redirect that points at a private gitlab subgroup issue tracker 2019-02-03 11-51-40 PST Bryce Harrington btw, the mailman3 ansible includes a gen_credentials.py script to make random passwords for testing purposes 2019-02-03 17-10-33 PST Bryce Harrington only problem might be that it opens a new bug for every email to that address, and that could be a spam annoyance. 2018-12-26 15-13-25 PST Bryce Harrington I dist-upgraded alpha.inkscape.org. Any concerns if I do the same on chat.inkscape.org? 2018-09-27 12-08-40 PDT Bryce Harrington I figure one production machine should be dedicated to backup, and only backup 2018-09-27 12-16-26 PDT Bryce Harrington yes 2019-02-03 17-11-35 PST Bryce Harrington maybe you have some other ideas to consider? 2018-09-27 12-06-52 PDT Bryce Harrington I'm betting we'll end up with more than 6-7 individual services, so we'll need to have some lesser services piggyback on the same vm 2020-08-21 09-57-11 PDT Bryce Harrington I did a dist-upgrade (since it's not been done in a while) but noticed some error with mongodb when you rebooted it today so didn't want to reboot and break things 2018-09-27 12-12-45 PDT Bryce Harrington I've thought we should keep at least 4G reserved for in case we need to start self-hosting gitlab. That's not planned for now; we're doing fine with their existing service. But we may need something a bit beefy if we have to go that route. I don't know 2018-12-27 01-51-09 PST Bryce Harrington ahh ok I was wondering about that status 2020-11-06 13-12-23 PST Bryce Harrington I should be around 2019-02-03 11-42-02 PST Bryce Harrington there is a README for how to create new service credentials, so me or you can create one for rocketchat admin 2020-11-08 08-46-19 PST Bryce Harrington I for synchronous chat, I'm on IRC freenode as #bryce (I hang out on #ubuntu-devel) however I'm not always watching that (work hours only mostly). 2018-09-27 12-15-42 PDT Bryce Harrington yes 2018-09-27 12-05-23 PDT Bryce Harrington you can see how I split things out in the file in the services repo 2019-02-03 17-08-19 PST Bryce Harrington like maybe infra@inkscape.org or something 2019-02-03 11-50-44 PST Bryce Harrington the mailman3 ansible scripts do set up letsencrypt 2020-09-09 15-14-54 PDT Bryce Harrington hi recena, yes you're right 2018-12-27 01-44-08 PST Bryce Harrington none, just an item on my todo list to keep the systems upgraded. 2019-02-03 11-43-37 PST Bryce Harrington ok, I also did an apt-get upgrade on the host, so it's got system updates too. I did not do a dist-upgrade so it's not 100% up to date 2020-11-08 08-23-14 PST Bryce Harrington https://gitlab.com/inkscape/infra/services/-/issues/43#note_437576404 2018-09-27 12-17-02 PDT Bryce Harrington cloudscale provides a way to scale the resources, so starting with 2G and moving up should be feasible. I haven't tried that out yet though. 2019-02-03 11-44-22 PST Bryce Harrington Ian has been a lot more active with mailman3 work, and we're nearly done 2020-11-18 00-22-04 PST Bryce Harrington hi recena, sorry again for being a blockage when you were working on the rocketchat upgrade previously. Let me know how I can help you. 2018-09-27 12-29-24 PDT Bryce Harrington the main priorities have been: mailman3, forums, wiki. And the supporting services for those - backup, dns 2018-09-27 12-07-52 PDT Bryce Harrington thus why I suggest putting rocketchat and weblate together one one machine, but if there is a better arrangement I'm open to thoughts 2019-02-03 11-37-47 PST Bryce Harrington but it has mailman3 overlaid on it, so the nginx config and probably other things are incorrect 2019-02-03 11-45-30 PST Bryce Harrington some of the remaining goofs could be addressed via documentation, so the backup info you're writing will help there. 2019-02-03 11-49-32 PST Bryce Harrington no, is that from mailman3? 2020-11-08 08-44-27 PST Bryce Harrington btw, my direct email is bryce@bryceharrington.org; I check this regularly when I'm online so that's the fastest way to contact me 2019-02-03 11-40-56 PST Bryce Harrington we have a GPG encrypted store for project credentials with controlled access for who can see it 2018-12-27 01-45-56 PST Bryce Harrington agreed. we don't have a mechanism to create a snapshot though 2020-11-06 13-06-53 PST Bryce Harrington hi recena, sorry again it took me a while to get the machine set up, just wanted to check in if you've had a chance to start work on the node, and if you ran into any other troubles I can help with 2019-02-03 11-44-41 PST Bryce Harrington yes it did, it saved my bacon. Good work 2018-09-27 12-26-57 PDT Bryce Harrington however, I can tell there are going to be a lot of unexpected services needed. Both rocketchat and weblate came up as requests only just recently, and are in neither of those lists. 2020-08-21 09-56-36 PDT Bryce Harrington heya :-)