Skip to content

Commit 47ede67

Browse files
authored
Fix the healthcheck in the new EKS environment. (#283)
Signed-off-by: Irving Popovetsky <irving@popovetsky.com>
1 parent 459ae19 commit 47ede67

5 files changed

Lines changed: 93 additions & 15 deletions

File tree

OPS.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Deploying a new backend version
2+
3+
Once a release is built and deployed by CircleCI, deploy it to an environment using ArgoCD.
4+
5+
1. First, to connect to ArgoCD:
6+
```
7+
kubectl -n argocd port-forward service/argocd-server 8443:443 &
8+
open https://localhost:8443
9+
```
10+
2. login - credentials are in 1password, or ask someone for help
11+
3. pick up the new version in staging.
12+
- go to https://localhost:8443/applications/pyback-staging,
13+
- click the hamburger menu (3 dots, blue button), -> Details -> Parameters
14+
- update the images field with the build ID as the tag, like: `operationcode/back-end:staging-846`
15+
- as the new pods deploy, tail their logs to check for errors
16+
- validate the staging environment (notes below)
17+
4. repeat those steps for the production environment
18+
19+
# Validating the staging environment
20+
21+
This requires a working node or docker environment. I found docker to be easier and more reliable but that was me :shrug:
22+
23+
When you run the front-end repo in localdev mode, it automatically connects to the staging environment.
24+
1. install dependencies: `docker run -it -v ${PWD}:/src -w /src node:lts yarn`
25+
2. run the dev server: `docker run -it -v ${PWD}:/src -w /src -p 127.0.0.1:3000:3000/tcp node:lts yarn dev --hostname 0.0.0.0`
26+
3. Connect to the dev server: `open http://localhost:3000`
27+
28+
# Certificate management with certbot
29+
30+
Certbot runs continously as a kube operator and refreshes certs for you. To ensure it is working,
31+
check the logs of the `cert-manager` pod, like:
32+
```
33+
kubectl -n cert-manager logs -f cert-manager-dcc48bf99-skhn7
34+
```
35+
36+
Current version running is v0.10.1
37+
38+
if you need for some reason to upgrade:
39+
1. read the release notes for all versions between current and desired, watch for breaking changes
40+
2. ignore the instructions about helm and kubectly apply, one minor version at a time
41+
```
42+
kubectl apply \
43+
--validate=false \
44+
-f https://github.com/jetstack/cert-manager/releases/download/v0.10.1/cert-manager.yaml
45+
```
46+
47+
certificates installed:
48+
```
49+
$ kubectl get Certificates --all-namespaces
50+
NAMESPACE NAME READY SECRET AGE
51+
monitoring grafana-tls True grafana-tls 299d
52+
operationcode-staging back-end-tls True back-end-tls 264d
53+
operationcode-staging resources-api-tls True resources-api-tls 299d
54+
operationcode back-end-tls True back-end-tls 264d
55+
operationcode resources-api-tls True resources-api-tls 299d
56+
```
57+

poetry.lock

Lines changed: 29 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ requests = "^2.21"
3737
sentry-sdk = "^0.10.1"
3838
six = "^1.12"
3939
honeycomb-beeline = "^2.11.4"
40+
django-allow-cidr = "^0.3.1"
4041

4142
[tool.poetry.dev-dependencies]
4243
bandit = "^1.6"

src/settings/environments/production.py

Lines changed: 3 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,9 @@
1010
ALLOWED_HOSTS += [s.strip() for s in os.environ["EXTRA_HOSTS"].split(",")]
1111

1212
# Needed for AWS health check
13-
try:
14-
import socket
15-
16-
local_ip = str(socket.gethostbyname(socket.gethostname()))
17-
ALLOWED_HOSTS.append(local_ip)
18-
except Exception as ex: # pragma: no cover
19-
print(ex)
13+
if "allow_cidr.middleware.AllowCIDRMiddleware" not in MIDDLEWARE: # noqa: F821
14+
MIDDLEWARE += ("allow_cidr.middleware.AllowCIDRMiddleware",) # noqa: F821
15+
ALLOWED_CIDR_NETS = ["192.168.0.0/16", "10.0.0.0/8", "172.16.0.0/12", "100.64.0.0/10"]
2016

2117
DATABASES = {
2218
"default": {

src/settings/environments/staging.py

Lines changed: 3 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,9 @@
1010
ALLOWED_HOSTS += [s.strip() for s in os.environ["EXTRA_HOSTS"].split(",")]
1111

1212
# Needed for AWS health check
13-
try:
14-
import socket
15-
16-
local_ip = str(socket.gethostbyname(socket.gethostname()))
17-
ALLOWED_HOSTS.append(local_ip)
18-
except Exception as ex: # pragma: no cover
19-
print(ex)
13+
if "allow_cidr.middleware.AllowCIDRMiddleware" not in MIDDLEWARE: # noqa: F821
14+
MIDDLEWARE += ("allow_cidr.middleware.AllowCIDRMiddleware",) # noqa: F821
15+
ALLOWED_CIDR_NETS = ["192.168.0.0/16", "10.0.0.0/8", "172.16.0.0/12", "100.64.0.0/10"]
2016

2117
DATABASES = {
2218
"default": {

0 commit comments

Comments
 (0)