Merge pull request #79 from microsoft/brian/updated-devenv

Developer environment improvements

Author: ndrix

docs/install.md

@@ -25,6 +25,11 @@ Ensure you have the following installed:
 
 ### **1️⃣ Local Deployment**
 
+> [!WARNING]
+> These local deployment instructions are not actively maintained and do not reflect the current structure of
+> the repository. It is recommended that you instead follow the devenv setup instructions from the `dusseldorf/`
+> directory [here](https://github.com/microsoft/dusseldorf/tree/main/dusseldorf).
+
 For detailed local deployment instructions, navigate to the `docs/local` directory and refer to the `Readme.md` file.
 
 #### **Step 1: Clone the Repository**

docs/local/Readme.md

@@ -1,5 +1,10 @@
 # **Dusseldorf Local Deployment Guide**
 
+> [!WARNING]
+> These local deployment instructions are not actively maintained and do not reflect the current structure of
+> the repository. It is recommended that you instead follow the devenv setup instructions from the `dusseldorf/`
+> directory [here](https://github.com/microsoft/dusseldorf/tree/main/dusseldorf).
+
 This guide provides instructions for deploying the Dusseldorf system locally using Docker Compose.
 
 ---

dusseldorf/.dockerignore

@@ -0,0 +1 @@
+**/node_modules

dusseldorf/README.md

@@ -2,31 +2,68 @@
 
 This folder contains all the components that make up the total platform.  These include:
 
-
 - `api`: the control plane / API.
 - `listener.dns`:  the DNS listener for the data plane.
 - `listener.http`: the HTTP/HTTPS listener in the data plane.
 - `ui`: the source for the frontend web user interface.
 - `zentralbibliothek`: a central library for listeners in the data plane.
 
-Each component is explained in more details below:
+## Developer Environment Setup
+
+### Prerequisites
+Please ensure that you have the following before starting:
+- An Azure AD tenant
+- An up-to-date version of Docker & Docker Compose
+- A terminal that supports bash (Note: If you are on Windows, it's recommended you use Ubuntu WSL and have your docker compose environment stored there.)
+
+### Setup
+In your Azure AD tenant, [create an Azure AD application](https://portal.azure.com/#view/Microsoft_AAD_RegisteredApps/CreateApplicationBlade/quickStartType~/null/isMSAApp~/false) with `http://localhost:8080/ui/` set as a redirect URI and `Single-page Application (SPA)` selected as the platform. Note the client ID and tenant ID of the newly created application.
+
+Then, clone this repository and enter the `dusseldorf/` directory (the one with this README in it) inside of a Linux-based terminal (if on Windows, use WSL). Then, execute `./generate_devenv.sh` and input your client ID and tenant ID obtained from Azure.
+
+To start your developer environment, ensure you have nothing running locally on port 8080 and run `docker compose up`. After it builds the containers, your environment should be up and running on `http://localhost:8080/ui/` and you should be able to login.
+
+### Testing
+
+Once your developer environment is running, the following services will be accessible on your computer:
+
+- **API & UI server**: The API will be up and running on port 8080
+- **HTTPS listener**: The HTTPS listener runs on port 443 by default. This is modifiable in your .env file, but note that it will be overwritten by the generate devenv script.
+- **DNS listener**: The DNS listener runs on port 10053.
+
+The API and UI are directly testable on http://localhost:8080/ui/ and you should be able to access them by logging in with an account on your Azure tenant that has access to the application.
+
+To test the HTTPS listener, you can modify the following curl command:
+
+```sh
+curl https://localhost:443/your-path-here -k \
+  -H "Host: your-zone-here.dusseldorf.local"
+```
+
+Make sure to keep the `-k` flag, which allows you to run the request without having to install the certificate. The Host header can be modified to be whatever host you are targeting to make a request to
+
+To test the DNS listener, install the `dig` command in your local environment and use `dig your-zone-here.dusseldorf.local +tcp @localhost -p 10053` to make requests to the DNS server.
+
+## Components
+
+Here is some more information about the different components of this directory:
 
-## API
+### API
 This is the control plane of Dusseldorf.  This REST API allows an authenticated user to create, manage and delete zones, monitor the requests made to them and manage rules set on their zones. Calls to this API must be authenticated using an authentication token in each request.  
 
 This API is implemented using the FastAPI framework and provide CRUD functionality on zones, rules by interacting with a MongoDB database.  
 
 For more information about this API, please consult the [API docs](api/README.md).
 
 
-## DNS Listener
+### DNS Listener
 This is a network listener that responds to DNS traffic, and listens on port 53/udp for any DNS requests.   The DNS (Domain Name System) protocol is responsible for translating hostnames to an IP address.  By acting as a domains' DNS server, Dusseldorf is able to monitor any name resolutions for those subdomains on the Internet.
 By default, this permissive DNS server reponds with its own IP address(es) to a name resolution.  This behaviour can be changed by assigning rules to your zone.   Currently we only support custom `A`, `AAAA`, `CNAME` and `TXT` records.
 
 Check the documentation for the DNS listener [here](listener.dns/README.md).
 
 
-## HTTP/HTTPS Listener
+### HTTP/HTTPS Listener
 This network listener can be setup to listen on cleartext HTTP traffic, or it can listen on HTTPS traffic, if you have the correct certificates available.  For a cleartext setup, this listener accepts HTTP requests on port 80/tcp, and respond with default, or customized HTTP responses.
 Just like the DNS listener acts as a DNS server, this listener implements a very permissive HTTP server, or web server.  Using rules, you can set custom content, headers and status codes.
 
@@ -38,14 +75,14 @@ You can find more information on how to run this listener in the [HTTP documenta
 
 
 
-## UI
+### UI
 This is the frontend web user interface.  It is implemented in React using the amazing [Fluent2](https://fluent2.microsoft.design/) design framework and provides a modern single-page-application (SPA) to communicate with the Dusseldorf API.
 
 These static pages are compiled and placed into a folder in the API, so it references a local `/api` endpoints, but can be setup to call another API endpoint altogether, too.
 The user interface's information can be found [here](ui/README.md).
 
 
-## Zentralbibliothek
+### Zentralbibliothek
 This is a central library which holds the rule engine, database logic and utility functions.
 
 

dusseldorf/api/src/api/main.py

@@ -25,10 +25,11 @@
 from controllers.rules_controller import router as rules_router
 from controllers.zones_controller import router as zones_router
 
-if not os.environ.get("API_TLS_CRT_FILE"):
-    raise Exception("API_TLS_CRT_FILE not found in environment variables")
-if not os.environ.get("API_TLS_KEY_FILE"):
-    raise Exception("API_TLS_KEY_FILE not found in environment variables")
+if os.environ.get("ENVIRONMENT") != "development":
+    if not os.environ.get("API_TLS_CRT_FILE"):
+        raise Exception("API_TLS_CRT_FILE not found in environment variables")
+    if not os.environ.get("API_TLS_KEY_FILE"):
+        raise Exception("API_TLS_KEY_FILE not found in environment variables")
 
 dusseldorf = FastAPI()
 
@@ -69,5 +70,7 @@ async def ui_redirect():
 dusseldorf.mount("/api", app)
 dusseldorf.mount("/ui", StaticFiles(directory="./ui", html=True), name="ui")
 
-uvicorn.run(dusseldorf, host="0.0.0.0", port=int(os.environ.get("API_PORT", 10443)), ssl_keyfile=os.environ.get("API_TLS_KEY_FILE"), ssl_certfile=os.environ.get("API_TLS_CRT_FILE"))
-# uvicorn.run(dusseldorf, host="0.0.0.0", port=8000, ssl_keyfile="/app/key.pem", ssl_certfile="/app/cert.pem")
+if os.environ.get("ENVIRONMENT") == "development":
+    uvicorn.run(dusseldorf, host="0.0.0.0", port=int(os.environ.get("API_PORT", 10443)))
+else:
+    uvicorn.run(dusseldorf, host="0.0.0.0", port=int(os.environ.get("API_PORT", 10443)), ssl_keyfile=os.environ.get("API_TLS_KEY_FILE"), ssl_certfile=os.environ.get("API_TLS_CRT_FILE"))

dusseldorf/api_Dockerfile

@@ -14,7 +14,7 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
     PYTHONUNBUFFERED=1 \
     ENVIRONMENT=production
 # Install system dependencies
-RUN tdnf -y update && tdnf -y install build-essential curl ca-certificates
+RUN tdnf -y update && tdnf -y install build-essential curl ca-certificates python3-devel
 # Install Python dependencies
 COPY api/src/api/requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt

dusseldorf/compose.yml

@@ -0,0 +1,121 @@
+
+services:
+  mongodb:
+    image: mongo:6.0
+    container_name: mongodb
+    ports:
+      - "27017:27017"
+    volumes:
+      - ./env/mongo-data:/data/db
+      - ./env/mongo-scripts/init.js:/docker-entrypoint-initdb.d/init.js:ro
+    environment:
+      MONGO_INITDB_ROOT_USERNAME: ${MONGO_USERNAME}
+      MONGO_INITDB_ROOT_PASSWORD: ${MONGO_PASSWORD}
+      MONGO_INITDB_DATABASE: ${MONGO_DB_NAME}
+    security_opt:
+      - no-new-privileges:true
+    tmpfs:
+      - /tmp
+    networks:
+      - local_network
+
+  api-server:
+    build:
+      context: .
+      dockerfile: api_Dockerfile
+    container_name: api-server
+    ports:
+      - "8080:8080"
+    environment:
+      DSSLDRF_CONNSTR: "mongodb://${MONGO_USERNAME}:${MONGO_PASSWORD}@mongodb:27017/${MONGO_DB_NAME}"
+      API_VERSION: ${API_VERSION}
+      API_PORT: 8080
+      ENVIRONMENT: ${ENVIRONMENT}
+      AZURE_CLIENT_ID: ${AZURE_CLIENT_ID}
+      AZURE_TENANT_ID: ${AZURE_TENANT_ID}
+    depends_on:
+      - mongodb
+    security_opt:
+      - no-new-privileges:true
+    networks:
+      - local_network
+    develop:
+      watch:
+        - action: sync
+          path: ./api
+          target: /app/api
+
+  listener-http:
+    build:
+      context: .
+      dockerfile: http-listener_Dockerfile
+    container_name: listener-http
+    ports:
+      - "${LSTNER_HTTP_PORT}:${LSTNER_HTTP_PORT}"
+    environment:
+      DSSLDRF_CONNSTR: "mongodb://${MONGO_USERNAME}:${MONGO_PASSWORD}@mongodb:27017/${MONGO_DB_NAME}"
+      DSSLDRF_TLS_CRT_FILE: /certs/tls.crt
+      DSSLDRF_TLS_KEY_FILE: /certs/tls.key
+      LSTNER_HTTP_PORT: ${LSTNER_HTTP_PORT}
+      LSTNER_HTTP_INTERFACE: ${LSTNER_HTTP_INTERFACE}
+    volumes:
+      - ${DSSLDRF_TLS_CRT_FILE}:/certs/tls.crt
+      - ${DSSLDRF_TLS_KEY_FILE}:/certs/tls.key
+    depends_on:
+      - mongodb
+    security_opt:
+      - no-new-privileges:true
+    networks:
+      local_network:
+        ipv4_address: 172.18.0.9
+    develop:
+      watch:
+        - action: sync
+          path: ./listener.http/src
+          target: /dusseldorf
+
+
+  listener-dns:
+    build:
+      context: .
+      dockerfile: dns-listener_Dockerfile
+    container_name: listener-dns
+    ports:
+      - "10053:${LSTNER_DNS_PORT}"
+    environment:
+      DSSLDRF_CONNSTR: "mongodb://${MONGO_USERNAME}:${MONGO_PASSWORD}@mongodb:27017/${MONGO_DB_NAME}"
+      LSTNER_DNS_PORT: ${LSTNER_DNS_PORT}
+      LSTNER_DNS_INTERFACE: ${LSTNER_DNS_INTERFACE}
+      LSTNER_DNS_UDP: ${LSTNER_DNS_UDP}
+    depends_on:
+      - mongodb
+    security_opt:
+      - no-new-privileges:true
+    networks:
+      local_network:
+        ipv4_address: 172.18.0.10
+    develop:
+      watch:
+        - action: sync
+          path: ./listener.dns/src
+          target: /dusseldorf
+
+  # ui:
+  #   image: ${ACR_NAME}.azurecr.io/legacy-ui:latest
+  #   container_name: ui
+  #   ports:
+  #     - "3000:3000"
+  #   depends_on:
+  #     - api-server
+  #   security_opt:
+  #     - no-new-privileges:true
+  #   networks:
+  #     - local_network
+
+networks:
+  local_network:
+    driver: bridge
+    ipam:
+      config:
+        - subnet: 172.18.0.0/16
+          gateway: 172.18.0.1

dusseldorf/generate_devenv.sh

@@ -0,0 +1,74 @@
+#!/bin/bash
+
+set -e
+
+echo "We are going to need some information to set up your development environment."
+echo "----------------------------------------------------"
+echo "We need the client ID and tenant ID of your Azure AD application that you want to use for your developer environment."
+read -p "Azure Client ID (example: dc1b6b75-8167-4baf-9e75-d3d1f755de1b): " AZURE_CLIENT_ID
+read -p "Azure Tenant ID (example: 72f988bf-86f1-41af-91ab-2d7cd011db47): " AZURE_TENANT_ID
+echo "----------------------------------------------------"
+echo "Setting up with client ID: $AZURE_CLIENT_ID and tenant ID: $AZURE_TENANT_ID"
+
+mkdir -p env/
+mkdir -p env/certs/
+echo "Creating certificates..."
+openssl req -newkey rsa:2048 -nodes -keyout "env/certs/tls.key" -x509 -days 365 -out "env/certs/tls.crt" -subj "/CN=localhost"
+echo "Certificates created successfully."
+
+echo "Making mongo files..."
+mkdir -p env/mongo-data/
+mkdir -p env/mongo-scripts/
+MONGO_PASSWORD=$(openssl rand -hex 12)
+cat <<EOF > env/mongo-scripts/init.js
+db = db.getSiblingDB("dusseldorf");
+db.createUser({
+  user: "admin",
+  pwd: "$MONGO_PASSWORD",
+  roles: [{ role: "readWrite", db: "dusseldorf" }]
+});
+
+db.createCollection("domains");
+db.createCollection("zones");
+db.createCollection("requests");
+db.createCollection("rules");
+
+db.domains.createIndex({domain: 1}, { unique: true });
+db.zones.createIndex({fqdn: 1}, { unique: true });
+db.requests.createIndex({
+  "zone": 1,
+  "time": 1
+});
+db.domains.insertOne({"domain": "dusseldorf.local", "public_ips": ["172.18.0.9"], "owner": "dusseldorf"});
+EOF
+
+echo "Generating .env file..."
+cat <<EOF > .env
+# Dusseldorf Environment Variables
+API_VERSION=1
+ENVIRONMENT=development
+AZURE_CLIENT_ID=$AZURE_CLIENT_ID
+AZURE_TENANT_ID=$AZURE_TENANT_ID
+
+MONGO_USERNAME=admin
+MONGO_PASSWORD=$MONGO_PASSWORD
+MONGO_DB_NAME=dusseldorf
+MONGODB_DB_NAME=dusseldorf
+
+DSSLDRF_TLS_CRT_FILE=./env/certs/tls.crt
+DSSLDRF_TLS_KEY_FILE=./env/certs/tls.key
+LSTNER_HTTP_PORT=443
+LSTNER_HTTP_INTERFACE=0.0.0.0
+
+LSTNER_DNS_PORT=10053
+LSTNER_DNS_INTERFACE=0.0.0.0
+LSTNER_DNS_UDP=false
+EOF
+
+echo "Generating react .env file..."
+cat <<EOF > ui/.env
+REACT_APP_CLIENT_ID=$AZURE_CLIENT_ID
+REACT_APP_TENANT_ID=$AZURE_TENANT_ID
+REACT_APP_API_HOST=http://localhost:8080/api
+EOF
+echo "Environment setup complete. You can now start running Dusseldorf with docker compose up."

dusseldorf/ui/src/DusseldorfConfig.ts

@@ -9,12 +9,12 @@
 // This is the hostname of the API we talk to.  
 // To set a manual one, do the following
 // localStorage.setItem("api_host", "https://localhost:1337")
-const API_HOST = window.localStorage.getItem("api_host") ??  "/api";
+const API_HOST = process.env.REACT_APP_API_HOST ?? window.localStorage.getItem("api_host") ?? "/api";
 
 const config = {
     // https://ms.portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationMenuBlade/Overview/appId/dc1b6b75-8167-4baf-9e75-d3d1f755de1b/isMSAApp/
-    client_id:      "dc1b6b75-8167-4baf-9e75-d3d1f755de1b",
-    tenant_id:      "72f988bf-86f1-41af-91ab-2d7cd011db47",
+    client_id:      process.env.REACT_APP_CLIENT_ID ?? "dc1b6b75-8167-4baf-9e75-d3d1f755de1b",
+    tenant_id:      process.env.REACT_APP_TENANT_ID ?? "72f988bf-86f1-41af-91ab-2d7cd011db47",
     appInsightsId:  "3836b094-0c3a-42b1-a3b3-9a81133f64fb"
 }