Tech Notes: SWLA CI/CD Pipeline

The SWLA Records platform is a Flask application backed by MySQL on GoDaddy, deployed to Render.com with a GitHub Actions CI/CD pipeline. This post documents the deployment architecture and the pipeline setup — useful reference for anyone running a similar stack.

Stack overview

Backend
Flask + SQLAlchemy + PyMySQL
Frontend
Vanilla JS + Jinja2 + CSS
Database
MySQL on GoDaddy (swla_dev / swla_prod)
Hosting
Render.com (Web Service)
CI/CD
GitHub Actions
Repo
blackrageous/swla_repos

How the pipeline works

Every push to the main branch triggers a GitHub Actions workflow that runs tests and — if they pass — signals Render.com to pull the latest code and redeploy. Render handles the build, dependency installation, and service restart automatically. The two database environments (swla_dev and swla_prod) are kept separate via environment variables injected at runtime, never committed to the repo.

GitHub Actions workflow

# .github/workflows/deploy.yml name: Deploy to Render on: push: branches: – main jobs: deploy: runs-on: ubuntu-latest steps: – name: Checkout code uses: actions/checkout@v3 – name: Set up Python uses: actions/setup-python@v4 with: python-version: ‘3.11’ – name: Install dependencies run: pip install -r requirements.txt – name: Trigger Render deploy run: | curl -X POST \ -H “Authorization: Bearer ${{ secrets.RENDER_API_KEY }}” \ “https://api.render.com/v1/services/${{ secrets.RENDER_SERVICE_ID }}/deploys”
Key point: The Render API key and service ID are stored as GitHub repository secrets (Settings → Secrets and variables → Actions) — never hardcoded. The same pattern applies to DB_HOST, DB_USER, DB_PASSWORD, and DB_PORT, which Render injects as environment variables at runtime.

Environment variable management

The Flask app reads its database connection from environment variables using a .env file locally and Render’s environment variable dashboard in production. The --env flag on the import and linking scripts selects between swla_dev and swla_prod at runtime, making it safe to test destructive operations (like a full re-link run) in dev before touching prod.

# Example: run linker against dev only python scripts/link_hebert_census.py –env dev –race B MU # Push to prod only after dev results are satisfactory python scripts/link_hebert_census.py –env prod –race B MU

Render.com cold starts

On Render’s free tier, web services spin down after 15 minutes of inactivity and take 10–30 seconds to wake on the next request. For a research tool with intermittent traffic this is acceptable. The startup screen (“Service waking up…”) is shown to users while the container initializes. Upgrading to a paid Render instance eliminates cold starts if response time becomes a priority.


Python Flask GitHub Actions CI/CD Render.com MySQL SQLAlchemy

SWLA RECORDS PLATFORM

For most people researching Black and Creole family history in Louisiana, 1865 is a wall. Before emancipation, records become sparse, names disappear, and the paper trail that genealogists depend on simply stops. The question of who your family was before the Civil War — where they lived, what they were called, who baptized them — has been effectively unanswerable for generations of researchers.

The Louisiana Heritage Platform is an attempt to break that wall.


Three archives, one platform

The platform links three independent historical archives spanning 1719 to 1880, covering notarial acts, Catholic sacramental records, and US Census data for Louisiana. Together they contain over 1.4 million records — and more importantly, they overlap in ways that allow individuals to be traced across them.

Archive Years Records Content
Hall Archive 1719–1820 104,729 Notarial acts, sales, manumissions, inventories
Hebert Archive 1720–1865 390,687 Catholic parish sacramental records — baptisms, marriages, burials
1880 US Census 1880 935,068 Louisiana households — race, age, occupation, birthplace, parish
Total 1719–1880 1,430,484
1.4M+ Total records across three archives
6,176 Confirmed Hall ↔ Hebert cross-archive links
161 years Span of coverage, 1719 to 1880
1865 The barrier this platform is built to break

Cross-archive linking

The core of the platform isn’t the records themselves — it’s the links between them. A person baptized in the Hebert archive in 1830 might appear in a Hall notarial act in 1818 and in the 1880 Census as an elderly head of household. Connecting those three appearances requires matching across different record types, spelling variations, parish geographies, and time gaps measured in decades.

The linking algorithm scores candidate matches on name similarity (using Soundex and given name comparison), parish geography, and year proximity. Matches scoring 90% or above are flagged as strong; 65–89% as possible. The Hall ↔ Hebert linker is live in production with 6,176 confirmed links. The Hebert ↔ Census linker has produced 1,515 links in development and is being tuned before a production push.


Who it’s for

The platform is designed for genealogical researchers working on Black and Creole family history in Louisiana — specifically those whose research hits the pre-emancipation wall. The Hebert archive’s Catholic sacramental records are particularly valuable because they documented enslaved individuals and free people of color by name, decades before emancipation-era records begin.

The site is currently password-protected while development continues. If you’re a researcher interested in access, send an email to garland.joseph@gmail.com.


Visit the Louisiana Heritage Platform at swla-records.garlandjoseph.org

Open SWLA Records →

Genealogy Louisiana Black history Creole 1880 Census Hebert Archive Hall Archive

Auto-Discovery with NMAP

June 26th, 2017 | Garland Joseph

I created this script in order to use it as a feed for enterprise management tools such as Nagios.   It’s a bit difficult to get a handle on auto-discovery tools within an enterprise management tool when it discovers what can be an overwhelming number of hosts.  NMAP is smart enough to translate the MAC address into a vendor if it can.

This script called discover uses the NMAP tool on Linux (tested on Centos).


#! /bin/bash
#
# Name: discover
#
# Garland Joseph, garland.joseph@gmail.com
# Date: June 2017
#
# Auto-discover on subnet using nmap, can be fed into something like nagios
# as a seed file after proper formatting.
#
# ----

if [[ -z ${1} ]]
then
cat <<EOD
$0 <subnet>
where
subnet, by example, is something like 192.168.1.0/24
EOD
exit
fi

nmap -sn ${1} | awk '
BEGIN{ printf("%-16s| %-18s| %-35s| %-30s\n","IP","MAC","NAME","VENDOR") }
/^Nmap scan report/ {
NAME=$5
x=$NF
gsub("[()]","",x)
IP=x
}

/^MAC Address/ {
MAC=$3
split($0,a,"(")
split(a[2],b,")")
VENDOR=b[1]
printf("%-16s| %-18s| %-35s| %-30s\n",IP,MAC,NAME,VENDOR)
}'

Here is an example of the output
[root@localhost ~]# ./discover 192.168.1.0/24
IP            | MAC               | NAME                               | VENDOR
192.168.1.1   | C8:D7:19:DE:54:2E | NyaRaePrimary                      | Cisco Consumer Products
192.168.1.100 | B8:27:EB:72:2A:4A | kodi1.grandenetworks.net           | Raspberry Pi Foundation
192.168.1.101 | 00:1F:3B:75:7F:EB | 192.168.1.101                      | Intel Corporate
192.168.1.102 | 6C:3B:E5:76:96:A5 | HP-Printer.grandenetworks.net      | Hewlett Packard
192.168.1.105 | A8:47:4A:AC:8F:89 | 192.168.1.105                      | Unknown
192.168.1.106 | F0:7D:68:0A:7C:8A | OllieMaeJoseph.grandenetworks.net  | D-Link
192.168.1.107 | 58:82:A8:81:C3:A6 | XboxOne                            | Unknown
192.168.1.109 | 28:56:5A:39:ED:FD | BRW28565A39EDFD.grandenetworks.net | Unknown
192.168.1.110 | 64:20:0C:90:24:D9 | Garlands-iPad.grandenetworks.net   | Apple
192.168.1.112 | 7C:D1:C3:17:0C:58 | Apple-TV.grandenetworks.net        | Apple
192.168.1.115 | B8:27:EB:12:DA:AC | kodi2.grandenetworks.net           | Raspberry Pi Foundation
192.168.1.118 | A4:77:33:8E:CE:C2 | Chromecast.grandenetworks.net      | Google
192.168.1.119 | 6C:AD:F8:5D:3A:D6 | 192.168.1.119                      | Azurewave Technologies
192.168.1.131 | F0:7D:68:0A:7A:D5 | EmmaEdwards.grandenetworks.net     | D-Link
192.168.1.135 | 28:10:7B:0C:3A:71 | EarlEdwards.grandenetworks.net     | D-Link International
192.168.1.136 | 00:09:B0:D6:A8:2A | 192.168.1.136                      | Onkyo
192.168.1.138 | 28:10:7B:0C:3A:74 | LeeJoseph.grandenetworks.net       | D-Link International
192.168.1.145 | D4:3D:7E:EF:93:99 | obama                              | Micro-Star Int'l Co
192.168.1.147 | E4:3E:D7:44:21:8F | LGwebOSTV.grandenetworks.net       | Unknown

Synchronized Windows to Linux Packet Capture

July 5th, 2017 | Garland Joseph

This script is part of a series of scripts that perform packet capture between two endpoints.  In this case, the endpoints are a Unix machine and a windows machine. This script was tested with the “source endpoint” as a Redhat Linux and the “target endpoint” a Windows 2016 Server machine.  

The circular traces are started on each machine and stopped whenever an event is detected on the Unix side.  In this case the event is to monitor a file (i.e., log) for a particular string.

Requirements: Wireshark installed on Windows. OpenSSH installed on Windows. 

Scenario

Unix to Windows Capture Scenario

Script

#Author: Garland R. Joseph, garland.joseph@gmail.com
# Date: May 2017
# u2wcap: See usage below. "Unix to Windows Capture"
#
# This script is offered as is. It is designed to
# run a circular trace using tcpdump on UNIX system
# and wireshark on Windows systems.
#
# You will either have to manually enter the password
# for the root account on the remote system or setup
# ssh keys from promptless access.
#
# The traces will stop once a key string SEARCH_STRING is
# found in LOG_FILE.
#
# Note: Some UNIX systems like LINUX Fedora will
# result in permsission denied when using
# tcpdump -W and -C options and writing to / or /root.
#
# Modify the REM_INTERFACE parameter below to fix the interface number
# on the windows system. Do a tshark -D to determine the interface number.
#
#
# -----

#
# Defaults
#

USAGE="u2wcap [-v] [ -c capture_file ] [ -w secs ] -h remote_host -l log_file -s search_string"
DEBUG=false
SLEEP_TIME="5" #seconds
LOCAL_CAPTURE_FILE="/tmp/capture"
TCPDUMPCMD="tcpdump -C 1 -W 2 -w ${LOCAL_CAPTURE_FILE}"

#
# Options for remote tracing
#

REM_CAP_FILE="capture.windows"
REM_USER="wireshark"
REM_INTERFACE="4"
FILESIZE=1000 #units or kB, so this means 1 Meg
#FILESIZE=500000 #512 Meg
#$FILESIZE=1000000 #units or kB, so this means 1 Gig
FILECOUNT="2" #creates a count of FILECOUNT of trace files at most of size FILESIZE
TSHARK_LOCATION="c:\progra~1\wireshark\tshark"
#TRACECMD="$TSHARK_LOCATION -b filesize:$FILESIZE -b files:$FILECOUNT -w ${REM_CAP_FILE}"
TRACECMD="$TSHARK_LOCATION -b filesize:$FILESIZE -b files:$FILECOUNT -w ${REM_CAP_FILE} -i ${REM_INTERFACE}"


#
# Process command line arguments
#

while getopts ":vc:w:l:s:h:" opts
do
case ${opts} in
v) DEBUG=true ;;
c) CAPTURE_FILE=${OPTARG} ;;
w) SLEEP_TIME=${OPTARG} ;;
s) SEARCH_STRING=${OPTARG} ;;
l) LOG_FILE=${OPTARG} ;;
h) REMOTE_HOST=${OPTARG} ;;
":") echo "Please specify a value for ${OPTARG}" ; exit ;;
\?) echo "${OPTARG} is not a valid switch" ; echo "${USAGE}" ; exit;;
esac
done

#
# Insure required values have been specified, check for existence of
# log file, getops should handle case of no values for -l and -s.
# A sanity check in the event getopts varies per unix
#

if [[ -z ${SEARCH_STRING} || -z ${LOG_FILE} || -z ${REMOTE_HOST} ]]
then
echo ${USAGE}
exit
fi
if ! [[ -f ${LOG_FILE} ]]
then
echo "File ${LOG_FILE} does not exist"
exit
fi

#
# Start trace on remote host
#
$(ssh ${REM_USER}@${REMOTE_HOST} ${TRACECMD})& 2>&1 > /dev/null

#
# Start trace on this host
#

${TCPDUMPCMD} 2>/dev/null 1>/dev/null & LOCAL_PID=$!
${DEBUG} && echo "${0}-I-LOCAL_PID, local pid is ${LOCAL_PID}."

#
# Monitor log file
#

old_count=`grep -c ${SEARCH_STRING} ${LOG_FILE}`
(( new_count=old_count ))
(( i = 0 ))
while (( old_count == new_count ))
do
(( i++ ))
${DEBUG} && echo "${0}-F-SLEEP, sleeping ${SLEEP_TIME}, iternation ${i}."
sleep ${SLEEP_TIME}
new_count=`grep -c ${SEARCH_STRING} ${LOG_FILE}`
done

#
# At this point, search string has been found, stop traces
#

kill ${LOCAL_PID}
ssh ${REM_USER}@${REMOTE_HOST} taskkill /f /fi \"imagename eq tshark*\"

#
# Reminders
#

echo "Consult files ${REM_CAP_FILE} on remote host ${REMOTE_HOST} and ${LOC_CAP_FILE} on local host."

exit