Frequently Asked Questions About Egor Howell Production-Grade ML Deploy Framework
22 answers covering everything from basics to advanced usage.
// Basics
What exactly makes a project 'production-grade' versus a hobby project?
Production-grade code has single-responsibility files, type annotations, docstrings, unit tests outnumbering source lines, a Makefile, dependency management via Poetry, CI/CD gates, and a live deployment. A hobby project is usually a single notebook with a trained model. The framework's three bars are sufficient quality, scalability in principle, and differentiation — most candidates lack all three.
What is the Personal Interest → 25 Questions Formula?
It's Egor's ideation method: list five personal interests outside work, write five questions for each (25 total), then identify which questions ML or data can answer. You pick the one that is most personally motivating and least likely to appear in other candidates' portfolios. This produces differentiated, sustainable ideas rather than generic tutorials everyone builds.
Why does the framework use Supabase instead of a custom backend?
Supabase is a backend-as-a-service providing a real-time Postgres database and auth without you building or managing a server. Your pipeline writes predictions to a table whose columns match your dictionary keys, and the Streamlit front-end API-calls that table directly. This removes an entire layer of backend engineering while still looking professional.
// How To
How do I set up Python versioning and dependencies for this project?
Install Homebrew, then Pyenv, set your local Python version (e.g. 3.12), install Poetry for that version, and run `poetry install` to build the virtual environment from pyproject.toml. Never mix Python versions across repos. Document dependencies in pyproject.toml; poetry.lock handles transitive dependencies automatically.
How should I split my ML code across files?
Group code by action, one primary callable per file: main.py (orchestration only), extractor.py (data ingestion), processor.py (preprocessing and feature engineering), model.py (training and inference), optimizer.py (downstream decisions), database.py (Supabase), streamlit_app.py (front-end), and settings.py (all constants). main.py calls these in sequence and contains no business logic itself.
How do I automate daily model retraining?
Create .github/workflows/daily_run.yml with a cron schedule (e.g. `0 9 * * *` for 9 AM UTC) and add workflow_dispatch for manual triggering. The steps checkout code, set up Python 3.12, install Poetry and dependencies, then run `poetry run python src/main.py`. Inject Supabase credentials from GitHub repository secrets as environment variables — never from local secrets.
How do I set up automated deployment on merge to main?
Create .github/workflows/deploy.yml triggered on push to main. The job SSHes into your VPS using a private key stored as a GitHub secret (paired with a public key on the VPS) and runs scripts/deploy.sh. That script navigates to the repo, runs git pull, checks for Poetry dependency changes, and restarts the Streamlit systemd service.
How do I configure the VPS to host the Streamlit app?
Purchase a VPS (reference: Hostinger KVM2 — 8GB RAM, 2 cores, Ubuntu), SSH in, and install Python, Pyenv, Poetry, and Nginx. Clone the repo, add Supabase credentials to a .env file, configure a systemd service with `Restart=always`, set up Nginx as a reverse proxy, and install Certbot for HTTPS. The service polls Supabase every ~300 seconds.
// Troubleshooting
My CircleCI checks keep failing on push — what should I check?
Run `make check` locally first — it runs format, lint (Ruff + Black), type-check (mypy), and pytest in sequence, mirroring CI. Common causes: missing type annotations failing mypy, unformatted code, or tests asserting only that functions run. Ensure your config.yml jobs run on Python 3.12 and install the project via Poetry before running each make command.
My predictions aren't appearing in Supabase — how do I debug this?
First run main.py locally and confirm predictions write with the correct timestamp. Check that your table column names exactly match your dictionary keys (e.g. as_of_date, ticker, predicted_price, weight). Verify the Supabase URL and service-role key are set as environment variables locally and mirrored as GitHub repository secrets. Never hardcode credentials — missing env vars are the most common cause.
The deploy workflow runs but the live site doesn't update — what's wrong?
Check that deploy.sh actually restarts the systemd service via `systemctl restart <service_name>` after git pull. Confirm the GitHub secret private key is paired with the public key on the VPS and SSH succeeds. Verify the systemd service has `Restart=always` and that Nginx is proxying correctly. Merge a dummy change and watch the workflow logs to isolate the failing step.
Why is my repo being treated like a notebook project by reviewers?
You're probably missing the high-signal production markers: a Makefile, single-responsibility files, tests outnumbering source lines, CI/CD config, and a protected main branch. Skipping the Makefile is a named pitfall — it's a low-effort, high-signal addition. Add `make check`, structure code by action, and enforce PR-based merges to shift perception immediately.
// Comparisons
How does this framework compare to using AWS SageMaker or a full cloud stack?
AWS and complex cloud infrastructure are usually overkill for a solo portfolio project — they slow you down and don't add proportionate signal. The framework deliberately uses Poetry, CircleCI, GitHub Actions, Supabase, Streamlit, and a VPS because these are what professionals actually use, ship faster, and look just as impressive when combined correctly. Complexity should serve the project, not the résumé.
How does this compare to following a generic ML tutorial project?
Generic tutorials produce projects every candidate has, stopping at a trained model. This framework enforces differentiation through the 25 Questions Formula, adds a downstream action layer, and ships a fully automated deployed system. The result signals engineering maturity — CI/CD, testing discipline, live deployment — which is far more compelling in interviews than another Titanic or house-price notebook.
Should I use Prophet or a deep learning model for forecasting?
Prefer a plug-and-play model like Prophet unless your problem genuinely demands more. Choosing an algorithm purely for sophistication is a named pitfall. The framework's whole point is that engineering and deployment are the harder, more differentiating skills — a simple, reliable model lets you invest your effort where it actually distinguishes you.
How does Streamlit compare to a JavaScript front-end for this project?
Streamlit is Python-native, fast to build, and appropriate for a portfolio dashboard — ship it first. If production complexity later demands richer interactivity, replace it with a JavaScript framework. This reflects the 'do it first, then do it right, then do it better' principle: get a working front-end live before investing in front-end sophistication.
// Advanced
How many tests do I actually need per function?
At least two or three per source function, covering the happy path, at least one edge case, and one failure mode. Assert on output columns, types, shapes, and values — not just that the function runs. The target is more test lines than source lines. Fewer test lines than source lines is a direct red flag to reviewers.
How do I protect the main branch and why does it matter?
In GitHub, enable branch protection on main: require a pull request and require all CircleCI checks (lint, type-check, test) to pass before merging. This prevents force-pushing and accidentally breaking production. Combined with the deploy-on-merge workflow, it means only vetted code reaches your live VPS — a strong signal of professional discipline.
What is the two-domain architecture and how do I map it to my project?
Pair a forecasting/ML component with an optimisation/action component so predictions drive a decision. Ask: what does my model predict, and what allocation, recommendation, or alert does that prediction drive? Example mappings: Prophet forecasts → Markowitz portfolio optimisation; injury probability → squad selection linear programme; SKU demand → inventory restocking optimisation. A predict-only system is a named weaker outcome.
How should I handle secrets across local, CI, and the VPS?
Never hardcode credentials anywhere. Store secrets in environment variables locally (.zshrc or .env), mirror them as GitHub repository secrets for CI/CD, and add them to a .env file on the VPS (not the system environment). The daily_run and deploy workflows inject them from GitHub secrets. Hardcoding keys is a named pitfall and a security red flag.
What does settings.py do and why keep it separate?
settings.py holds all project constants — tickers, date ranges, model hyperparameters, risk or cost parameters, SKU lists, lead times. Keeping configuration separate means you can tune behaviour without touching business logic, which improves readability, testability, and scalability. It's a small architectural choice that signals production thinking and makes your modules single-responsibility and reusable.
How do I validate the full pipeline before showing a recruiter?
Run main.py locally and confirm predictions appear in Supabase with the correct timestamp. Open the Streamlit app locally and confirm data renders. Merge a dummy change to main and confirm the deploy workflow fires and the live VPS updates. Run `make check` and confirm all linting, type-checking, and tests pass. Only then is the project recruiter-ready.