MarketGenie

01Overview

Build a sound system first. The returns target is derived from measured paper-trading expectancy, never assumed. Success is defined by gates, not income.

MarketGenie runs a full Indian-market day unattended (except one daily token-approval tap), trading Nifty / Bank Nifty / Sensex options on the Upstox API. It pairs an LLM-driven news edge with a deterministic risk engine and an execution router that swaps between a paper fill-simulator, the Upstox sandbox, and live order placement by config alone.

Four success gates (design objective)

1. System runs a full market day unattended (except the daily token tap).
2. Paper trading shows positive expectancy after costs over ≥80 trades / ≥6 weeks.
3. Risk engine demonstrably halts trading — in drill and in anger.
4. Only then: live capital, starting at 20%.

02Architecture

Three temporal phases, one execution router with four backends. Data flows top-to-bottom; the router is the single seam between strategy and broker. Gate ladder: shadow (record only, no orders) → paper (fill simulator) → sandbox (Upstox sandbox API, no fills) → live (real orders; requires NT_ALLOW_LIVE=1 env — never settable from YAML, so the self-reflection loop cannot enable live trading).

Cross-cutting design rules

The shared execution seam

trade_signal() is the single entry-to-flat path for any signal: kill-check → risk plan → entry → monitor → exit. Two callers reuse it, so the gap strategy and announcement strategy share one tested core.

run_day()          ──┐  derives S2 gap signal, then delegates
IntradaySession()  ──┼─►  trade_signal(cfg, sig, spot=…, sources=…, …)
                      │       kill-check → RiskEngine.plan → execute → PositionMonitor → exit
replay (SimClock)  ──┘  same loop, archived store instead of live feed

03Data flow

Two flows matter most: how a single announcement becomes (or doesn't become) a trade, and how the daemon orchestrates a whole day.

Signal pipeline — announcement → trade

Daemon day-cycle

non-trading day  → sleep to next 08:30
trading day:
  08:45  start capture  ─┐
  09:10  start trade ────┼─ run concurrently (asyncio.gather), each fault-isolated
         (gap + intraday)│   capture fills the store the IntradaySession polls
  14:30  last new intraday entry (ops.intraday_until)
  15:12  hard square-off
  15:35  capture ends
  15:45  EOD → render report + refine proposal → Telegram
  every loop: touch var/heartbeat   (healthcheck input; stale >5min = unhealthy)

The same IntradaySession loop serves live and replay. In replay, RecordedSources + SimClock serve an archived day deterministically — clock-bounded so no future data leaks into a decision. This symmetry is the M6 integration-validation guarantee.

04Decision flow

How a signal becomes a trade — or gets blocked. Two views: follow one signal through every gate, or watch a whole trading day on the clock. Pick a scenario to see the accept and block branches the code actually takes.

05Strategy portfolio

S3 and S4 are enabled by default in config/default.yaml (paper mode — no live orders); toggle s3.enabled / s4.enabled. See Execution modes for how mode gates order placement, and Configure for every knob.

06Risk engine & kill switches

Fractional Kelly sizer — kelly_fraction (0.25) × edge, capped at risk_per_trade_pct (1.5% of capital). Live win-rate/payoff from a rolling 30-day journal window; priors until 30 trades exist. Lot size and margin are runtime lookups.

The monitor polls the halt sentinel on a 1s tick, so /halt flattens paper positions in <5s (drill-measured 0.51s). /resume clears the file, but L2/L3 latches stay latched for the session.

07Execution modes

One pipeline, four interchangeable execution backends. trading.mode (or run --mode) picks the backend, and orchestrator.run_cli resolves it through make_router() before any signal is evaluated — so an unknown or unsafe mode is rejected at the entry gate, never mid-trade.

make_router(mode, …) is the single factory; an unrecognised mode resolves to UNKNOWN_MODE and degrades safely (no orders placed). Live fill reconciliation and position monitoring against real fills is M7 — not yet wired; see Roadmap.

08Install

Requirements: Python ≥3.12, git. Docker only for containerized deploy.

git clone <repo-url> marketgenie && cd neurotrade
python3.12 -m venv .venv
.venv/bin/pip install -e ".[dev,fyers]"

.venv/bin/pip install 'aiohttp>=3.10'

Verify:

.venv/bin/python -m pytest tests/ -q      # 541 passed
.venv/bin/ruff check src tests            # clean
.venv/bin/python -m marketgenie token-status

09Deploy

cp .env.example .env       # fill secrets
docker compose up -d --build

helm install nt deploy/helm/neurotrade \
  --set secret.data.UPSTOX_API_KEY=... \
  --set secret.data.TELEGRAM_BOT_TOKEN=...

cd deploy/ansible
cp inventory.example.ini inventory.ini   # fill VPS host
cp vars.example.yml vars.yml             # fill secrets + repo url
ansible-playbook -i inventory.ini provision.yml -e @vars.yml   # provision
ansible-playbook -i inventory.ini deploy.yml  -e @vars.yml     # update

scripts/token_ritual.sh user@vps-host

10Use — CLI

python -m marketgenie <command>

Telegram bot (configured chat only): /halt /resume /status /pnl /signals [ticker] (shows signals or analyzes ticker), /weights.

11Monitoring

docker compose up starts Prometheus (:9090) and Grafana (:3000, anonymous viewer) with two auto-provisioned dashboards. No extra config needed.

Metric scrape endpoints

/metrics is exposed in-process via prometheus_client. Both processes are scraped:

Three metric families

Two Grafana dashboards

Grafana annotations from the journal

Each journal write triggers a Grafana annotation via the Annotations API:

blocked_entries — "could-have-traded" concept

The journal column blocked_entries counts signals that passed the ≥65% combiner gate but were blocked downstream by the risk engine or a kill switch. This is the "could-have-traded" signal: a non-zero value doesn't mean bad performance — it means the guard rails were engaged. Tracking it lets you distinguish a quiet market (no signals) from a gated market (signals, but blocked).

Gate reason codes recorded in nt_blocked_total{reason}:

Observability data-flow

12Admin console & decision audit

A read-mostly operator console (app/admin_api.py, Material 3) over the journal, plus a fault-isolated decision-audit trail. Every signal — traded or blocked — is written to the decisions table with its full rationale, so nothing the system decides is invisible.

Console (M8c)

Decision audit (M8b)

journal/audit.py safe_log_decision() never raises — an audit-write failure cannot break the trade path. A passing signal records entered; an entry that unwinds mid-flight records abort; a rejected one records why:

size_delta_gate covers the size / delta-cap / max-positions rejections. Mode & token gating from the execution router surfaces as an upper-case block reason — LIVE_NOT_ENABLED, UNKNOWN_MODE, TOKEN_ABSENT, SANDBOX_TOKEN_ABSENT — on the same nt_blocked_total{reason} metric the dashboards aggregate.

13Self-reflection loop

Opt-in (reflect.enabled). The system proposes its own improvements — but a human and CI are always between a proposal and production.

At EOD the reflect agent (app/reflect.py) reviews the decision audit and the day's trades, then emits a validated change-set — small, evidence-backed edits to config knobs, code, or docs. Deterministic code (app/reflect_apply.py) applies it to a reflect/<date> branch and opens a Pull Request.

Market-grounded feedback

app/trade_review.py scores closed trades against what the market actually did afterwards and feeds that review back into the reflection prompt — so proposals are grounded in realised outcomes, not the model's priors alone.

Guard rails

• Never auto-merges — requires green CI and your approval.
• Protected paths (.github/, tests/, risk & execution code) get a ⚠ HIGH-RISK banner on the PR.
• Cannot enable live trading — NT_ALLOW_LIVE is env-only (see Execution modes), outside the agent's reach.
• Requires the REFLECT_PAT secret. Dry run: docker compose run --rm daemon reflect --dry-run.

14Guides

Ten task-focused walkthroughs — four lifecycle (setup → live), six per-strategy playbooks. Each follows the same shape: what the app does → the knobs that steer it → run / observe → verify. The system trades autonomously; these show how to tune and watch it, not how to hand-place trades.

cp .env.example .env            # paste the four secrets above
docker compose up -d --build    # daemon + bot + Prometheus + Grafana

# one-time + each trading morning: approve the Upstox token
docker compose run --rm --service-ports daemon login
#   (on a remote VPS, use scripts/token_ritual.sh user@vps-host instead — it
#    SSH-forwards localhost:8721 so the OAuth browser stays on your laptop)
docker compose run --rm daemon token-status     # expect: usable
docker compose run --rm daemon instruments        # cache the instrument master

# the daemon already runs the day cycle; to trigger a single day manually:
docker compose run --rm daemon run --mode paper

# after a few trading days, score the campaign:
docker compose run --rm daemon campaign --since 2026-06-14

# 1. validate V3 order payload shapes against the sandbox first
docker compose run --rm daemon sandbox-check

# 2. arm + prove the kill switches in paper before risking capital
docker compose run --rm daemon drill --scenario l2
docker compose run --rm daemon drill --scenario l3

# 3. set trading.mode: live in config/default.yaml, redeploy, then:
docker compose run --rm daemon run --mode live

# capture a live session's news/FII/VIX/chain to the archive
docker compose run --rm daemon capture --until 15:35

# replay a captured day through the M3 signal layer (offline, no orders)
docker compose run --rm daemon replay --date 2026-06-14 --mock --limit 25

# or replay a full day through every layer end-to-end
docker compose run --rm daemon replay-day --date 2026-06-14 --mock

# every morning — the Upstox token expires 03:30 IST with no headless renewal.
# this forwards localhost:8721 over SSH so the OAuth browser stays on your laptop:
scripts/token_ritual.sh user@vps-host

# read the day's results (also auto-sent to Telegram at 15:45):
docker compose run --rm daemon report --date 2026-06-14

# tune: the refine agent proposes source-weight changes post-market (never auto-applied).
# review the proposal, then edit the individual weights under signals: in
# config/default.yaml — weight_news / weight_technical / weight_fii — and redeploy.

# dry-run a paper day so you can see the gap decision
docker compose run --rm daemon run --mode paper

# read the day's outcome (also sent to Telegram ~15:45 IST)
docker compose run --rm daemon report --date 2026-06-14
# or ask the bot intraday:  /signals   /pnl   /status

# replay a captured announcement day through the signal layer (offline, no orders)
docker compose run --rm daemon replay --date 2026-06-14 --mock --limit 25

# the refine agent proposes weight changes post-market (never auto-applied):
docker compose run --rm daemon report --date 2026-06-14   #   /weights on the bot

# prove the hedge/spread + kill behaviour in a paper drill (no capital)
docker compose run --rm daemon drill --scenario l2

# watch a real day choose naked vs spread by premium
docker compose run --rm daemon run --mode paper
docker compose run --rm daemon report --date 2026-06-14

# prove the L2 / L3 kill switches latch in paper before trusting the limits
docker compose run --rm daemon drill --scenario l2
docker compose run --rm daemon drill --scenario l3
#   bot: /status shows kill level + halt;  admin console shows it live

# on a NIFTY expiry day, run a paper day and watch for the S3 entry
docker compose run --rm daemon run --mode paper
docker compose run --rm daemon report --date 2026-06-16

docker compose run --rm daemon run --mode paper
docker compose run --rm daemon report --date 2026-06-16
#   bot: /signals shows the live VIX-spike trigger state

15Configure

Configuration is config/default.yaml → pydantic Settings. Secrets always come from environment variables, never the file.

Secrets (env / .env)

UPSTOX_API_KEY=...        UPSTOX_API_SECRET=...
FYERS_CLIENT_ID=...       FYERS_SECRET_KEY=...    FYERS_PIN=...
ANTHROPIC_API_KEY=...     TAVILY_API_KEY=...      # Tavily = web-research (opt-in)
TELEGRAM_BOT_TOKEN=...    TELEGRAM_CHAT_ID=...    # bot serves ONLY this chat
UPSTOX_SANDBOX_TOKEN=...  # separate 30-day portal token
NT_ADMIN_TOKEN=...        REFLECT_PAT=...         # admin console · self-reflection PRs
NT_BASE_DOMAIN=...        CLOUDFLARE_API_TOKEN=...  # Caddy reverse proxy (subdomain TLS) — opt-in
# NT_ALLOW_LIVE=1   → env-only live-trading gate (NEVER in YAML — see §07)

Where to get each value — provider setup

Every value below is obtained manually from the provider, then pasted into .env. The When needed column tells you which are mandatory vs opt-in.

Sections & defaults

LLM provider routing

llm:
  providers:
    news_fast:    {provider: anthropic, model: claude-haiku-4-5}
    news_deep:    {provider: anthropic, model: claude-sonnet-4-6}
    debate_bull:  {provider: anthropic, model: claude-haiku-4-5}
    debate_bear:  {provider: anthropic, model: claude-haiku-4-5}
    debate_judge: {provider: anthropic, model: claude-sonnet-4-6}
    refine:       {provider: anthropic, model: claude-haiku-4-5}
    reflect:      {provider: anthropic, model: claude-sonnet-4-6}
    # research:   {provider: anthropic, model: claude-sonnet-4-6}   # add when enabling research
  daily_budget_inr: 100

Swap any role to OpenAI / Gemini / Ollama by changing provider + model — no code change. Override any of this from the environment without editing the repo — see Intel → LLM env routing.

S3 / S4 strategy knobs

s3:                    # 0DTE expiry-day momentum (NIFTY weekly expiry only)
  enabled: true
  window_start: "09:45"
  window_end:   "13:00"
  target_pct: 0.40
  stop_pct:   0.30
  exit_by:    "15:00"  # hard square-off before ~15:30 settlement
s4:                    # VIX-spike momentum → BUY_PE
  enabled: true
  spike_pct: 10.0      # % jump in India VIX vs prior close that arms an entry
  max_entries: 2       # daily cap (latches per spike, re-arms when VIX settles)

S4's target_pct/stop_pct/exit_by are unset by default and fall back to the global trading exits; each s3.* exit key likewise falls back when unset.

16Intel — news, research & LLM routing

Three opt-in-OFF units that widen and harden the information edge. All default disabled; enable per unit in config/env.

LLM env overrides & credential pool

The CredentialPool rotates comma-separated keys round-robin and fails over to the next provider when a provider's keys are exhausted; on full exhaustion the pipeline degrades to raw confidence rather than failing.

# Global kill-switch (all roles → one model):
MARKETGENIE_LLM_PROVIDER=anthropic
MARKETGENIE_LLM_MODEL=claude-haiku-4-5

# Per-role escape (role uppercased, hyphens → underscores):
MARKETGENIE_LLM_NEWS_DEEP_PROVIDER=anthropic
MARKETGENIE_LLM_NEWS_DEEP_MODEL=claude-opus-4-8

# Credential pool — comma-separated keys; round-robin, fail over on exhaustion:
MARKETGENIE_LLM_KEYS_ANTHROPIC=key1,key2
MARKETGENIE_LLM_KEYS_OPENAI=key1
# Cross-provider fallback order when all keys for a provider are exhausted:
MARKETGENIE_LLM_FALLBACK=anthropic,openai

Spend stays bounded by llm.daily_budget_inr. The YAML route blocks live in Configure.

17Storage

Two stores — the journal (signals, trades, positions, debates, decisions) and marketdata (ticks, candles, snapshots, events). Both run on SQLAlchemy Core; the dialect is chosen by URL, so the same code serves SQLite and Postgres.

# point at Postgres, then create the schema:
export MARKETGENIE_JOURNAL_DB_URL=postgresql+psycopg://user:pass@host/db
python -m marketgenie db upgrade        # Alembic migrations on a fresh database

storage/factory.py picks the store from config; storage/engine.py builds a dialect-correct engine (SQLite WAL vs a Postgres pool); storage/schema.py defines the Core tables. SQLite needs zero setup and is the default for paper campaigns; Postgres is for durable, multi-process or cloud deploys.

18Module explorer

A curated map of the system's Python modules across 12 architectural layers — app, data, signals, risk, execution, llm, broker, market, journal, storage, research and obs. Filter by layer or search above.

19Tune

All knobs live in config/default.yaml — edit + restart, no redeploy logic.

Weight tuning is evidence-driven: the refine agent scores each source's hit-rate from the journal and proposes new weights after each session. Proposals are never auto-applied — review via /weights or the EOD report, then edit config manually.

20Testing & validation

.venv/bin/python -m pytest tests/ -q              # 541 passed, 2 skipped, 6 deselected (live)
.venv/bin/python -m pytest tests/integration/ -q  # cross-component + full-day E2E
.venv/bin/python -m pytest -m live                # network tests (needs token)
.venv/bin/ruff check src tests

• TDD throughout — every feature shipped failing-test-first.
• tests/integration/ is the M6 entry-condition gate: a synthetic full session through pollers → agents → debate → combiner → risk → paper fills → journal, output asserted.
• replay-day --mock runs the same path on canned LLM output, gating on journal consistency (all positions closed, no orphaned legs, failures surfaced).
• Drills exercise the kill switches in isolation, no network.
• Fault-isolation test — test_server_bind_failure_does_not_raise: verifies that a second metrics server on an occupied port logs and never raises, keeping the trade path safe.

21Roadmap & milestones

M6 → M7 graduation gates (measured by campaign)

• Positive expectancy after costs over ≥80 trades / ≥6 weeks
• Max daily drawdown ≤2.5% · zero L3 events from system defects
• News→order p95 <20s · paper fills spot-audited against market prints
• AlgoTest backtests don't contradict paper

22Verified market facts

Every load-bearing market/API claim was re-verified 2026-06-10 against primary sources (SEBI/NSE circulars, live API probes, instrument masters).