2026-04-20· 11 min read

From Nine DRY RUN Failures to Go-Live: Joseph's T+2 Triple Lock, 458 Tests, 18 Crons

Joseph, an automated investment execution agent, entered live observation on 2026-04-20. This is the full technical write-up — nine failure modes, the T+2 triple lock, Shioaji pitfalls, and every cron in the schedule.

#joseph#claude-code#dry-run#triple-lock#shioaji#automation#trading-system#series-3

TL;DR if you're in a hurry

Joseph is an automated investment execution subsystem. On Monday 2026-04-20 at 08:00 Taipei time it passed three gates and entered live observation mode. Stats: 25,941 lines of Python, 40 test files / 458 tests, 18 cron schedules, 106 commits (first: 2026-01-26 v3.3). Nine DRY RUN failure modes were found and fixed before going live. This post covers every technical detail I can share publicly — but not the triple-lock implementation, credentials, holdings, or live order history.

Series overview

[Part 1] Identity: three projects killed before Joseph survived
[Part 2] Methodology: seven principles for independent AI development
[Part 3 — this post] Build log: DRY RUN failures, T+2 triple lock, Shioaji, crons

1. System snapshot at go-live

Metric	Value	Source
Python source	25,941 lines (excludes venv / cache / archive)	`find . -name "*.py"`
Test files	40	`tests/test_*.py`
Tests	458 (go-live declaration, pytest collected)	`docs/GO_DECISION_20260420.md`
Docs	74 markdown files	`docs/*.md`
Cron schedules	18	`crontab -l`
Git commits	106	`git rev-list --count HEAD`
First commit	2026-01-26, v3.3 Ultimate	`git log`
Latest commit	2026-04-18, GO decision	`git log`
Core modules	`core/` 123 files; `layers/` four-tier architecture	`find core / layers`

2. Nine DRY RUN failure modes

Joseph ran three DRY RUN batches before go-live: 4/7–4/10 initial, 4/13 formal, 4/15–4/17 extended. Each batch simulated real orders without placing them. Every batch found something. Chronologically:

1. Zero cash (2026-04-10)

Symptom: ShadowExecutor initialized and returned cash=$0, pool weights all zero, no orders possible
Root cause: Missing async get_balance() / get_positions() with fallbacks
Fix: Added async queries + fallback
Takeaway: Async interface None needs explicit semantics — "not back yet" vs "really nothing"

2. Scanner time-window boundary (2026-04-10)

Symptom: 08:30 judged "closed," scanner didn't run
Root cause: Boundary timestamp off
Fix: Changed to 08:30 ≤ t < 13:30 open

3. Allocations all zero when equity=0 (2026-04-10)

Symptom: When equity computed to 0, allocation module produced all-zero pool weights
Root cause: Strategy Engine missing equity fallback
Fix: Added equity = cash + marked_value fallback

4. Shadow_ledger polluted by 33 leftover E2E trades (2026-04-13)

Symptom: holding_days showed 38, system had run 1 day
Root cause: tests/test_e2e_dryrun_0410.py didn't clean state; wrote fake trades dated 2026-03-06 into shadow_ledger.json
Fix: FIX_REPORT_0413.md reset shadow_ledger (cash=10000, positions=[], history=[])

5. Gmail HTML attachment UTF-8 broken (2026-04-13)

Symptom: iOS Safari opened Gmail HTML attachments with mojibake Chinese
Root cause: Missing <meta charset="UTF-8"> + wrong MIMEBase encoding
Fix: core/report_formatter.py added charset meta; core/reporter.py switched to MIMEText(utf-8)

6. Wrong holding_days (2026-04-13)

Symptom: holding_days=38
Root cause: Side effect of #4 (first_buy_time=2026-03-06 was leftover test data)
Fix: Resolved automatically when #4 was fixed

7. cron_wrapper.sh wrong path (2026-04-13)

Symptom: All crons failed, logs said "No such file or directory"
Root cause: scripts/cron_wrapper.sh hardcoded old account path /home/uop3364, machine had migrated to /home/molinkailazy
Fix: Changed absolute path to new account
Takeaway: Hardcoded paths are debt. Migrating once teaches you.

8. cron_scan.py duplicate import (2026-04-13)

Symptom: from pathlib import Path imported twice in one function
Root cause: Editor oversight
Fix: Removed duplicate at line 27
Why it matters: Pure mechanical oversight gets caught by DRY RUN too. That's the value.

9. Telegram fired alerts outside trading hours (2026-04-18 00:01)

Symptom: 00:01 in the morning, bot sent 6 Circuit Breaker alerts + 1 error notification
Root cause: Doc claimed PYTEST_CURRENT_TEST guard existed; code had no such guard. Manual pytest run hit the real bot.
Fix: Implemented PYTEST_CURRENT_TEST check + conftest.py autouse fixture mock + cleaned up related tests. Five phases.
The lesson: Don't trust the doc that says "implemented." Trust whether the test passes.

3. T+2 triple lock (conceptual level)

T+2 refers to Taiwan stock settlement — trade on T, settle on T+2. Cash from a sell doesn't arrive for two business days. Systems must distinguish "book cash" from "available cash" — or they'll reuse uncleared funds to re-enter.

What the triple lock locks isn't T+2 settlement itself. It locks the act of placing a real order. All three independent switches must align before the system will execute:

Layer	Location	Meaning	DRY RUN	Go-Live
1	`config/config.json` strategy flag	Whether system is in live mode	false	true
2	`config/config.json` execution guard flag	Additional "forbid live trading" flag, active even if layer 1 is true	true	false
3	`.env` environment variable	Human operator per-session authorization (not in source control)	unset	1

Why three?

Layer 1 alone could be tripped by a code change
Layers 1 + 2 both live in source — still in the code blast radius
Layer 3 is in env vars, not in git — it represents a human explicitly authorizing this session

Verification point: core/trader_live.py._verify_triple_lock(). Any missing layer → LiveTradingForbiddenError with a message that doesn't leak credentials.

Intentionally not disclosed: the exact key names, value formats, validation order, and error message text. These aren't security primitives (defense-in-depth, not the primary line), but keeping them private reduces mistakes — no one accidentally mixes up the wrong config value.

4. Four Shioaji pitfalls

Shioaji is the quant-trading API from SinoPac. Joseph's last two weeks before go-live hit these four:

1. T+2 logic entirely missing (2026-04-16)

Found: api.account_balance() / list_positions() / settlements() not wired; old capital_manager.py had been archived; the new skeleton didn't re-add an equivalent module
Fix: Task #6-PreLive (2026-04-18) added pending_settlements.py + a new capital_manager.py + TraderLive bridge

2. Pre-trade gate reading shadow_ledger instead of real balance (2026-04-17)

Found: shadow_executor inline gate used shadow_ledger.current_cash, not Shioaji live balance. Fine during DRY RUN. Live would violate Rule 1.3 ("use live account balance as source of truth").
Fix: Task #6 added capital_manager._can_afford() + Shioaji bridge layer

3. SELL proceeds booked instantly (shadow vs T+2 mismatch)

Found: Shadow ledger semantics were "instant settlement" — sell credits cash immediately. Real T+2 waits two days; on D+1, unsettled cash could be reused to buy.
Fix: Task #6 added pending_settlements tracking actual settlement dates; available_cash excludes unsettled amounts

4. Silent login failure risk

Found: Shioaji login failure returned {"success": False, "error": ...} instead of raising. Callers might silently continue.
Fix: Task #6 introduced ShioajiAPIFailedError / ShioajiOrderRejected exception types; Task #7 added _classify_order_status() for second-pass status interpretation + _fail_dict fallback

These four became the foundation for Gate 11 (TraderLive real-order capability) — 72 unit tests covering FILLED / CANCELLED / REJECTED / FAILED terminal states + polling + fallback. All green = go.

5. Eighteen cron schedules (weekday / weekly rhythm)

Joseph runs on a GCP VM. All times UTC+8 (Taipei).

Time	Task	Schedule
07:50	Boot notification (system wakeup)	Weekday
08:00	Pre-market data	Weekday
08:05	TraderLive self-test	Weekday
08:55	Pre-market bridge health check	Weekday
09:00	Market-open scan	Weekday
09:00–12:00	Intraday market data (hourly)	Weekday
10:00–13:00	Intraday scan (hourly)	Weekday
13:30	Market close processing	Weekday
13:30	Daily performance review	Weekday
13:35	Close-of-day market data	Weekday
13:40	Investment ratio check	Weekday
13:45	Monthly report	Weekday
14:00	Weekly report	Friday
14:00	Backup	Weekday
14:30	Git push (version-control daily logs)	Weekday
01:00	refresh-history	Sunday

Total: 18 schedule entries (hourly jobs expand further).

Why is there a TraderLive self-test at 08:05? Because if TraderLive's live capability breaks 55 minutes before open, the 09:00 scan can compute orders but can't send them. The self-test catches this pre-open and fires a Telegram alert.

Why another health check at 08:55? Redundancy. 08:05 is early detection; 08:55 is last-mile (5 min before open). If 08:55 fails, the order gate closes for the day — observe-only mode.

6. Three gates passed in the 2026-04-18 dry run

Gate 4: Tech health

✅ 5/5 PASS
Checked: system dependencies, cron.log noise floor, cache validator, cron schedule integrity, log rotation

Gate 10: T+2 guards

✅ 3 PASS + 1 expected WARN
10.1 stress test 6/6 PASS
10.2 / 10.3 pending_settlements.json schema valid
10.4 settle_heartbeat produced after 08:05 pre-market (expected WARN at 4/18 test time — before that window)

Gate 11: TraderLive live-order capability

✅ 72 tests PASS
_call_shioaji_place_order covers all terminal states + polling + fallback
Shioaji login confirmed

Three gates green → Monday 2026-04-20 at 08:00 startup.

7. Risk, disclosure, what isn't public

This post does not disclose

Specific holdings, entry prices, position sizes, or order decisions
Exact key names, value formats, or validation order of the triple lock
Shioaji API keys, account numbers, credential paths, or .env contents
shadow_ledger actual trade history

Positioning and risk

Joseph is a personal investment record and execution subsystem. Not financial advice, not a signal service, not a guide to copying trades.
This three-part series records "how a non-engineer used AI to build a maintainable real system." It's not promotion of any investment method.
Going live doesn't imply positive performance. Right now Joseph has passed engineering validation only. Performance validation will require months of live observation.
I take no responsibility for investment actions third parties take on the basis of this writing.

FAQ

Q1: Is 458 the pytest collected count or the test function count? GO_DECISION_20260420.md says 458 — the actual pytest-collected count at go-live. A grep -c "^def test_" across the test dir counts 416 test functions. The difference is parameterized cases expanding. Both numbers are real, different contexts.

Q2: What are the three layer key names of the T+2 triple lock? Not public. Publishing them would raise the risk of someone copy-pasting wrong values and isn't useful to anyone else.

Q3: How many days of DRY RUN total? Three batches: 4/7–4/10 (initial; caught failure modes 1–3), 4/13 (formal; caught 4–8), 4/15–4/17 (extended; stability validation; mode 9 hit 4/18 as the Telegram leak).

Q4: Are the Shioaji issues Shioaji's fault or Joseph's? All Joseph's. Shioaji's API behavior is entirely reasonable. Joseph's early skeleton hadn't wired the right modules, didn't match T+2 semantics. Blaming the third-party API is the cheapest excuse.

Q5: 18 crons, don't they collide? They can. That's why cron_wrapper.sh and lock files exist. If a task is still running, the next invocation skips and logs it — no stacking.

Q6: Will there be failure modes 10, 11, 12 after go-live? Almost certainly. Live environments contain scenarios DRY RUN never touches (partial fills, halts, disconnects). I'll write post-live notes as each new mode shows up.

Q7: Why does Shioaji ordering have four terminal states? FILLED (filled), CANCELLED (cancelled), REJECTED (exchange rejected), FAILED (API-layer error). Exhaustive match is enforced by Task #7 — any unhandled status raises UnknownOrderStatusError. No silent unknowns.

Q8: Will this system be open-sourced? Not now. Reasons: (1) account-level risk logic is tightly coupled to personal circumstances — directly reusing is dangerous; (2) the triple lock and credential handling need case-by-case adjustment, publishing removes that protection for imitators; (3) I don't yet have the confidence to say the architecture is worth copying. After Joseph runs live for 12+ months with verified performance, I might open up non-sensitive modules.

Closing

This is the densest of the three posts, deliberately packed with numbers — because "real system" vs "imaginary system" comes down to whether numbers back it. Joseph going live today doesn't mean it lives forever. But it's at least in a verifiable, fixable, retraceable state.

If you're building your own system with AI, I hope these three posts save you some time. Killing three projects isn't shameful. Faking a success story is.

Back to the start: Part 1: Killing three projects before Joseph survived →

Methodology recap: Part 2: Seven principles from working with Claude →