/experience · field manual, section 02
experience
Production agents, the retrieval under them, and the evaluations that keep them honest.
detail leveleverything · 24 points
01
capabilities
Multi-agent systems
Routed specialists with typed tools, approval gates and checkpointed, interruptible runs, from a 17-tool COO agent to a LangGraph RCA investigator.
Retrieval
Scope-first GraphRAG over 2.7M chunks, embeddings-then-SQL with deterministic broadening, and HNSW iterative scans tuned to 195/200 recall at 0.25 s.
Evaluation
Planted-answer sets, point-in-time backtests with staged LLM critics, held-out fault suites and dual-provider migration harnesses.
Domains
Startup operations, automotive repair, football scouting, Kubernetes incident response and research tooling.
02jul 2025 – present
TheAgentic.AI
AI Engineer · AI infrastructure · San Francisco, California
jul 2025 – present
AI COO platform · founder operations
- Built the agent layer on pydantic-ai: a 17-tool COO orchestrator across pipeline, mail, calendar, team chat and data room, plus mail and meeting analysers that propose CRM moves.
- Rebuilt investor-fit scoring against a planted-answer eval (12 startups × 100 investors, 1,200 assessments a run): AUC 0.51 → 0.82, zero drift across repeated runs.
- Grounded search over 150k investors: embeddings pick candidates, SQL enforces filters, a fixed ladder broadens; HNSW iterative scans at 195/200 recall in 0.25 s (vs 6.3 s strict).
- Cut integration API traffic ~97% (3,244 → 99 calls per connection a day) and indexed the full data room: 0.29 s median search, worst case 42 s → 9 s.
- Moved every feature to a new model provider behind a 76-case dual-provider harness: 8 defects caught before ship, prompt caching cut cost 23–34%.
jul 2025 – present
Repair assistant · automotive
- Designed the diagnostic flow: clarifying questions, then three ranked root causes with diagnostic steps, procedures, parts, wiring schematics, TSBs and recalls, streamed over SSE.
- Three retrieval backends routed per vehicle: a 2.7M-chunk Neo4j GraphRAG, a 70K-manual OEM corpus (cosine + BM25 + operation ranking) and live web research.
- Organised retrieval as scope → rank → read: wrong-vehicle pages 60% → 4.4%, target-document hits 1/8 → 8/8, zero-result rate 60% → 20% with hybrid search.
- Pre-built the manual taxonomy and page index: lookups 30.6 s → 0.1 ms, page reads 12.8× faster, torque answers 147 s → 28 s, corpus 88 GB → 20 GB at full coverage.
jul 2025 – present
Football scouting · sports analytics
- Routed agents for discovery, valuation (12-axiom framework), transfer attribution (7-point verdict) and pre-match tactics over 572K players and 3.1M transfers.
- Grounded every answer in the warehouse without letting a model write raw SQL: agents query through schema-checked, parameterised tools, and any ad-hoc query is parsed down to one read-only SELECT over approved views, capped at 20 s.
- Point-in-time backtesting (before-date cut-offs on every data tool) judged by two-stage LLM critics: 53% → 71% across 106 replayed transfers over 5 seasons.
- Cut discovery queries ~50 s → 7 s by removing N+1 lookups; ingestion from 3 data sources plus an injury pipeline covering 11K injured players.
jul 2025 – present
Kubernetes RCA agent · site reliability
- Alert to postmortem: webhook ingest, dedup and SEV scoring, topology-grouped incidents, and a checkpointed LangGraph investigator with human-in-the-loop interrupts.
- Verdicts earn their confidence: 2–5 hypotheses with confirm and kill tests, 11 evidence gates and a cross-vendor falsifier; 8/12 strict on held-out faults.
- Owned PII redaction (card with IIN + Luhn, UPI, phone, email, token shapes) at every model, tool, storage and stream boundary, with zero false masks across 204 simulated responses.
- 22 investigation tools over 11 telemetry sources (Prometheus, Loki, Tempo, Jaeger, CloudWatch, the Kubernetes API and more), with read-only RBAC and SSRF guards.
03jun 2026
LiveGraph
Live multi-agent research canvas · personal project
jun 2026
Architect & builder
- A planner draws a live DAG of research, critic, prosecutor/defender debate, judge and editor agents; any node can be steered, edited, vetoed or rerun mid-run.
- Incremental recompute like a build system (sha256 input hashing, cached reuse, early cutoff): a local edit costs 38–59% of a fresh run.
- Cut a full run 373 s → 154 s at ~$0.036 per run with prompted-JSON outputs (about 3× faster than tool-call arguments), selective reasoning and provider pinning.
- Built the eval harness first: 30 custom questions plus a FRAMES subset, scoring factual precision, citation precision/recall and Brier calibration, with a no-debate ablation over 60 pilot runs.
- Event-sourced on Postgres with WebSocket fan-out and timeline replay; hybrid pgvector retrieval with trust-aware ranking; 167 pytest + 32 vitest tests.
04dec 2023 – feb 2024
Imperative Business Ventures
Mumbai, India
dec 2023 – feb 2024
Python Intern
- Built an AI reconciliation tool that matched debit and credit balances and flagged discrepancies.
- Shipped complaint sentiment scoring and document chatbots over CSV, Excel and text.
06
education
Thadomal Shahani Engineering CollegeBachelor of Engineering · University of Mumbai
2021 – 2025