Projects / CoinTrends.ai
CoinTrends.ai active
cointrends · last seen 2026-10-22 02:44
Monetization/product strategy session: agreed to build additively on the existing platform (no repositioning, same tiers/horizon/architecture). Gap-fill backlog in order: re-enable stopped feeds + recommendations; #123 NaN fix + prediction scorecard; baseline gate in ml-training; alerts; paper trading; Genie retrieval over own data; API keys after Locked Door; affiliate + newsletter; Open Markets pilot. Revenue = make Standard/Premium worth paying for + API + affiliate/newsletter. SOM est. $200-600k ARR on current architecture before Open Markets.
blocker 1 open
Lock down core-service /db/execStoredProc: public, no auth, runs any schema.function as postgres
urgent
The crypto538-core-service-production endpoint is reachable by anyone: Cloud Run ingress is "all" and allUsers has run.invoker. The handler (stored_proc.go handleExecStoredProc) has no authentication and no allowlist. It accepts any "schema.function" name plus arguments and runs it on the shared Cloud SQL instance as postgres, which is also used by AdaptiveSAT, Vajra and CitrineOS. That includes the dynamic-DDL routines (table_creation_package, tb_*_generator, sp_initialdata_generator). Not exploited or tested. Suggested fix: allowlist the procs actually used plus a shared-secret or IAM check, or remove the route if unused. Validate callers first, per the cleanup rule.
crypto538-core-service/stored_proc.go:135crypto538-core-service/main.go:494
added 10-20 12:44
· by saravanan@scrumclaw.ai
· claude-cowork
todo 36 open
Live predictions are worse than "price stays the same" and ~50% on direction — fix before selling predictions
urgent
DB query 2026-10-19, last 30 days, slice 24 (2h ahead), all coins except TON: model error is ~2x the naive no-change baseline (BTC 0.555% vs 0.333%, ETH 0.868% vs 0.436%, DOGE 2.80% vs 0.71%), direction hit rate 49-51% (coin flip), and strong up-bias (DOGE predicts up 100% of the time, SOL/AVAX 89%). Also 2,489 predictions stored as NaN in 30 days. dbo.predictionvsactual is empty, so accuracy was never tracked. Retraining (#90) alone may not fix this — evaluate against the naive baseline in #115 and consider the model/target design.
#90#115dbo.<pair>_predictiondatadbo.predictionvsactual
added 10-19 14:07
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: I want you to access the database and query how the data looks like
claude: Compared 30 days of 2h-ahead predictions with actual prices: model error ~2x a no-change baseline, direction ~50%.
Lock down core service: require auth, especially /db/execStoredProc, /deletefile, /uploadfile, /purgeStageData, /insertdb
urgent
crypto538-core-service is deployed --allow-unauthenticated. Options: make it private (IAM invoker = web-service SA + Scheduler SA) and have the webapp go through web-service only; or add an API-key/OIDC check on admin/write routes. Ties to the item about the webapp calling core directly.
crypto538-core-service/main.go:486-532crypto538-core-service/cloudbuild.yaml
added 10-16 19:08
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: The core service needs no authentication, including /db/execStoredProc, /insertdb, /uploadfile, /deletefile.
saravanan: create cointrends in tracker and push these items there
Rotate hardcoded API keys (CoinMarketCap, TAAPI, OpenAI) and web-service .env secrets; move to Secret Manager
urgent
Keys hardcoded in core main.go constants (per CODE_REVIEW.md #1). web-service .env / .env.staging / .env.production committed with TOKEN_SECRET_KEY, ENCRYPTION_KEY, Stripe keys etc. Rotate, then inject via Secret Manager env refs on Cloud Run.
crypto538-core-service/main.go:157-165crypto538-core-service/main.go:331crypto538-web-service/.envcrypto538-core-service/CODE_REVIEW.md
added 10-16 19:08
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: CoinMarketCap, TAAPI and OpenAI keys are hardcoded in main.go; web-service .env files hold JWT and Stripe secrets.
saravanan: create cointrends in tracker and push these items there
Revoke the committed prod service-account keys and switch to keyless auth
urgent
prj-crypto538-prod-260529-*.json key files are in crypto538-core-service, ml-core-service and CoinTrends_Python. Disable/delete those keys in IAM, remove from repos (history too, or treat as burned), rely on Cloud Run service identity / Workload Identity Federation.
crypto538-core-service/prj-crypto538-prod-260529-9964278419d2.jsonml-core-service/prj-crypto538-prod-260529-c7d42674bd4d.json
added 10-16 19:08
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: Production service-account key JSONs are committed in two repos and CoinTrends_Python.
saravanan: create cointrends in tracker and push these items there
Rotate the postgres superuser password and remove it from code
urgent
Password is committed/hardcoded in crypto538-core-service/config/production.yaml + staging.yaml, ml-core-service/main.py, and every CoinTrends_Python script (with the public DB IP). Rotation affects every service on the shared instance. Move to Secret Manager (db_password secret already exists) and read at runtime.
crypto538-core-service/config/production.yamlcrypto538-core-service/config/staging.yamlml-core-service/main.py::get_db_connectionCoinTrends_Python/*.py
added 10-16 19:08
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: The postgres superuser password is committed in several places and every service connects as that superuser.
saravanan: create cointrends in tracker and push these items there
Close Cloud SQL to the internet: remove 0.0.0.0 authorized network and require SSL
urgent
Instance crypto-track-321e8b45 has a public IP (34.57.238.28), requireSsl=false (ALLOW_UNENCRYPTED_AND_ENCRYPTED), and an authorized network "for EV Connection" = 0.0.0.0 — effectively open to the internet with only the postgres password protecting it. Instance is shared with AdaptiveSAT, Vajra, CitrineOS, tracker. Coordinate with Vajra before removing the EV entry; prefer Cloud SQL connector/Auth Proxy for all clients.
gcloud sql instances describe crypto-track-321e8b45
added 10-16 19:08
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: The database is effectively open to the internet: public IP, SSL optional, authorized network 0.0.0.0.
saravanan: create cointrends in tracker and push these items there
Re-enable stopped feeds: news, Fear & Greed, gainers/losers, trade recommendations
high
fetchCryptoNews, insertFearAndGreedHandler commented out in core insertDataIntoDB; recommendations stopped 2025-03; gainers 2025-10-13. Uncomment / re-schedule, refresh provider keys (via Secret Manager). Brings Market Insights back to life. Gap-fill #1.
crypto538-core-service/main.go
added 10-22 02:44
· by saravanan@scrumclaw.ai
· claude-cowork
2026-10-22: news (RSS), Fear & Greed, gainers/losers and trending re-enabled every 30 min inside /insertdb (core rev 00080); first run 14:00 UTC OK in 1.4 s. Dedupe + unique keys + daily purge (news/F&G 90 days, snapshots 2 days). Remaining here: trade recommendations — owner: later.
PR "Coin Roster": make cryptomaster the only coin list; onboard any coin with dbo.onboard_coin()
high
8 steps, one PR per layer: DB onboarding function; DB collection; DB readers; core-service roster cache; ml-core-service; webapp; ml-training Batch task count; end-to-end proof with a new coin. Full instructions: cointrends/plans/PR-coin-roster.md (also in the project docs as claude/PR-coin-roster.md). Open questions: ship before or after Locked Door (#129); how DDL gets executed; which repo holds DB migrations.
plans/PR-coin-roster.md#129#118
added 10-20 12:53
· by saravanan@scrumclaw.ai
· claude-cowork
2026-10-22: step 6 (webapp) live via PR #59/#60 — aliases from the live coin list, icon fallback; NormalCoinInfo 'BTC' text is in a JSX comment (never rendered). HYPE + ZEC onboarded, DOGE/ADA/AVAX retired (011/012). Remaining: step 7 (ml-training taskCount helper, per-asset slice) and step 8 (HYPE/ZEC models after retrain).
2026-10-23: step 7 done (ml-training PR #4: batch/submit.sh sets taskCount from the active coins). Per-asset slice/interval deferred to Open Markets #131 (all crypto = slice 24/120 min). Remaining: step 8 = HYPE/ZEC models after enough history (retrain, not before Oct 27).
Create a dedicated least-privilege DB user for ml-training before scheduling Batch training
high
Batch training currently connects as `postgres`, the admin of the shared Cloud SQL instance crypto-track-321e8b45, which other products also use. Before scheduling, create a user with access to only the CryptoTrack_Prod objects training needs: getcryptodata, cryptomaster, cryptopredictionslicemaster and updatetracking. Store its password in a new secret and point batch/*.json at it.
HopewellPartners/ml-training batch/*.json#83-#87
added 10-20 02:16
· by saravanan@scrumclaw.ai
· claude-cowork
Training 4/5: Run a full retrain and validate new models against predictionvsactual before switching inference
high
Write new models to a staging prefix first, backtest vs the Oct-2025 models (MAE/RMSE, directional accuracy) on recent data, then promote by updating cryptopredictionslicemaster paths. Keep the old files as rollback.
dbo.predictionvsactualdbo.cryptopredictionslicemaster
added 10-16 19:12
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based)
claude: Step 4: full retrain to a staging prefix, backtest vs old models, then promote.
Training 3/5: Create Cloud Run Job with L4 GPU, dedicated service account, Secret Manager creds, task-index fan-out
high
gcloud run jobs create ml-training --gpu 1 --gpu-type nvidia-l4 (us-central1; confirm GPU quota/region availability first). Dedicated SA with only: Cloud SQL Client, read/write on gs://nlp_ml_files/prediction_files, secret accessor on the DB secret. --tasks = assets × 12 offsets (120 today), --parallelism capped to quota; CLOUD_RUN_TASK_INDEX → (asset, minute). Codify in crypto538-terraform. Note: #98 — only this SA should be able to write model files.
#98crypto538-terraform/environments/prod/main.tf
added 10-16 19:12
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based)
claude: Step 3: Cloud Run Job with L4 GPU, least-privilege SA, task-index fan-out over asset × minute.
Training 2/5: Build GPU container image (CUDA PyTorch base) via Cloud Build → Artifact Registry
high
Dockerfile on a CUDA-enabled PyTorch base, pinned versions, psycopg2 + cloud-sql-python-connector + google-cloud-storage. Push to us-central1-docker.pkg.dev/prj-crypto538-prod-260529/crypto538-container-images/ml-training. Cloud Build config alongside.
added 10-16 19:12
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based)
claude: Step 2: GPU container image built by Cloud Build into Artifact Registry.
Training 1/5: Refactor CoinTrends_Python into one secret-free training package (merge orchestrator + trainer)
high
Take *_latest.py versions only. Fold master_create_update_latest.py decision logic (create vs incremental update, check_record_count) and api_latest.py/ModelCreation_latest.py training into one CLI entrypoint: train --asset BTCUSD --minute 15. Remove Flask/localhost HTTP, hardcoded DB host/password, SA key, Windows paths; read config from env. Fix cursor.commit() and NaN-branch bugs (#91). Make device selection automatic (cuda if available else cpu). Once secret-free, this can go into a git repo (unblocks decision #108).
CoinTrends_Python/master_create_update_latest.pyCoinTrends_Python/api_latest.pyCoinTrends_Python/ModelCreation_latest.pyCoinTrends_Python/darnn.py
added 10-16 19:11
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based)
claude: Broke #90 into subtasks; step 1 is a single secret-free training entrypoint.
Code read 2026-10-16 — the *_latest training path is incomplete, not just split up: (1) CreateModel.train_model() only saves .pth/.npy to a local ModelFiles folder; no GCS upload and no write of modelfile/scalerfile to cryptopredictionslicemaster on success (returned path also omits the _{minute} suffix and is unused). (2) /updatemodel loads and overwrites LOCAL files, 50 fixed epochs, no validation/early stop, evaluates on training data, no upload. (3) Nothing in the Postgres-era code writes dbo.updatetracking (old MSSQL api.py did), so check_record_count's baseline never moves. (4) Bug in api_latest.py:193 `target_max = target_max` should be target_max_curr. Explains why all bucket files share one timestamp (2025-10-30 02:08) — a manual bulk upload. New trainer must add: GCS upload, slicemaster path update, updatetracking insert, validation for incremental updates.
Built 2026-10-16: ~/Documents/GitHub/cointrends/ml-training (not in git yet; secret-free). Per Saravanan: ONE program (trainer.py) with TWO separate methods — ModelTrainer.create_model (new model from scratch) and ModelTrainer.train_model (continue training an existing model) — not mixed. CLI: `python trainer.py create|train --asset BTCUSD --minute N|--all-minutes [--publish]`. Default writes to prediction_files_staging and leaves the DB alone; --publish writes live, updates cryptopredictionslicemaster (create) and updatetracking (both); train backs up live files first and only replaces a model if it's at least as accurate on the latest 100 rows. Also fixes: best-epoch weights, cursor.commit(), target_max widening (#91), batch-of-1 crash in darnn.py. tests/smoke_test.py passes (both methods, CLI flows on a fake DB, and loading the output with ml-core-service's own darnn.py + predict steps). train_model also verified against the live BTCUSD_model_120_0.pth (read-only). Dockerfile: CPU default, GPU via build arg — ready for the CPU-vs-GPU test (#113/#114). Not yet run against the real DB.
Generalise per-asset hard-coding — prerequisite for stocks, commodities and indexes
high
Each coin has its own table family (pricedata, ohlcv, predictiondata, indicators, indicators_plus + view). Coin list is hard-coded in CASE statements in getcryptodata, getpricedataforcurrencies, getcurrencydetails, and in crypto_id == 1 or ... lists in ml-core-service and CoinTrends_Python. Design an asset-agnostic schema (asset master + long-format tables keyed by asset_id) and drive code from cryptomaster/asset master.
dbo.getcryptodatadbo.getpricedataforcurrenciesdbo.getcurrencydetailsml-core-service/main.py::func
added 10-16 19:09
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: Coin-specific logic is hard-coded everywhere; adding any asset means editing many places by hand.
saravanan: create cointrends in tracker and push these items there
Replace log.Fatalf in core-service request paths and goroutines with returned errors
high
insertDataIntoDB and other handlers call log.Fatalf on upstream/DB errors, which kills the whole Cloud Run instance (CODE_REVIEW #7, #17). Return errors and let the handler respond 500.
crypto538-core-service/main.go::insertDataIntoDBcrypto538-core-service/CODE_REVIEW.md
added 10-16 19:09
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: The core service calls log.Fatalf inside request paths, which kills the whole process on any upstream error.
saravanan: create cointrends in tracker and push these items there
Create least-privilege DB users per service and a read-only analysis role
high
core-service, ml-core-service and web-service all connect as postgres superuser on a shared instance. Create e.g. cointrends_core (rw on CryptoTrack_Prod.dbo), cointrends_ml (read + prediction tables), cointrends_web (CryptoUserBilling_Prod), cointrends_ro (SELECT only, for analysis/Claude).
added 10-16 19:09
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: There's no read-only database role, and every service connects as the superuser.
saravanan: create cointrends in tracker and push these items there
Fix training pipeline bugs: cursor.commit() and undefined connection in NaN-loss branch
high
ModelCreation_latest.py calls cursor.commit() — psycopg2 cursors have no commit() (needs cnxn.commit()), and the second NaN branch uses cnxn which is only defined in the first branch. Result: a failed training run never gets marked activeflag='I'. Also .cuda() hard-coded; pyodbc import leftover.
CoinTrends_Python/ModelCreation_latest.py:315-340
added 10-16 19:09
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: The training pipeline has real bugs: cursor.commit() doesn't exist in psycopg2 and the NaN-loss branch uses an undefined connection.
saravanan: create cointrends in tracker and push these items there
Move model training to cloud and retrain — prediction models last trained 2025-10-30
high
Inference runs every 5 min (Scheduler prediction_calls → ml /run_prediction) on 120 DARNN models in gs://nlp_ml_files/prediction_files last written 2025-10-30. Training lives in CoinTrends_Python (local Windows GPU box, localhost Flask chain :17000 → :5000). Target: Cloud Run Job or Vertex AI custom job with GPU, parameterised by asset + minute offset, triggered by Scheduler, creds from Secret Manager, metadata write-back to cryptopredictionslicemaster.
CoinTrends_Python/master_create_update_latest.pyCoinTrends_Python/api_latest.pyCoinTrends_Python/ModelCreation_latest.pygs://nlp_ml_files/prediction_files/
added 10-16 19:09
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: Predictions run on year-old models — the 120 model files were last trained 30 Oct 2025 and nothing retrains them.
saravanan: create cointrends in tracker and push these items there
Target confirmed by Saravanan (2026-10-16): Cloud Run **Jobs with GPU** (not Vertex AI). gcloud in this project supports `gcloud run jobs create --gpu 1 --gpu-type nvidia-l4`. Shape: one container image with orchestrator + trainer merged (no localhost :17000/:5000 chain); one job execution with N tasks, CLOUD_RUN_TASK_INDEX mapped to (asset, minute offset); Cloud Scheduler triggers it; DB via Cloud SQL connector, creds from Secret Manager; models written to gs://nlp_ml_files/prediction_files/; cryptopredictionslicemaster updated on success. Subtasks added under this item.
API keys for Premium/Developer in front of existing core endpoints (after Locked Door #129)
API-key middleware in web-service proxying /getPriceData, /getCryptoIndicators, /getActualPrediction etc. Sell as B2B to bot builders / small fintechs. Depends on #129. Gap-fill #7.
#129
added 10-22 02:44
· by saravanan@scrumclaw.ai
· claude-cowork
Ask Genie: retrieval over our own data (indicators, predictions, scorecard, news)
Keep Ollama; add retrieval so Genie answers "why is BTC predicted down?" from our tables. API LLM only as optional fallback. Gap-fill #6.
added 10-22 02:44
· by saravanan@scrumclaw.ai
· claude-cowork
Paper trading (currently "coming soon") on existing pricedata + predictions
paper_trades table in CryptoUserBilling_Prod, two web-service endpoints, UI page. Premium perk. Gap-fill #5.
added 10-22 02:44
· by saravanan@scrumclaw.ai
· claude-cowork
Alerts: prediction flips / confidence changes via email (Azure) for Standard+ users
Watchlist + alert prefs in web-service (CryptoUserBilling_Prod); reuse Azure Communication Email. Gap-fill #4.
added 10-22 02:44
· by saravanan@scrumclaw.ai
· claude-cowork
PR "Open Markets": extend the platform to stocks, indexes, commodities and FX (after Coin Roster)
Design in cointrends/plans/PR-open-markets.md (also in project docs). Main pieces: per-asset prediction config (horizon, step, offsets, feature set); calendar-aware sampling for markets that close; versioned feature sets with model metadata; context data as roster assets (replaces the frozen marketindex/metalsdata); pluggable data providers with locally computed indicators; site routes and entitlements per asset class. Open decisions: pilot assets, data provider and budget, URL scheme, subscriptions model.
plans/PR-open-markets.mdplans/PR-coin-roster.md#130
added 10-20 12:55
· by saravanan@scrumclaw.ai
· claude-cowork
Training 5/5: Schedule recurring retrains with Cloud Scheduler and add failure alerting
Cloud Scheduler → Cloud Run Jobs run API (cadence TBD, e.g. weekly full / daily incremental using the existing updatetracking logic). Alert on failed executions and on stale models (modelfile older than N days). Retire the Windows box flow.
added 10-16 19:12
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based)
claude: Step 5: Scheduler-driven retrains plus alerting; retire the local box.
Align pricing page with what exists (analyst chat, trading agent; Premium vs Standard coins)
/pricing promises "Live chat with an Analyst" and "On-demand automated trading agent" (not built). Premium lists the same 10-coin access as Standard. Either build, mark "coming soon", or remove.
↗ https://www.cointrends.ai/pricingcrypto538-webapp/src/subscriptionPlansPage/page.jsx
added 10-16 19:09
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: The pricing page promises features that don't exist (analyst chat, trading agent); Premium lists the same coins as Standard.
saravanan: create cointrends in tracker and push these items there
Consolidate the three LLM backends and cut always-on Ollama cost
crypto538-askgenie-service (Ollama llama3, 16 GB, 4 CPU, min-instances=1, no CPU throttling — billed 24/7) plus a separate llama-cpu Cloud Run service plus OpenAI calls from core. Pick one backend for Genie/recommendations and scale the rest to zero or delete.
crypto538-askgenie-service/cloudbuild.yamlcrypto538-core-service/main.go::llamaCompletion
added 10-16 19:09
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: Three LLM backends run side by side; Ollama runs at 16 GB with min-instances=1 and is billed around the clock.
saravanan: create cointrends in tracker and push these items there
Revisit the 12-models-per-asset design before scaling to new asset classes
One DARNN per 5-minute offset (desired_minute 0..55) → 12 models per coin, 120 today. Consider one model per asset with minute-of-hour as a feature, and a horizon suited to each asset class (e.g. daily for stocks/indexes).
CoinTrends_Python/trigger_updated.py
added 10-16 19:09
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: Each asset needs 12 models (one per 5-minute offset), which multiplies training cost and makes the model count grow fast.
saravanan: create cointrends in tracker and push these items there
Restrict write access to gs://nlp_ml_files — ML service unpickles models with weights_only=False
ml-core-service torch.load(..., weights_only=False) and np.load(allow_pickle=True) on files from GCS: anyone who can write to that bucket can execute code in the ML service. Limit bucket writers to the training job identity; consider safetensors / weights-only state_dict.
ml-core-service/main.py::predict
added 10-16 19:09
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: The ML service unpickles model files from GCS (weights_only=False); bucket write access = code execution.
saravanan: create cointrends in tracker and push these items there
Route the webapp through web-service only (stop calling core-service directly)
crypto538-webapp uses REACT_APP_COIN_DATA_BASE_URL pointing at the public core-service Cloud Run URL, bypassing web-service auth/plan checks. Needed before core-service can be made private.
crypto538-webapp/.env.production::REACT_APP_COIN_DATA_BASE_URL
added 10-16 19:09
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: The webapp calls the core service directly, bypassing the web service's authentication layer.
saravanan: create cointrends in tracker and push these items there
Use one shared DB connection pool in core-service instead of a new pool per call
getDbConnection() → connectWithConnector() builds a new sql.DB (and Cloud SQL dialer) on nearly every call, with defer db.Close(). Create one pool at startup and reuse it; set pool limits once.
crypto538-core-service/main.go:570-650
added 10-16 19:09
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: A new connection pool is created on almost every call, instead of one shared pool.
saravanan: create cointrends in tracker and push these items there
Work through open CODE_REVIEW.md issues in core-service (timeouts, goroutines, bounds checks, double Close, T-SQL @ID)
April 2026 review lists 20 open issues. Highlights: no HTTP client timeouts (#6), unbounded goroutines in fetchCryptoNews (#8), Choices[0] with no bounds check (#3), double db.Close() in UpdateCryptoNewsExtractByURL (#4), T-SQL @ID syntax in IsNewsExtractNull fails on Postgres (#5), no MaxBytesReader (#11), rows.Err() unchecked (#12).
crypto538-core-service/CODE_REVIEW.md
added 10-16 19:09
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: The April code review's 20 issues are still open (timeouts, goroutines, Choices[0], double Close, T-SQL @ID).
saravanan: create cointrends in tracker and push these items there
Zero-build revenue: exchange affiliate links on coin pages + revive newsletter with weekly auto-digest
low
Affiliate CTAs on /en/coins/:id; weekly digest from scorecard + Market Insights via Azure email on existing /products/newsletter; sponsorship slot. Gap-fill #8.
added 10-22 02:44
· by saravanan@scrumclaw.ai
· claude-cowork
Final cleanup: remove/confirm everything Claude added to GCP during the review
low
Footprint so far (none of it changes how prod runs): (1) IAM: Cloud SQL service account p475046183936-kjdopc@gcp-sa-cloud-sql granted objectAdmin on gs://crypto538-prj-crypto538-prod-260529-bucket (needed for schema/query exports) — rollback: gsutil iam ch -d. (2) gs://crypto538-prj-crypto538-prod-260529-bucket/claude-schema-exports/ (2 schema-only .sql files; query CSVs are deleted after each read). (3) Cloud Build 7d835ecf may have pushed ml-training:test-cpu / :test-gpu to Artifact Registry crypto538-container-images — delete if unused. (4) No Cloud Run jobs, no DB writes, no scheduler changes. Local only: ml-training/, ship.sh, commit_razorpay_deprecation.sh, 'Tracker project_id' line in cointrends/CLAUDE.md. Merged code change: PR #70 (comments only).
gs://crypto538-prj-crypto538-prod-260529-bucket/claude-schema-exports/Cloud Build 7d835ecf-2e1a-430a-bc11-040a85bd4ee6↗ HopewellPartners/crypto538-core-service#70
added 10-19 14:19
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: we need a clean up step by step with proper validation and rollback at the end of all our work
claude: Listing everything I've added to GCP so the final cleanup can remove or keep it deliberately.
Restore engineering basics: tests in Cloud Build, a working stage env, READMEs
low
Test step commented out in core-service cloudbuild.yaml; stage LB and purge job commented out in terraform; webapp/web-service/terraform sit on a stage branch diverged from master; READMEs empty; log retention 30 days. Also split the 3.9k-line core main.go into packages.
crypto538-core-service/cloudbuild.yamlcrypto538-terraform/environments/stage/main.tf
added 10-16 19:10
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: Testing and environments are thin: tests disabled in Cloud Build, stage infra commented out, READMEs empty.
saravanan: create cointrends in tracker and push these items there
Clean up CryptoTrack_Prod junk and legacy objects
low
Duplicate public.getcryptodata variants, public.gainers_json, public.market_metal_data, dbo.table_1, dbo.table_2, dbo.debugtable, dbo.jsondatatowork; legacy dbo.usertable / dbo.paymentdetails / manageuserandpayments() (users now live in CryptoUserBilling_Prod). Check row counts and archive before dropping.
added 10-16 19:10
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: The database carries MSSQL heritage: duplicate functions in public, junk tables, legacy usertable/paymentdetails.
saravanan: create cointrends in tracker and push these items there
Remove deprecated Razorpay code from core-service (marked in PR #70)
low
Delete razorpay_webhook.go, razorpay_webhook_test.go and the /webhooks/razorpay route; also the dead /users and /payments handlers. Zero traffic in 30 days of Cloud Run logs. Deprecation comments merged 2026-10-16 (c3ef045).
↗ HopewellPartners/crypto538-core-service#70crypto538-core-service/razorpay_webhook.gocrypto538-core-service/main.go:490
added 10-16 19:10
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: Razorpay we dont use it - just add comments above the code blocks and do not delete. we will do it later.
claude: Deprecation notes merged in PR #70; removal tracked as a later cleanup.
decision 7 open
Monetization strategy: build additively on existing platform, no repositioning or tier changes
Keep Basic/Standard $10/Premium $15, 120-min horizon, current architecture. Make tiers worth paying for using features already half-built (all coins, alerts, scorecard, Genie-on-own-data, paper trading, API key). Extra channels: exchange affiliate links, newsletter sponsorship, B2B API = existing core endpoints behind an API key. Open Markets later as designed (additive). Estimates: crypto-only TAM ~$0.3-0.6B, multi-asset $3-6B; SAM $150-300M crypto / $1B+ with Open Markets; SOM $200-600k ARR on current architecture.
added 10-22 02:44
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: I believe your suggestions would break what we built - I wanted something built on what we already have to fill the gaps.
claude: reframed as additive gap-fill on existing components, same tiers and horizon.
Model training runs as GCP Batch jobs on GPU (Spot L4), not Cloud Run — supersedes #111
Saravanan 2026-10-20: train on GPU using Google Cloud Batch. Shape: same ml-training container (CUDA build) in Artifact Registry; Batch job with one task per asset (BATCH_TASK_INDEX), g2-standard-4 (1x L4) Spot VMs, installGpuDrivers, parallelism ≤ quota (L4: 8 on-demand / 8 preemptible in us-central1), dedicated service account, DB via Cloud SQL connector, password from Secret Manager, logs to Cloud Logging. Triggered by Cloud Scheduler calling the Batch jobs API. Batch API not yet enabled. Updates #113/#114/#116.
#90#111#113#114#116
added 10-20 01:51
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: Lets use GPU to train - I want the batch process in gcp - not the cloud run with GPU
saravanan: tell me how to deploy that way
Staging stays retired — production-only, with local tests, pre-release backup, live validation and a ready rollback
Only production runs in the cloud. Leftovers: 3 idle staging Cloud Run services in prj-crypto538-stage-443113 (no triggers deploy to them), stage.cointrends.ai LB commented out in terraform, CryptoTrack_Stage / CryptoUserBilling_Stage DBs on the prod instance. Merging to the `stage` branch deploys nothing; `master` deploys prod. Leftovers can be reviewed in the final cleanup.
added 10-20 01:45
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: we stopped staging environmnt - its only production is enabled in cloud
claude: Confirmed; offered to log a cheap staging revival.
saravanan: leave staging retired
Correction (Claude, 2026-10-20): staging is NOT fully off. The stage project prj-crypto538-stage-443113 still has 5 active Cloud Build triggers: ml-core-service-stage, crypto538-core-service-stage, crypto538-webapp-stage, crypto538-web-service-stage (all on ^stage$) and crypto538-terraform-stage (^prod$). Today: web-service-stage deployed PR #28 to the idle staging web service (SUCCESS); webapp-stage FAILED twice (PR #56, #57) because the target bucket gs://crypto538-webapp-staging no longer exists — React build itself succeeded. Production unaffected. Options: disable the stage triggers (reversible) now, or leave until final cleanup; every push to a stage branch will keep producing failed webapp builds/emails until then.
No changes to running systems until a final, step-by-step cleanup phase with validation and rollback
Saravanan 2026-10-19: do not stop or change anything that is running (schedulers, Cloud Run services, DB rows such as slices 1-23, live models). Findings and fixes are collected now; all production changes happen together at the end as an ordered cleanup plan where every step has a validation check and a rollback.
added 10-19 14:19
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: Keep all the changes to later stage - i do not want to stop anything that is running. we need a clean up step by step with proper validation and rollback at the end of all our work
claude: Recorded; production stays untouched until the final cleanup phase.
Clarified by Saravanan 2026-10-19: the "defer to the end" rule applies to CLEANUP only — removing or switching off things that currently run or exist (dead code like Razorpay, slices 1-23, junk/legacy tables, unused services/LLM backends, Claude's GCP footprint #121). FIXING issues (bugs, broken features, security fixes, training to cloud) proceeds as normal, one change at a time through ship.sh/PR with his OK. Cleanup items: #104 Razorpay removal, #105 DB junk, #100 LLM consolidation, #121 Claude footprint, slices 1-23 deactivation.
Model training moves to Cloud Run Jobs with GPU (NVIDIA L4) — replaces the local Windows GPU box
Chosen over Vertex AI custom jobs: same platform as the rest of the stack (Cloud Run, Artifact Registry, Cloud Build, Scheduler), pay-per-execution, task-index fan-out fits the asset × minute-offset model grid. Parent work item: #90.
#90
added 10-16 19:11
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based)
claude: It's #90; recorded Cloud Run Jobs with GPU as the target and broke it into subtasks.
Keep CoinTrends_Python (training code) outside git until secrets are stripped
The folder holds the DB password, public DB IP and an SA key in every script. Revisit after the secret-rotation items are done.
added 10-16 19:10
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: right now - let it sit outside of the repo - due to the secrets it contains.
claude: Recorded; it stays out of the repos for now.
Dropped Razorpay for CoinTrends — Stripe via web-service is the only billing path
Razorpay was only used for the VajraEV experiment. Code kept for now with deprecation comments (PR #70); deletion later.
added 10-16 19:10
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: i think razorpay is an experiment - you can drop it
saravanan: no i was using vajraev related only.
claude: Stripe through web-service + CryptoUserBilling_Prod is the only billing path.
note 3 open
DB snapshot 2026-10-19: what's fresh and what stopped
Fresh (5-min): pricedata (~518k rows/coin since 2024-10-29), indicators, marketindex, metalsdata, predictiondata, pricedatacache. Stopped: OHLCV Mar/Apr 2025; tradingrecommendation 2025-03-26; cryptonews & feargreedindex 2025-10-03; gainers/losers/trending 2025-10-13; updatetracking last 2025-01-30 (so the retrain baseline is 20 months old); TON price 2026-06-17. Empty: predictionvsactual, predictiontracking. Slice master has 24 slices per coin (5..120 min) all flagged active with modelfile set, but only slice 24 (120 min) models exist in GCS and are used. Legacy usertable 8 rows, paymentdetails 5 rows.
added 10-19 14:07
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: I want you to access the database and query how the data looks like
claude: Ran read-only queries via Cloud SQL export; summary of fresh vs stopped feeds.
Correction from Saravanan (2026-10-19): the slice setup is intentional. Earlier design predicted all 24 horizons (slices 1-24, 5..120 min). Current design: every 5 minutes predict the price at +120 min (slice 24 only), choosing the model file by the current minute offset (_0, _5 ... _55). Slices 1-23 are leftovers from the old design — not used, but still flagged active with modelfile set; optional low-priority cleanup to mark them inactive. The accuracy check in #117 measured exactly this design (prediction at t vs actual at t+120).
Git flow: use ~/Documents/GitHub/cointrends/ship.sh — one short-lived branch per change, merged, deleted and pruned in one run
Cloud sessions started from the Cowork folder have no GitHub repo access, so Claude edits the local checkout and Saravanan runs ship.sh (branch → commit → push → PR → squash-merge → delete branch → ff base → git fetch --prune). No orphaned branches.
added 10-16 19:10
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: i dont want lot of branches created and orphaned
saravanan: Add that [git fetch --prune] as part of your script
claude: ship.sh now does the full flow and prunes stale refs on success or failure.
Users/billing live in CryptoUserBilling_Prod; market/ML data in CryptoTrack_Prod — both on shared instance crypto-track-321e8b45
Instance (POSTGRES_14, 1 vCPU / 3.75 GB, 20 GB) is shared with AdaptiveSAT, Vajra, CitrineOS and the tracker. Schema digests in the claude.ai CoinTrends project: claude/cointrends-db-schema-notes.md and claude/cointrends-architecture-notes.md.
added 10-16 19:10
· by saravanan@scrumclaw.ai
· claude-cowork
from the conversationsaravanan: user lives in cryptouserbilling_prod
claude: Got it — pulled that schema too; user, payment_details, user_reviews plus TypeORM bookkeeping.
Add item
Recently completed
Prediction scorecard: score every stored prediction vs actual and vs naive baseline; fix #123 NaN
todo
done 10-23 01:32
· by saravanan@scrumclaw.ai
ml-training: enforce beats-naive-baseline gate before publishing a model file
todo
done 10-22 22:07
· by saravanan@scrumclaw.ai
Add a managed certificate for bare cointrends.ai (browsers show a cert error; http redirects there)
todo
done 10-22 16:36
· by saravanan@scrumclaw.ai
TON price feed stopped 2026-06-17 but TON predictions keep being written on stale data
todo
done 10-22 14:48
· by saravanan@scrumclaw.ai
ml-training: fill missing indicator values the same way as ml-core-service before retraining
todo
done 10-22 14:48
· by saravanan@scrumclaw.ai
ml-core-service picks the model by the current minute, not the 5-minute slot (cold start → 404, no prediction)
todo
done 10-22 14:48
· by saravanan@scrumclaw.ai
Refresh coin roster: onboard current top-10 by market cap, retire dead coins via onboard_coin
todo
done 10-22 14:48
· by saravanan@scrumclaw.ai
Return HTTP 200 for SPA deep links from the load balancer (SEO)
todo
done 10-22 14:48
· by saravanan@scrumclaw.ai
Webapp deploys don't reach users promptly: index.html cached by Cloud CDN and browsers
todo
done 10-22 14:48
· by saravanan@scrumclaw.ai
Fix Highcharts indicator CDN path (503s) on coin pages
todo
done 10-22 14:48
· by saravanan@scrumclaw.ai
dbo.getmostrecentindicators is broken: reads non-existent dbo.<coin>usd_indicators tables
todo
done 10-22 14:48
· by saravanan@scrumclaw.ai
Prediction accuracy chart fails for BTC/SOL: core returns empty 200 when data contains NaN
todo
done 10-22 14:48
· by saravanan@scrumclaw.ai
Re-enable Fear & Greed, gainers/losers and trending inserts in the collector
todo
done 10-22 14:48
· by saravanan@scrumclaw.ai
Re-enable crypto news ingestion — Market Insights stuck at Oct 2025
todo
done 10-22 14:47
· by saravanan@scrumclaw.ai
Indicator rows link to the previous price row (5-min offset in training data)
todo
done 10-21 13:35
· by saravanan@scrumclaw.ai
Fix coin detail/prediction pages: isSubscriptions never matches the `symbol` field
todo
done 10-20 01:43
· by saravanan@scrumclaw.ai
Webapp: coin-page deep links/refresh bounce to homepage (duplicate 'user' cookie) + ticker URL support
todo
done 10-20 01:43
· by saravanan@scrumclaw.ai
Log
activeMonetization/product strategy session: agreed to build additively on the existing platform (no repositioning, same tiers/horizon/architecture). Gap-fill backlog in order: re-enable stopped feeds + recommendations; #123 NaN fix + prediction scorecard; baseline gate in ml-training; alerts; paper trading; Genie retrieval over own data; API keys after Locked Door; affiliate + newsletter; Open Markets pilot. Revenue = make Standard/Premium worth paying for + API + affiliate/newsletter. SOM est. $200-600k ARR on current architecture before Open Markets.
· saravanan@scrumclaw.ai
from the conversationsaravanan: what are the best avenues to make money, what to build for reliability/usefulness, TAM/SAM/SOM?
claude: proposed repositioning + new tiers + scorecard-led GTM.
saravanan: that would break what we built - I want to build on what we already have to fill the gaps.
claude: reframed as additive gap-fill backlog on existing components; asked to log to tracker.
activeml-training on GCP Batch.
Done:
- New private repo HopewellPartners/ml-training (447a734) with HISTORY.md.
- Batch API enabled in prod.
- Service account ml-training@ created with 6 scoped roles.
- Job files set to DB user postgres + secret db_password (fingerprint matches live).
Pending: PR with those job-file and doc changes (user runs ship.sh).
NEXT: step 4, build and push the GPU image (cloudbuild-gpu.yaml), then step 6, test run batch-test.json to staging only.
· saravanan@scrumclaw.ai
idlePaused at Saravanan's request. Done: ml-training/trainer.py built (one program, separate create_model / train_model methods), offline tests pass, verified against a live BTC model. In flight when paused: Cloud Build 7d835ecf (us-central1) building ml-training:test-cpu and :test-gpu images for the CPU-vs-GPU test — images only, no Cloud Run jobs created, nothing published. NEXT when resumed: re-auth gcloud (token expired), check build status, then run the 10-epoch CPU vs L4 GPU comparison on one coin/minute to staging.
· saravanan@scrumclaw.ai
from the conversationsaravanan: lets relook at it today. Pause it for now
claude: Paused; recorded where the CPU-vs-GPU test stands.
activeFull review of the 6 repos, live site, GCP and both DB schemas done. Logged 25 items: 6 urgent (coin-page 404, Cloud SQL open to internet, committed DB password / SA keys / API keys, unauthenticated core service), 7 high (stale news & widgets, models last trained Oct 2025 + training to cloud, training bugs, least-privilege DB roles, log.Fatalf crashes, per-asset hard-coding blocking stocks/commodities), plus normal/low cleanup, 2 decisions, 2 notes. Razorpay deprecation merged (PR #70). NEXT: Saravanan's step-by-step plan; quickest win is the one-line coin-page fix.
· saravanan@scrumclaw.ai
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: Grouped findings: broken for users, security, reliability, roadmap blockers, cost.
saravanan: create cointrends in tracker and push these items there
idleProject initialized: CoinTrends.ai
· saravanan@scrumclaw.ai