What's pending across every project
14 open · sorted by priority then due-date
Filtered by tag ml clear
todo
urgent
Live predictions are worse than "price stays the same" and ~50% on direction — fix before selling predictions
DB query 2026-10-19, last 30 days, slice 24 (2h ahead), all coins except TON: model error is ~2x the naive no-change baseline (BTC 0.555% vs 0.333%, ETH 0.868% vs 0.436%, DOGE 2.80% vs 0.71%), direction hit rate 49-51% (coin flip), and strong up-bias (DOGE predicts up 100% of the time, SOL/AVAX 89%). Also 2,489 predictions stored as NaN in 30 days. dbo.predictionvsactual is empty, so accuracy was never tracked. Retraining (#90) alone may not fix this — evaluate against the naive baseline in #115 and consider the model/target design.
#90#115dbo.<pair>_predictiondatadbo.predictionvsactual
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: I want you to access the database and query how the data looks like
claude: Compared 30 days of 2h-ahead predictions with actual prices: model error ~2x a no-change baseline, direction ~50%.
7d ago
10-19 14:07
10-19 14:07
todo
high
Move model training to cloud and retrain — prediction models last trained 2025-10-30
Inference runs every 5 min (Scheduler prediction_calls → ml /run_prediction) on 120 DARNN models in gs://nlp_ml_files/prediction_files last written 2025-10-30. Training lives in CoinTrends_Python (local Windows GPU box, localhost Flask chain :17000 → :5000). Target: Cloud Run Job or Vertex AI custom job with GPU, parameterised by asset + minute offset, triggered by Scheduler, creds from Secret Manager, metadata write-back to cryptopredictionslicemaster.
CoinTrends_Python/master_create_update_latest.pyCoinTrends_Python/api_latest.pyCoinTrends_Python/ModelCreation_latest.pygs://nlp_ml_files/prediction_files/
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: Predictions run on year-old models — the 120 model files were last trained 30 Oct 2025 and nothing retrains them.
saravanan: create cointrends in tracker and push these items there
💬 1 comment
10d ago
10-16 19:09
10-16 19:09
todo
high
Fix training pipeline bugs: cursor.commit() and undefined connection in NaN-loss branch
ModelCreation_latest.py calls cursor.commit() — psycopg2 cursors have no commit() (needs cnxn.commit()), and the second NaN branch uses cnxn which is only defined in the first branch. Result: a failed training run never gets marked activeflag='I'. Also .cuda() hard-coded; pyodbc import leftover.
CoinTrends_Python/ModelCreation_latest.py:315-340
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: The training pipeline has real bugs: cursor.commit() doesn't exist in psycopg2 and the NaN-loss branch uses an undefined connection.
saravanan: create cointrends in tracker and push these items there
10d ago
10-16 19:09
10-16 19:09
todo
high
Training 1/5: Refactor CoinTrends_Python into one secret-free training package (merge orchestrator + trainer)
Take *_latest.py versions only. Fold master_create_update_latest.py decision logic (create vs incremental update, check_record_count) and api_latest.py/ModelCreation_latest.py training into one CLI entrypoint: train --asset BTCUSD --minute 15. Remove Flask/localhost HTTP, hardcoded DB host/password, SA key, Windows paths; read config from env. Fix cursor.commit() and NaN-branch bugs (#91). Make device selection automatic (cuda if available else cpu). Once secret-free, this can go into a git repo (unblocks decision #108).
CoinTrends_Python/master_create_update_latest.pyCoinTrends_Python/api_latest.pyCoinTrends_Python/ModelCreation_latest.pyCoinTrends_Python/darnn.py
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based)
claude: Broke #90 into subtasks; step 1 is a single secret-free training entrypoint.
💬 2 comments
10d ago
10-16 19:11
10-16 19:11
todo
high
Training 2/5: Build GPU container image (CUDA PyTorch base) via Cloud Build → Artifact Registry
Dockerfile on a CUDA-enabled PyTorch base, pinned versions, psycopg2 + cloud-sql-python-connector + google-cloud-storage. Push to us-central1-docker.pkg.dev/prj-crypto538-prod-260529/crypto538-container-images/ml-training. Cloud Build config alongside.
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based)
claude: Step 2: GPU container image built by Cloud Build into Artifact Registry.
10d ago
10-16 19:12
10-16 19:12
todo
high
Training 3/5: Create Cloud Run Job with L4 GPU, dedicated service account, Secret Manager creds, task-index fan-out
gcloud run jobs create ml-training --gpu 1 --gpu-type nvidia-l4 (us-central1; confirm GPU quota/region availability first). Dedicated SA with only: Cloud SQL Client, read/write on gs://nlp_ml_files/prediction_files, secret accessor on the DB secret. --tasks = assets × 12 offsets (120 today), --parallelism capped to quota; CLOUD_RUN_TASK_INDEX → (asset, minute). Codify in crypto538-terraform. Note: #98 — only this SA should be able to write model files.
#98crypto538-terraform/environments/prod/main.tf
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based)
claude: Step 3: Cloud Run Job with L4 GPU, least-privilege SA, task-index fan-out over asset × minute.
10d ago
10-16 19:12
10-16 19:12
todo
high
Training 4/5: Run a full retrain and validate new models against predictionvsactual before switching inference
Write new models to a staging prefix first, backtest vs the Oct-2025 models (MAE/RMSE, directional accuracy) on recent data, then promote by updating cryptopredictionslicemaster paths. Keep the old files as rollback.
dbo.predictionvsactualdbo.cryptopredictionslicemaster
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based)
claude: Step 4: full retrain to a staging prefix, backtest vs old models, then promote.
10d ago
10-16 19:12
10-16 19:12
todo
high
PR "Coin Roster": make cryptomaster the only coin list; onboard any coin with dbo.onboard_coin()
8 steps, one PR per layer: DB onboarding function; DB collection; DB readers; core-service roster cache; ml-core-service; webapp; ml-training Batch task count; end-to-end proof with a new coin. Full instructions: cointrends/plans/PR-coin-roster.md (also in the project docs as claude/PR-coin-roster.md). Open questions: ship before or after Locked Door (#129); how DDL gets executed; which repo holds DB migrations.
plans/PR-coin-roster.md#129#118
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
💬 2 comments
6d ago
10-20 12:53
10-20 12:53
todo
Restrict write access to gs://nlp_ml_files — ML service unpickles models with weights_only=False
ml-core-service torch.load(..., weights_only=False) and np.load(allow_pickle=True) on files from GCS: anyone who can write to that bucket can execute code in the ML service. Limit bucket writers to the training job identity; consider safetensors / weights-only state_dict.
ml-core-service/main.py::predict
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: The ML service unpickles model files from GCS (weights_only=False); bucket write access = code execution.
saravanan: create cointrends in tracker and push these items there
10d ago
10-16 19:09
10-16 19:09
todo
Revisit the 12-models-per-asset design before scaling to new asset classes
One DARNN per 5-minute offset (desired_minute 0..55) → 12 models per coin, 120 today. Consider one model per asset with minute-of-hour as a feature, and a horizon suited to each asset class (e.g. daily for stocks/indexes).
CoinTrends_Python/trigger_updated.py
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: what are the issues you identified in cointrends?
claude: Each asset needs 12 models (one per 5-minute offset), which multiplies training cost and makes the model count grow fast.
saravanan: create cointrends in tracker and push these items there
10d ago
10-16 19:09
10-16 19:09
decision
Keep CoinTrends_Python (training code) outside git until secrets are stripped
The folder holds the DB password, public DB IP and an SA key in every script. Revisit after the secret-rotation items are done.
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: right now - let it sit outside of the repo - due to the secrets it contains.
claude: Recorded; it stays out of the repos for now.
10d ago
10-16 19:10
10-16 19:10
decision
Model training moves to Cloud Run Jobs with GPU (NVIDIA L4) — replaces the local Windows GPU box
Chosen over Vertex AI custom jobs: same platform as the rest of the stack (Cloud Run, Artifact Registry, Cloud Build, Scheduler), pay-per-execution, task-index fan-out fits the asset × minute-offset model grid. Parent work item: #90.
#90
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based)
claude: It's #90; recorded Cloud Run Jobs with GPU as the target and broke it into subtasks.
10d ago
10-16 19:11
10-16 19:11
todo
Training 5/5: Schedule recurring retrains with Cloud Scheduler and add failure alerting
Cloud Scheduler → Cloud Run Jobs run API (cadence TBD, e.g. weekly full / daily incremental using the existing updatetracking logic). Alert on failed executions and on stale models (modelfile older than N days). Retire the Windows box flow.
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based)
claude: Step 5: Scheduler-driven retrains plus alerting; retire the local box.
10d ago
10-16 19:12
10-16 19:12
decision
Model training runs as GCP Batch jobs on GPU (Spot L4), not Cloud Run — supersedes #111
Saravanan 2026-10-20: train on GPU using Google Cloud Batch. Shape: same ml-training container (CUDA build) in Artifact Registry; Batch job with one task per asset (BATCH_TASK_INDEX), g2-standard-4 (1x L4) Spot VMs, installGpuDrivers, parallelism ≤ quota (L4: 8 on-demand / 8 preemptible in us-central1), dedicated service account, DB via Cloud SQL connector, password from Secret Manager, logs to Cloud Logging. Triggered by Cloud Scheduler calling the Batch jobs API. Batch API not yet enabled. Updates #113/#114/#116.
#90#111#113#114#116
CoinTrends.ai
· cointrends
· by saravanan@scrumclaw.ai
from the conversationsaravanan: Lets use GPU to train - I want the batch process in gcp - not the cloud run with GPU
saravanan: tell me how to deploy that way
7d ago
10-20 01:51
10-20 01:51