Claude Project Tracker

What's pending across every project

9 open · sorted by priority then due-date

kind all blockers todos decisions notes

priority all urgent high low

Filtered by tag training clear

todo high

Move model training to cloud and retrain — prediction models last trained 2025-10-30

Inference runs every 5 min (Scheduler prediction_calls → ml /run_prediction) on 120 DARNN models in gs://nlp_ml_files/prediction_files last written 2025-10-30. Training lives in CoinTrends_Python (local Windows GPU box, localhost Flask chain :17000 → :5000). Target: Cloud Run Job or Vertex AI custom job with GPU, parameterised by asset + minute offset, triggered by Scheduler, creds from Secret Manager, metadata write-back to cryptopredictionslicemaster.

ml training infra

CoinTrends_Python/master_create_update_latest.pyCoinTrends_Python/api_latest.pyCoinTrends_Python/ModelCreation_latest.pygs://nlp_ml_files/prediction_files/

CoinTrends.ai · cointrends · by saravanan@scrumclaw.ai

from the conversationsaravanan: what are the issues you identified in cointrends? claude: Predictions run on year-old models — the 120 model files were last trained 30 Oct 2025 and nothing retrains them. saravanan: create cointrends in tracker and push these items there

💬 1 comment

10d ago
10-16 19:09

todo high

Fix training pipeline bugs: cursor.commit() and undefined connection in NaN-loss branch

ModelCreation_latest.py calls cursor.commit() — psycopg2 cursors have no commit() (needs cnxn.commit()), and the second NaN branch uses cnxn which is only defined in the first branch. Result: a failed training run never gets marked activeflag='I'. Also .cuda() hard-coded; pyodbc import leftover.

bug ml training

CoinTrends_Python/ModelCreation_latest.py:315-340

CoinTrends.ai · cointrends · by saravanan@scrumclaw.ai

from the conversationsaravanan: what are the issues you identified in cointrends? claude: The training pipeline has real bugs: cursor.commit() doesn't exist in psycopg2 and the NaN-loss branch uses an undefined connection. saravanan: create cointrends in tracker and push these items there

10d ago
10-16 19:09

todo high

Training 1/5: Refactor CoinTrends_Python into one secret-free training package (merge orchestrator + trainer)

Take *_latest.py versions only. Fold master_create_update_latest.py decision logic (create vs incremental update, check_record_count) and api_latest.py/ModelCreation_latest.py training into one CLI entrypoint: train --asset BTCUSD --minute 15. Remove Flask/localhost HTTP, hardcoded DB host/password, SA key, Windows paths; read config from env. Fix cursor.commit() and NaN-branch bugs (#91). Make device selection automatic (cuda if available else cpu). Once secret-free, this can go into a git repo (unblocks decision #108).

ml training refactor

CoinTrends_Python/master_create_update_latest.pyCoinTrends_Python/api_latest.pyCoinTrends_Python/ModelCreation_latest.pyCoinTrends_Python/darnn.py

CoinTrends.ai · cointrends · by saravanan@scrumclaw.ai

from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based) claude: Broke #90 into subtasks; step 1 is a single secret-free training entrypoint.

💬 2 comments

10d ago
10-16 19:11

todo high

Training 2/5: Build GPU container image (CUDA PyTorch base) via Cloud Build → Artifact Registry

Dockerfile on a CUDA-enabled PyTorch base, pinned versions, psycopg2 + cloud-sql-python-connector + google-cloud-storage. Push to us-central1-docker.pkg.dev/prj-crypto538-prod-260529/crypto538-container-images/ml-training. Cloud Build config alongside.

ml training infra

CoinTrends.ai · cointrends · by saravanan@scrumclaw.ai

from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based) claude: Step 2: GPU container image built by Cloud Build into Artifact Registry.

10d ago
10-16 19:12

todo high

Training 3/5: Create Cloud Run Job with L4 GPU, dedicated service account, Secret Manager creds, task-index fan-out

gcloud run jobs create ml-training --gpu 1 --gpu-type nvidia-l4 (us-central1; confirm GPU quota/region availability first). Dedicated SA with only: Cloud SQL Client, read/write on gs://nlp_ml_files/prediction_files, secret accessor on the DB secret. --tasks = assets × 12 offsets (120 today), --parallelism capped to quota; CLOUD_RUN_TASK_INDEX → (asset, minute). Codify in crypto538-terraform. Note: #98 — only this SA should be able to write model files.

ml training infra

#98crypto538-terraform/environments/prod/main.tf

CoinTrends.ai · cointrends · by saravanan@scrumclaw.ai

from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based) claude: Step 3: Cloud Run Job with L4 GPU, least-privilege SA, task-index fan-out over asset × minute.

10d ago
10-16 19:12

todo high

Training 4/5: Run a full retrain and validate new models against predictionvsactual before switching inference

Write new models to a staging prefix first, backtest vs the Oct-2025 models (MAE/RMSE, directional accuracy) on recent data, then promote by updating cryptopredictionslicemaster paths. Keep the old files as rollback.

ml training testing

dbo.predictionvsactualdbo.cryptopredictionslicemaster

CoinTrends.ai · cointrends · by saravanan@scrumclaw.ai

from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based) claude: Step 4: full retrain to a staging prefix, backtest vs old models, then promote.

10d ago
10-16 19:12

decision

Model training moves to Cloud Run Jobs with GPU (NVIDIA L4) — replaces the local Windows GPU box

Chosen over Vertex AI custom jobs: same platform as the rest of the stack (Cloud Run, Artifact Registry, Cloud Build, Scheduler), pay-per-execution, task-index fan-out fits the asset × minute-offset model grid. Parent work item: #90.

ml training decision

#90

CoinTrends.ai · cointrends · by saravanan@scrumclaw.ai

from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based) claude: It's #90; recorded Cloud Run Jobs with GPU as the target and broke it into subtasks.

10d ago
10-16 19:11

todo

Training 5/5: Schedule recurring retrains with Cloud Scheduler and add failure alerting

Cloud Scheduler → Cloud Run Jobs run API (cadence TBD, e.g. weekly full / daily incremental using the existing updatetracking logic). Alert on failed executions and on stale models (modelfile older than N days). Retire the Windows box flow.

ml training ops

CoinTrends.ai · cointrends · by saravanan@scrumclaw.ai

from the conversationsaravanan: where is the model training to be moved to CloudRun (GPU based) claude: Step 5: Scheduler-driven retrains plus alerting; retire the local box.

10d ago
10-16 19:12

decision

Model training runs as GCP Batch jobs on GPU (Spot L4), not Cloud Run — supersedes #111

Saravanan 2026-10-20: train on GPU using Google Cloud Batch. Shape: same ml-training container (CUDA build) in Artifact Registry; Batch job with one task per asset (BATCH_TASK_INDEX), g2-standard-4 (1x L4) Spot VMs, installGpuDrivers, parallelism ≤ quota (L4: 8 on-demand / 8 preemptible in us-central1), dedicated service account, DB via Cloud SQL connector, password from Secret Manager, logs to Cloud Logging. Triggered by Cloud Scheduler calling the Batch jobs API. Batch API not yet enabled. Updates #113/#114/#116.

ml training decision batch

#90#111#113#114#116

CoinTrends.ai · cointrends · by saravanan@scrumclaw.ai

from the conversationsaravanan: Lets use GPU to train - I want the batch process in gcp - not the cloud run with GPU saravanan: tell me how to deploy that way

7d ago
10-20 01:51

What's pending across every project

Add an item