Mercity/figure-skating-ctm

Figure Skating CTM (Continuous Thought Machine) Action Classifiers

Continuous Thought Machine architecture for classifying figure-skating actions (jumps, spins, sequences) from pose-skeleton time series: a Conv1D backbone feeding a CTM "thinking loop" (recurrent neuron model + synchronization-based attention over the backbone features), run for a fixed or adaptively-halted number of steps before classification.

This repo holds four checkpoints — two thinking-step configurations, each trained on both label granularities:

file	variant	thinking steps	label space	classes
model_ctm-10_fine.pt	CTM-10 (vanilla, fixed)	exactly 10, uniform weighting	fine-grained	28
model_ctm-10_coarse.pt	CTM-10 (vanilla, fixed)	exactly 10, uniform weighting	coarse	11
model_ctm-10+act_fine.pt	CTM-10+ACT (adaptive halting)	up to 10, learned per-sample halting	fine-grained	28
model_ctm-10+act_coarse.pt	CTM-10+ACT (adaptive halting)	up to 10, learned per-sample halting	coarse	11

Not transformers-compatible — plain PyTorch nn.Module + state_dict checkpoints. Model source code (SkatingCTM in the project's model_ctm.py, which depends on ctm_components.py, model.py, and model_gcn.py) is not included in this repo — you'll need the project source to reconstruct the class before loading the state dict.

How to load

from huggingface_hub import hf_hub_download
import torch

repo_id = "Mercity/figure-skating-ctm"
ckpt_path = hf_hub_download(repo_id, "model_ctm-10_fine.pt")
ckpt = torch.load(ckpt_path, map_location="cpu", weights_only=False)

# reconstruct with the project's SkatingCTM (model_ctm.py):
# model = SkatingCTM(
#     ckpt["in_features"], ckpt["num_classes"],
#     max_iterations=ckpt["max_iterations"], memory_length=ckpt["memory_length"],
#     use_act=ckpt["use_act"], backbone=ckpt["backbone"],
#     node_features=ckpt["node_features"], gcn_layers=ckpt["gcn_layers"],
# )
# model.load_state_dict(ckpt["state_dict"])
# model.eval()

# ckpt also carries the standardization stats used at train time and the taxonomy:
mu, sd = ckpt["feature_mean"], ckpt["feature_std"]     # normalize inputs with (x - mu) / sd
taxonomy = ckpt["taxonomy"]                            # {0: "3Toeloop", 1: "3Loop", ...}

Swap the filename to load any of the other three variants.

Config

Shared across all four checkpoints:

Backbone	conv (Conv1D stem(128) → ConvBlock(192,k3) → ConvBlock(256,k3) → ConvBlock(384,k5), d_backbone=384)
CTM core	d_model=256, memory_length=50, n_synch=64, num_heads=8, dropout=0.1
Optimizer	AdamW, lr=3e-4, weight_decay=1e-4
Batch size	64
Grad clip	1.0
Class weight cap	5.0
Early stop	patience 20 on val macro-F1, 100-epoch cap
Seed	42
Params	2.18M (fine) / 2.15M (coarse)

Where CTM-10 vs CTM-10+ACT differ:

	CTM-10	CTM-10+ACT
Thinking steps	fixed 10 every sample, uniform loss/prediction weight (1/10 each)	up to 10, learned per-sample halting (use_act=True); halt bias init −3.0
ponder_lambda	0.0 (no ACT term)	0.01
act_epsilon	n/a	0.01

⚠️ Note: in both fine and coarse CTM-10+ACT runs, test-time mean thinking steps measured 10.0 — every sample used the full step budget; no sample ever halted early. With a max of 10 iterations and a halt bias initialized to favor ~20 steps, ACT never got the chance to demonstrate adaptive depth in these checkpoints — treat CTM-10+ACT here as "CTM-10 + a ponder-cost regularizer on a fixed-depth computation," not genuinely adaptive-depth inference.

Data

Trained on skeleton-extracted figure-skating clips (YOLO11n-pose, COCO-17 joints, imgsz=384) from an internal Mercity dataset. Stratified train/val/test split, seed 42. Not included in this repo (private internal dataset).

Results (test split)

CTM-10 — Fine-grained (28-class)

Early-stopped epoch 99 (best val macro-F1 0.638) · test mean thinking steps: 10.0

Overall: accuracy 0.8799 · precision (macro/weighted) 0.5055 / 0.8756 · recall 0.4962 / 0.8799 · F1 0.4948 / 0.8749

class	precision	recall	f1	support
3Toeloop	0.692	0.711	0.701	38
3Loop	0.739	0.810	0.773	21
2Axel	0.771	0.860	0.813	43
CamelSpin	0.976	0.968	0.972	125
SitSpin	0.963	0.992	0.978	132
UprightSpin	0.953	0.944	0.949	108
2Salchow	0.800	0.800	0.800	5
2Toeloop	0.833	0.781	0.806	32
3Salchow	0.609	0.667	0.636	21
3Axel	0.714	0.667	0.690	15
3Flip	0.727	0.444	0.552	36
3Lutz	0.612	0.732	0.667	41
NoBasic	0.962	0.806	0.877	31
2Lutz	0.500	0.714	0.588	7
4Salchow	0.000	0.000	0.000	3
4Flip	0.000	0.000	0.000	1
4Toeloop	0.000	0.000	0.000	6
4Lutz	0.000	0.000	0.000	1
4Loop	0.000	0.000	0.000	1
2Flip	0.667	0.250	0.364	8
2Loop	0.667	0.750	0.706	8
1Axel	0.000	0.000	0.000	3
1Loop	1.000	1.000	1.000	1
1Salchow	0.000	0.000	0.000	1
1Toeloop	0.000	0.000	0.000	1
1Flip	0.000	0.000	0.000	2
1Lutz	0.000	0.000	0.000	1
Sequence	0.968	0.997	0.982	307
macro avg	0.505	0.496	0.495	999
weighted avg	0.876	0.880	0.875	999

CTM-10 — Coarse (11-class)

Early-stopped epoch 96 (best val macro-F1 0.849) · test mean thinking steps: 10.0

Overall: accuracy 0.8929 · precision (macro/weighted) 0.8188 / 0.8977 · recall 0.8276 / 0.8929 · F1 0.8161 / 0.8925

class	precision	recall	f1	support
Axel	0.966	0.934	0.950	61
Toeloop	0.663	0.714	0.688	77
Salchow	0.773	0.567	0.654	30
Loop	0.630	0.935	0.753	31
Flip	0.683	0.596	0.636	47
Lutz	0.690	0.580	0.630	50
CamelSpin	0.992	0.960	0.976	125
SitSpin	0.975	0.902	0.937	132
UprightSpin	0.963	0.954	0.958	108
Sequence	0.974	0.993	0.984	307
NoBasic	0.698	0.968	0.811	31
macro avg	0.819	0.828	0.816	999
weighted avg	0.898	0.893	0.892	999

CTM-10+ACT — Fine-grained (28-class)

Early-stopped epoch 66 (best val macro-F1 0.550) · test mean thinking steps: 10.0

Overall: accuracy 0.8559 · precision (macro/weighted) 0.4840 / 0.8645 · recall 0.5175 / 0.8559 · F1 0.4775 / 0.8573

class	precision	recall	f1	support
3Toeloop	0.622	0.737	0.675	38
3Loop	0.650	0.619	0.634	21
2Axel	0.829	0.791	0.810	43
CamelSpin	0.976	0.968	0.972	125
SitSpin	0.962	0.955	0.958	132
UprightSpin	0.950	0.889	0.919	108
2Salchow	1.000	0.400	0.571	5
2Toeloop	0.792	0.594	0.679	32
3Salchow	0.565	0.619	0.591	21
3Axel	0.500	0.733	0.595	15
3Flip	0.564	0.611	0.587	36
3Lutz	0.643	0.659	0.651	41
NoBasic	0.818	0.871	0.844	31
2Lutz	0.667	0.571	0.615	7
4Salchow	0.000	0.000	0.000	3
4Flip	0.000	0.000	0.000	1
4Toeloop	0.250	0.500	0.333	6
4Lutz	0.000	0.000	0.000	1
4Loop	0.000	0.000	0.000	1
2Flip	0.800	0.500	0.615	8
2Loop	0.400	0.500	0.444	8
1Axel	0.000	0.000	0.000	3
1Loop	0.333	1.000	0.500	1
1Salchow	0.000	0.000	0.000	1
1Toeloop	0.250	1.000	0.400	1
1Flip	0.000	0.000	0.000	2
1Lutz	0.000	0.000	0.000	1
Sequence	0.980	0.974	0.977	307
macro avg	0.484	0.518	0.477	999
weighted avg	0.864	0.856	0.857	999

CTM-10+ACT — Coarse (11-class)

Early-stopped epoch 57 (best val macro-F1 0.823) · test mean thinking steps: 10.0

Overall: accuracy 0.8979 · precision (macro/weighted) 0.8201 / 0.9012 · recall 0.8362 / 0.8979 · F1 0.8262 / 0.8986

class	precision	recall	f1	support
Axel	0.917	0.902	0.909	61
Toeloop	0.720	0.766	0.742	77
Salchow	0.645	0.667	0.656	30
Loop	0.711	0.871	0.783	31
Flip	0.682	0.638	0.659	47
Lutz	0.717	0.660	0.688	50
CamelSpin	0.976	0.968	0.972	125
SitSpin	1.000	0.917	0.957	132
UprightSpin	0.934	0.917	0.925	108
Sequence	0.984	0.990	0.987	307
NoBasic	0.737	0.903	0.812	31
macro avg	0.820	0.836	0.826	999
weighted avg	0.901	0.898	0.899	999

Notable failure modes (from confusion matrix analysis, coarse label space)

• Flip ↔ Lutz is the dominant confusion in both variants — edge jumps, genuinely hard to distinguish from skeleton motion alone (mirrors real judging difficulty).
• CTM-10 (fixed) shows the field's worst Salchow recall (56.7%) and Lutz recall (58.0%), plus unusually strong SitSpin→NoBasic leakage (9/132) not seen in other architectures.
• CTM-10+ACT improves slightly over CTM-10 on most coarse classes but never surpasses CNN-BiLSTM or GCN on macro-F1.
• Fine-grained macro-F1 in both variants is dragged down almost entirely by 1-2-sample tail classes (4Flip, 4Lutz, 4Loop, 1Salchow, 1Toeloop, 1Flip, 1Lutz) — a data scarcity issue, not an architecture weakness.

License

Internal/proprietary — not licensed for external use. Contact the Mercity team for access terms.