Gemma-3-1B-IT Math SFT
google/gemma-3-1b-itλ₯Ό νκ΅μ΄ μν λ¬Έμ (GSM8K)μ λν΄ κ΅μ¬ μ¦λ₯ SFTν λͺ¨λΈ.
μ±λ₯
| Benchmark | Score |
|---|---|
| HRM8K eval GSM8K (264λ¬Έμ , Korean) | ~44.9% (3ν νκ· ) |
| HRM8K eval MATH (577λ¬Έμ , Korean) | ~17% |
νκ°: temperature=0, vLLM μλΉ, max_tokens=2048
λ°μ΄ν° μμ± νμ΄νλΌμΈ
μλ³Έ λ°μ΄ν°
- GSM8K train set: μμ΄ μ΄λ± μν 7,473λ¬Έμ (openai/gsm8k)
- νκ° λ°μ΄ν°: HRM8K eval set 841λ¬Έμ (GSM8K 264 + MATH 577, νκ΅μ΄)
κ΅μ¬ λͺ¨λΈ
- λͺ¨λΈ: (AWQ 4bit)
- μλΉ: vLLM 0.11.0 (ꡬλ²μ νμ, AWQ νΈν),
- μ€μ: μ΄ κ΅μ¬ λͺ¨λΈμ HRM8K νμ΅ λ°μ΄ν°λ₯Ό μμ±ν λ°λ‘ κ·Έ λͺ¨λΈ. λ€λ₯Έ κ΅μ¬ λͺ¨λΈ(Qwen3.5-9B/35B λ±)μ μ€νμΌ λΆμΌμΉλ‘ -10%p μ΄μ μ±λ₯ νλ½.
2λ¨κ³ λ°μ΄ν° μμ±
μ΅μ’ νμ΅ λ°μ΄ν° νμ
- μ΄ 26,254κ° (train 95% / eval 5% split)
- μμ€ν ν둬ννΈ:
νμ΅ μ€μ
μ¬ν λ°©λ²
INFO 03-19 14:51:58 [init.py:216] Automatically detected platform cuda. [1;36m(APIServer pid=3426235)[0;0m INFO 03-19 14:52:04 [api_server.py:1839] vLLM API server version 0.11.0 [1;36m(APIServer pid=3426235)[0;0m INFO 03-19 14:52:04 [utils.py:233] non-default args: {'model_tag': 'cpatonn/Qwen3-30B-A3B-Instruct-2507-AWQ-4bit', 'port': 8001, 'model': 'cpatonn/Qwen3-30B-A3B-Instruct-2507-AWQ-4bit', 'max_model_len': 4096, 'gpu_memory_utilization': 0.8} [1;36m(APIServer pid=3426235)[0;0m INFO 03-19 14:52:06 [model.py:547] Resolved architecture: Qwen3MoeForCausalLM [1;36m(APIServer pid=3426235)[0;0m INFO 03-19 14:52:06 [model.py:1510] Using max model len 4096 [1;36m(APIServer pid=3426235)[0;0m INFO 03-19 14:52:07 [scheduler.py:205] Chunked prefill is enabled with max_num_batched_tokens=8192. INFO 03-19 14:52:19 [init.py:216] Automatically detected platform cuda. [1;36m(EngineCore_DP0 pid=3426796)[0;0m INFO 03-19 14:52:25 [core.py:644] Waiting for init message from front-end. [1;36m(EngineCore_DP0 pid=3426796)[0;0m INFO 03-19 14:52:25 [core.py:77] Initializing a V1 LLM engine (v0.11.0) with config: model='cpatonn/Qwen3-30B-A3B-Instruct-2507-AWQ-4bit', speculative_config=None, tokenizer='cpatonn/Qwen3-30B-A3B-Instruct-2507-AWQ-4bit', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=cpatonn/Qwen3-30B-A3B-Instruct-2507-AWQ-4bit, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention","vllm.sparse_attn_indexer"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null} [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [1;36m(EngineCore_DP0 pid=3426796)[0;0m INFO 03-19 14:52:26 [parallel_state.py:1208] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0 [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] EngineCore failed to start. [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] Traceback (most recent call last): [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 699, in run_engine_core [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] engine_core = EngineCoreProc(*args, **kwargs) [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 498, in init [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] super().init(vllm_config, executor_class, log_stats, [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 83, in init [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] self.model_executor = executor_class(vllm_config) [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in init [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] self._init_executor() [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 54, in _init_executor [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] self.collective_rpc("init_device") [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] return [run_method(self.driver_worker, method, args, kwargs)] [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/utils/init.py", line 3122, in run_method [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] return func(*args, **kwargs) [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^ [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/worker/worker_base.py", line 259, in init_device [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] self.worker.init_device() # type: ignore [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 187, in init_device [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] raise ValueError( [1;36m(EngineCore_DP0 pid=3426796)[0;0m ERROR 03-19 14:52:26 [core.py:708] ValueError: Free memory on device (7.29/93.1 GiB) on startup is less than desired GPU memory utilization (0.8, 74.48 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes. INFO 03-19 14:52:34 [init.py:216] Automatically detected platform cuda. [1;36m(APIServer pid=3427292)[0;0m INFO 03-19 14:52:40 [api_server.py:1839] vLLM API server version 0.11.0 [1;36m(APIServer pid=3427292)[0;0m INFO 03-19 14:52:40 [utils.py:233] non-default args: {'model_tag': './outputs/models/c17d-gemma-3-1b-it-Math', 'model': './outputs/models/c17d-gemma-3-1b-it-Math', 'dtype': 'bfloat16', 'max_model_len': 4096, 'gpu_memory_utilization': 0.85} [1;36m(APIServer pid=3427292)[0;0m INFO 03-19 14:52:51 [model.py:547] Resolved architecture: Gemma3ForCausalLM [1;36m(APIServer pid=3427292)[0;0m INFO 03-19 14:52:51 [model.py:1510] Using max model len 4096 [1;36m(APIServer pid=3427292)[0;0m INFO 03-19 14:52:51 [scheduler.py:205] Chunked prefill is enabled with max_num_batched_tokens=8192. INFO 03-19 14:52:58 [init.py:216] Automatically detected platform cuda. [1;36m(EngineCore_DP0 pid=3428117)[0;0m INFO 03-19 14:53:03 [core.py:644] Waiting for init message from front-end. [1;36m(EngineCore_DP0 pid=3428117)[0;0m INFO 03-19 14:53:03 [core.py:77] Initializing a V1 LLM engine (v0.11.0) with config: model='./outputs/models/c17d-gemma-3-1b-it-Math', speculative_config=None, tokenizer='./outputs/models/c17d-gemma-3-1b-it-Math', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=./outputs/models/c17d-gemma-3-1b-it-Math, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention","vllm.sparse_attn_indexer"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null} [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [1;36m(EngineCore_DP0 pid=3428117)[0;0m INFO 03-19 14:53:05 [parallel_state.py:1208] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0 [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] EngineCore failed to start. [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] Traceback (most recent call last): [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 699, in run_engine_core [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] engine_core = EngineCoreProc(*args, **kwargs) [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 498, in init [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] super().init(vllm_config, executor_class, log_stats, [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 83, in init [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] self.model_executor = executor_class(vllm_config) [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/executor/executor_base.py", line 54, in init [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] self._init_executor() [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 54, in _init_executor [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] self.collective_rpc("init_device") [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] return [run_method(self.driver_worker, method, args, kwargs)] [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/utils/init.py", line 3122, in run_method [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] return func(*args, **kwargs) [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^ [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/worker/worker_base.py", line 259, in init_device [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] self.worker.init_device() # type: ignore [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] ^^^^^^^^^^^^^^^^^^^^^^^^^ [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] File "/tmp/.venv/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 187, in init_device [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] raise ValueError( [1;36m(EngineCore_DP0 pid=3428117)[0;0m ERROR 03-19 14:53:05 [core.py:708] ValueError: Free memory on device (7.29/93.1 GiB) on startup is less than desired GPU memory utilization (0.85, 79.13 GiB). Decrease GPU memory utilization or reduce GPU memory used by other processes.
νμΌ
- : SFT νμ΅ μ€ν¬λ¦½νΈ
- : HRM8K νκ° μ€ν¬λ¦½νΈ (vLLM OpenAI API νΈν μλ² νμ)
- Downloads last month
- 99