nc-GPT

中文说明

nc-GPT is a local GPT-style language model training project built around a single consumer GPU, currently an NVIDIA RTX 4090 with 24 GB VRAM. The project starts from reproducible data preparation, then tokenizer training, token packing, and now includes a working GPT pretraining stack.

The immediate goal is not to clone a chat assistant. It is to build a clean, inspectable, bilingual Chinese/English base model training pipeline that can later support continued pretraining and style tuning on personal fiction data.

Release Boundary

This Git repository is a code-only open-source release. Source code and documentation are licensed under Apache-2.0. Training corpora, packed token arrays, trained tokenizer artifacts, checkpoints, run logs, and model weights are excluded from Git.

Public distribution of ncgpt-1p2b-base weights is not planned. The project owner chose a code-only release because the OpenCSG and Ultra-FineWeb license chains remain unresolved. See DATA_LICENSE_AUDIT.md for the decision record and source revisions. The code license does not grant any rights to datasets, trained tokenizer artifacts, or model weights.

Status

Stage	Status	Output
Base corpus collection	Done	data/corpus_full/*.jsonl.zst
64K bilingual tokenizer	Done	data/tokenizer/nc_bpe_64k.model
Token packing	Done	data/tokens_full/{train,val,test}.bin
GPT training code	Done	ncgpt/, scripts/train.py, scripts/sample.py
1.2B CUDA probe	Done	configs/train_1p2b_probe.yaml
Model pretraining	Done	checkpoints/ncgpt_1p2b_4090/latest.pt
Fixed final checkpoint evaluation	Done	reports/pretrain_final_eval.json
Optimizer-free release and generation acceptance	Done	releases/ncgpt-1p2b-base/
Code-only open-source preparation	Done	LICENSE, NOTICE, license audit
Public model-weight release	Not planned	Code-only release decision
Private CPT data preparation and chat audit	Done	scripts/prepare_continued_data.py, scripts/audit_chat_data.py
Weights-only CPT initialization and 1.2B CUDA probe	Done	configs/train_1p2b_cpt_probe.yaml
Continued pretraining on private fiction	Waiting for data	No private corpus found in the workspace yet
Chat SFT	Not started	Audit first; SFT normalization follows

Project Goals

• Train a GPT-2-style decoder-only language model locally on RTX 4090-class hardware.
• Use modern Chinese and English as the main pretraining languages.
• Keep data provenance, split metadata, and token statistics inspectable.
• Separate base pretraining data from later domain/style data.
• Build the project in stages so each completed stage can be reproduced and verified independently.

Repository Layout

configs/
  base_corpus.yaml          # Hugging Face source list and corpus limits
  tokenizer.yaml            # tokenizer sampling and token packing config
  private_cpt_tokens.yaml   # private continued-pretraining token packing config
  train_1p2b_4090.yaml      # long-run 1.235B GPT training config for RTX 4090
  train_1p2b_cpt.yaml       # private continued-pretraining config
  train_1p2b_cpt_probe.yaml # one-step weights-only initialization probe
  train_1p2b_probe.yaml     # one-step 1.235B memory probe, no checkpoint save
  train_smoke.yaml          # tiny end-to-end training test
ncgpt/
  chat_audit.py             # content-free chat schema and possible-PII audit
  data.py                   # numpy.memmap token dataloader
  model.py                  # GPT decoder-only model
  private_data.py           # private text normalization, chunking, and splitting
  tokenizer.py              # SentencePiece wrapper
  utils.py                  # config, optimizer, checkpoint helpers
scripts/
  audit_chat_data.py        # audit private chat JSONL without copying message text
  prepare_corpus.py         # stream, clean, deduplicate, split, and compress text
  prepare_continued_data.py # prepare private fiction/domain text for CPT
  inspect_dataset.py        # inspect Hugging Face dataset fields/configs
  train_tokenizer.py        # sample text and train SentencePiece BPE tokenizer
  pack_tokens.py            # encode JSONL.zst corpus to uint16 token arrays
  inspect_model.py          # inspect parameter count and memory estimate
  evaluate_checkpoints.py   # compare checkpoints on fixed windows and test the winner
  export_release.py         # export an optimizer-free checkpoint and manifest
  validate_release.py       # reload a release and run bilingual generation acceptance
  train.py                  # mmap training loop with eval/checkpoint/resume
  sample.py                 # generate text from a checkpoint
data/
  corpus_full/              # generated compressed JSONL corpus
  private/                  # generated private corpora, reports, and token files
  tokenizer/                # generated tokenizer and tokenizer sample
  tokens_full/              # generated mmap-ready token files
checkpoints/                # generated training checkpoints, ignored by git
logs/                       # long-running job logs
reports/                    # reproducible model evaluation reports
releases/                   # local release packages; large .pt weights stay out of git

Generated data, checkpoints, and caches are intentionally ignored by git.

Data Sources

The current base corpus uses these public Hugging Face datasets:

• opencsg/Fineweb-Edu-Chinese-V2.1
• openbmb/Ultra-FineWeb
• HuggingFaceFW/fineweb-edu
• HuggingFaceFW/fineweb
• wikimedia/wikipedia Chinese and English dumps

The pipeline writes provenance fields into every JSONL record, including source name, dataset name, pinned dataset revision, config, language, license note, document hash, and text.

Important: this repository contains scripts and generated local metadata only. It does not grant redistribution rights for upstream datasets. Check each dataset's license and upstream terms before publishing data, trained weights, or commercial outputs.

The exact Hugging Face revisions used by the completed corpus are now pinned in configs/base_corpus.yaml. The detailed source-by-source release review is in DATA_LICENSE_AUDIT.md.

Completed Corpus Scale

Full corpus manifest: data/corpus_full/manifest.full.json

Source	Documents	Uncompressed JSONL bytes
fineweb_edu_chinese_v21	12,264,676	65.00 GB
ultra_fineweb_zh	5,206,069	25.00 GB
fineweb_edu_en	6,863,907	35.00 GB
fineweb_en	5,733,380	20.00 GB
wikipedia_zh	496,051	2.37 GB
wikipedia_en	3,262,465	12.00 GB
Total	33,826,548	159.37 GB

The compressed .jsonl.zst corpus is about 57.97 GB.

Tokenizer

Tokenizer manifest: data/tokenizer/nc_bpe_64k.meta.json

Field	Value
Type	SentencePiece BPE
Vocabulary size	65,536
Special document separator	<|endoftext|>
eot_id	1
Byte fallback	Enabled
Token dtype	uint16

The tokenizer was trained from a 1.80 GB balanced sample of the full corpus. Round-trip checks against real Chinese corpus text passed.

Packed Token Scale

Token manifest: data/tokens_full/manifest.json

In practical terms, the base pretraining dataset is a 34B-token-scale corpus: about 34.33B training tokens and 34.50B tokens including validation and test.

Split	Documents	Tokens	File
train	33,657,468	34,326,810,540	68.65 GB
val	135,280	139,033,818	278 MB
test	33,800	34,615,191	69 MB
Total	33,826,548	34,500,459,549	69.00 GB

The packed token files are contiguous uint16 arrays and are read directly with numpy.memmap during training.

Setup

cd path\to\nc_gpt
.\.venv\Scripts\python -m pip install torch --index-url https://download.pytorch.org/whl/cu128
.\.venv\Scripts\python -m pip install -r requirements.txt

The current local environment has been verified with PyTorch 2.11.0+cu128, CUDA available, and NVIDIA GeForce RTX 4090 detected.

The scripts set Hugging Face cache to .hf_cache/ inside this workspace so large downloads stay on the workspace drive.

Reproduce Data Preparation

Smoke corpus:

.\.venv\Scripts\python scripts\prepare_corpus.py --profile smoke --keep-going

Full corpus:

.\.venv\Scripts\python scripts\prepare_corpus.py --profile full --keep-going --output-dir data\corpus_full

Train tokenizer:

.\.venv\Scripts\python scripts\train_tokenizer.py --config configs\tokenizer.yaml

Pack tokens:

.\.venv\Scripts\python scripts\pack_tokens.py --config configs\tokenizer.yaml --splits val test train

Training Stack

The training stack now includes:

• direct mmap batching from data/tokens_full/*.bin
• GPT decoder-only model with tied token embedding / LM head
• scaled dot-product causal attention
• bf16 mixed precision on CUDA
• optional bf16 model parameters for 24 GB VRAM feasibility
• gradient accumulation
• gradient clipping
• cosine LR schedule with warmup
• validation loss evaluation
• best.pt and latest.pt checkpoint saves
• resume from checkpoint
• weights-only initialization with model-structure validation
• automatic step calculation from private token count and requested epochs
• tokenizer-backed sample generation

The 1.2B target config is:

Field	Value
Parameters	1,234,822,656
Layers	40
Hidden size	1,536
Attention heads	24
Context length	1,024
Vocabulary	65,536
Micro batch	1
Gradient accumulation	64
Tokens per optimizer step	65,536
Parameter dtype	bf16
Activation checkpointing	Enabled

Inspect the config:

.\.venv\Scripts\python scripts\inspect_model.py --config configs\train_1p2b_4090.yaml

The current estimate for the 1.2B config is about 9.2 GiB of persistent parameter/gradient/AdamW-state memory, excluding activations, CUDA workspace, and allocator fragmentation.

Run the tiny end-to-end smoke test:

.\.venv\Scripts\python scripts\train.py --config configs\train_smoke.yaml
.\.venv\Scripts\python scripts\sample.py --checkpoint checkpoints\smoke\latest.pt --prompt "Hello" --max-new-tokens 20

Run the 1.2B one-step memory probe:

.\.venv\Scripts\python scripts\train.py --config configs\train_1p2b_probe.yaml

The probe has successfully run one real 1024-token microbatch with forward, backward, and AdamW step on the local RTX 4090 without saving a large checkpoint.

Start the 1.2B long run:

.\.venv\Scripts\python scripts\train.py --config configs\train_1p2b_4090.yaml

Resume explicitly:

.\.venv\Scripts\python scripts\train.py --config configs\train_1p2b_4090.yaml --resume checkpoints\ncgpt_1p2b_4090\latest.pt

Checkpoint note: a full 1.2B latest.pt or best.pt with optimizer state can be several GB, so keep enough free disk space before starting the long run.

Private Continued Pretraining

Personal fiction and other long-form domain text belong in continued pretraining (CPT), while chat records are kept separate for a later SFT pipeline. No private source files were present when this stage was implemented, so real private-data training has not started. Everything under data/ and checkpoints/ remains ignored by Git.

The fiction preparation command accepts UTF-8 .txt, .md, and .jsonl files. JSONL rows use the first available field among text, content, body, document, and article by default; custom dot-separated field names can be passed with --text-fields. The pipeline normalizes text, chunks it, performs exact SHA-256 deduplication, creates deterministic train/validation/test splits, writes compressed JSONL, and records a private manifest.

Place source files under data\private\raw\fiction, then run:

.\.venv\Scripts\python scripts\prepare_continued_data.py --input data\private\raw\fiction
.\.venv\Scripts\python scripts\pack_tokens.py --config configs\private_cpt_tokens.yaml --tokenizer data\tokenizer\nc_bpe_64k.model
.\.venv\Scripts\python scripts\train.py --config configs\train_1p2b_cpt.yaml

configs/train_1p2b_cpt.yaml starts from the local optimizer-free base release, resets AdamW, uses a conservative peak learning rate of 1e-5, and requests two token-equivalent passes over the private training set. Batches use random mmap windows, so this is a token budget rather than a guarantee that every document is visited exactly twice. Once packing has produced a manifest, the trainer converts max_epochs into an exact max_iters value and saves the resolved schedule. Existing latest.pt in the CPT output directory takes precedence for automatic resume.

The weights-only initialization path was accepted on the local RTX 4090 on 2026-08-12: all 1,234,822,656 parameters loaded strictly from base iteration 500000, followed by one 1024-token forward/backward/AdamW step at loss 3.0132. The measured training-loop time was 0.525 s (14.5 s wall time including process startup and weight loading). Reproduce the no-checkpoint probe with:

.\.venv\Scripts\python scripts\train.py --config configs\train_1p2b_cpt_probe.yaml

Before using private chat logs, run the content-free audit:

.\.venv\Scripts\python scripts\audit_chat_data.py --input path\to\chat.jsonl

The report contains schema counts, role counts, hashes, and heuristic possible-PII counts, but no message text or raw conversation identifiers. This is an audit only, not an SFT conversion. Consent, ownership, third-party privacy review, conversation normalization, train/validation splitting, and assistant-only loss masking are still required before chat fine-tuning.

Pretraining Result

The first 1.2B pretraining run completed normally at 2026-08-12T18:59:47+0800, after 500000 optimizer steps and 32,768,000,000 training tokens.

Field	Value
Config	configs/train_1p2b_4090.yaml
Current PID	None (completed; final PID 2444)
Original start	2026-07-01T11:25:34+0800
Completed at	2026-08-12T18:59:47+0800
Final checkpoint	checkpoints/ncgpt_1p2b_4090/latest.pt at iter 500000
Final segment stdout log	logs/train_1p2b_20260811_200300.out.log
Final segment stderr log	logs/train_1p2b_20260811_200300.err.log (empty)
Total active training time	3,554,741.483 s (41d 03:25:41.483)
Calendar elapsed time	42d 07:34:13
Final validation loss	3.0839298 (50 random validation batches)
Best validation loss observed during training	2.7991593 (iter 439000, 50 random batches)
Base checkpoint selected by fixed evaluation	latest.pt (iter 500000)
Runtime state	checkpoints/ncgpt_1p2b_4090/training_state.json

Fixed Checkpoint Evaluation

After training, best.pt and latest.pt were reevaluated on the same 1000 fixed validation windows. Each window contains 1024 tokens, for 1,024,000 validation tokens per checkpoint. Checkpoint selection used validation data only; the winner was then evaluated once on an independent 1,024,000-token test sample. Fixed positions, per-window losses, and full statistics are stored in reports/pretrain_final_eval.json.

Checkpoint	Iter	Fixed validation loss	Perplexity	95% CI
best.pt	439000	3.038289	20.869	[2.997992, 3.078585]
latest.pt	500000	3.037422	20.851	[2.997117, 3.077728]

In the paired per-window comparison, the mean best - latest loss difference is +0.000866, with a 95% CI of [+0.000518, +0.001215]. The effect is small, but latest.pt has a consistent advantage on these fixed validation windows and is the selected base checkpoint. Its independent test loss is 3.044250, with perplexity 20.994 and a 95% CI of [3.002792, 3.085708].

Reproduce the evaluation:

.\.venv\Scripts\python scripts\evaluate_checkpoints.py `
  --config configs\train_1p2b_4090.yaml `
  --checkpoints checkpoints\ncgpt_1p2b_4090\best.pt checkpoints\ncgpt_1p2b_4090\latest.pt `
  --validation-windows 1000 --test-windows 1000 --batch-size 1 `
  --seed 20260812 --output reports\pretrain_final_eval.json

The training-time value 2.7991593 was the minimum observed across repeated 50-window random evaluations. It is not directly comparable to a result from a different random sample. The fixed evaluation removes window differences between checkpoints and uses validation selection followed by held-out test confirmation.

Release Weights And Generation Acceptance

The final latest.pt was exported to releases/ncgpt-1p2b-base/model.pt. The release contains only the model state, configuration, and training metadata; AdamW optimizer state is absent. File size dropped from 7,409,258,467 bytes to 2,469,746,770 bytes (2.300 GiB), a 66.67% reduction.

Item	Result
Release SHA-256	4db617c359ad345634eb5e910b4cc58397d2e5a19741041471b54e76450bcfb0
Optimizer	Not included
Strict parameter load	Passed, 1,234,822,656 parameters
Tokenizer/hash verification	Passed
Chinese generation	Packaging acceptance passed; strong phrase repetition appeared later
English generation	Packaging acceptance passed; sample was broadly coherent
Full report	releases/ncgpt-1p2b-base/validation.json

Export and rerun acceptance:

.\.venv\Scripts\python scripts\export_release.py `
  --checkpoint checkpoints\ncgpt_1p2b_4090\latest.pt `
  --tokenizer data\tokenizer\nc_bpe_64k.model `
  --tokenizer-vocab data\tokenizer\nc_bpe_64k.vocab `
  --output-dir releases\ncgpt-1p2b-base

.\.venv\Scripts\python scripts\validate_release.py `
  --release-dir releases\ncgpt-1p2b-base --device cuda `
  --max-new-tokens 160 --temperature 0.8 --top-k 200

See releases/ncgpt-1p2b-base/README.md for package contents, inference usage, evaluation results, and limitations. Acceptance proves that the package is intact and independently loadable; it does not imply instruction tuning or production writing quality.

Segment	PID	Started	Last state update	Last iter	Active time	stdout log
1	43004	2026-07-01T11:25:34+0800	2026-07-17T13:16:15+0800	203850	1,389,040.491 s	logs/train_1p2b_20260701_112522.out.log
2	8704	2026-07-17T16:57:45+0800	2026-07-23T20:28:37+0800	275550	531,052.365 s	logs/train_1p2b_20260717_165730.out.log
3	6260	2026-07-23T21:22:28+0800	2026-07-24T12:42:42+0800	283420	55,213.853 s	logs/train_1p2b_20260723_212215.out.log
4	43768	2026-07-24T14:40:03+0800	2026-08-02T13:42:10+0800	390970	774,127.557 s	logs/train_1p2b_20260724_143950.out.log
5	21296	2026-08-02T18:09:29+0800	2026-08-04T03:00:26+0800	406290	118,256.100 s	logs/train_1p2b_20260802_180917.out.log
6	34328	2026-08-04T12:07:23+0800	2026-08-11T12:01:52+0800	488000	604,458.520 s	logs/train_1p2b_20260804_120708.out.log
7	2444	2026-08-11T20:03:15+0800	2026-08-12T18:59:47+0800	500000	82,592.597 s	logs/train_1p2b_20260811_200300.out.log

Segment 1 was interrupted after iter 203850 and resumed from the iter 203000 checkpoint. Segment 2 ended with a CUDA illegal-memory-access error after iter 275550, alongside NVIDIA driver errors while another GPU training workload was launched. Segment 3 resumed from the complete iter 275000 checkpoint and will recompute at most 550 optimizer steps from segment 2. Segment 3 was intentionally paused to release the GPU. Its most recent complete checkpoint is iter 283000, so the next resume will recompute at most 420 optimizer steps. Segment 4 resumed from that checkpoint after the other GPU workload finished. Segment 4 was intentionally paused before a planned power outage. The next automatic checkpoint at iter 391000 could not be reached inside the requested half-hour window, so the next resume will use the complete iter 390000 checkpoint and recompute at most 970 optimizer steps. Segment 5 resumed from that checkpoint after power was restored, then exited during backward near iter 406290 with a CUDA out-of-memory error. The latest complete checkpoint is iter 406000; no concurrent NVIDIA driver reset event was found. Segment 6 resumed from that checkpoint after the GPU became available and recomputed at most 290 optimizer steps from segment 5. Segment 6 was intentionally paused after the automatic iter 488000 checkpoint was fully written. The next run can resume directly from that checkpoint without recomputing completed optimizer steps. Progress is 97.6%, with 12000 steps, or 786,432,000 training tokens, remaining. Segment 7 resumed from that checkpoint after the GPU became available again and completed normally at iter 500000. The final latest.pt was fully saved and stderr remained empty.

training_state.json records the final iteration, tokens seen, completion time, and elapsed time for the final process segment. Total active training time is the sum of all seven segments above; calendar elapsed time includes pauses and interruptions.

Contributing And Security

Contributions are welcome under the process in CONTRIBUTING.md. Please report security issues privately as described in SECURITY.md, not through a public issue.

License

Repository source code and documentation are licensed under the Apache License 2.0. See NOTICE and THIRD_PARTY_NOTICES.md for attribution. This license does not cover training datasets, generated corpora, token arrays, checkpoints, or model weights.

Notes

This project is in active local development. README files should be updated after each completed stage so the repository remains an accurate record of the current state.