Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 253 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.8942. Min Range: -4.8942

Max Act: 4.8942. Min Act: -0.0000

Data Index: 381747 (C4 (Web Text))

Max Activating Token Index: 66

Text #1

Max Range: 4.8942. Min Range: -4.8942

Max Act: 4.4231. Min Act: -0.0000

Data Index: 859522 (C4 (Web Text))

Max Activating Token Index: 91

Text #2

Max Range: 4.8942. Min Range: -4.8942

Max Act: 4.6341. Min Act: -0.0000

Data Index: 74703 (C4 (Web Text))

Max Activating Token Index: 76

Text #3

Max Range: 4.8942. Min Range: -4.8942

Max Act: 4.4369. Min Act: -0.0000

Data Index: 1305584 (C4 (Web Text))

Max Activating Token Index: 156

Text #4

Max Range: 4.8942. Min Range: -4.8942

Max Act: 4.1198. Min Act: -0.0000

Data Index: 410162 (C4 (Web Text))

Max Activating Token Index: 223

Text #5

Max Range: 4.8942. Min Range: -4.8942

Max Act: 4.1117. Min Act: -0.0000

Data Index: 812470 (C4 (Web Text))

Max Activating Token Index: 144

Text #6

Max Range: 4.8942. Min Range: -4.8942

Max Act: 4.0415. Min Act: -0.0000

Data Index: 1085931 (C4 (Web Text))

Max Activating Token Index: 444

Text #7

Max Range: 4.8942. Min Range: -4.8942

Max Act: 4.1760. Min Act: -0.0000

Data Index: 220458 (C4 (Web Text))

Max Activating Token Index: 88

Text #8

Max Range: 4.8942. Min Range: -4.8942

Max Act: 3.8093. Min Act: -0.0000

Data Index: 260119 (C4 (Web Text))

Max Activating Token Index: 354

Text #9

Max Range: 4.8942. Min Range: -4.8942

Max Act: 3.6141. Min Act: -0.0000

Data Index: 562011 (C4 (Web Text))

Max Activating Token Index: 420

Text #10

Max Range: 4.8942. Min Range: -4.8942

Max Act: 4.0892. Min Act: -0.0000

Data Index: 1312541 (C4 (Web Text))

Max Activating Token Index: 251

Text #11

Max Range: 4.8942. Min Range: -4.8942

Max Act: 3.7657. Min Act: -0.0000

Data Index: 1024477 (C4 (Web Text))

Max Activating Token Index: 601

Text #12

Max Range: 4.8942. Min Range: -4.8942

Max Act: 3.8791. Min Act: -0.0000

Data Index: 1032246 (C4 (Web Text))

Max Activating Token Index: 802

Text #13

Max Range: 4.8942. Min Range: -4.8942

Max Act: 3.8358. Min Act: -0.0000

Data Index: 464537 (C4 (Web Text))

Max Activating Token Index: 216

Text #14

Max Range: 4.8942. Min Range: -4.8942

Max Act: 3.6080. Min Act: -0.0000

Data Index: 277 (C4 (Web Text))

Max Activating Token Index: 211

Text #15

Max Range: 4.8942. Min Range: -4.8942

Max Act: 3.5874. Min Act: -0.0000

Data Index: 730489 (C4 (Web Text))

Max Activating Token Index: 585

Text #16

Max Range: 4.8942. Min Range: -4.8942

Max Act: 3.4481. Min Act: -0.0000

Data Index: 1267916 (C4 (Web Text))

Max Activating Token Index: 977

Text #17

Max Range: 4.8942. Min Range: -4.8942

Max Act: 3.7773. Min Act: -0.0000

Data Index: 836686 (C4 (Web Text))

Max Activating Token Index: 481

Text #18

Max Range: 4.8942. Min Range: -4.8942

Max Act: 3.5922. Min Act: -0.0000

Data Index: 238419 (C4 (Web Text))

Max Activating Token Index: 314

Text #19

Max Range: 4.8942. Min Range: -4.8942

Max Act: 4.0770. Min Act: -0.0000

Data Index: 395680 (C4 (Web Text))

Max Activating Token Index: 888