Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3550 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.1220. Min Range: -4.1220

Max Act: 3.2462. Min Act: -0.0001

Data Index: 739136 (C4 (Web Text))

Max Activating Token Index: 896

Text #1

Max Range: 4.1220. Min Range: -4.1220

Max Act: 4.1220. Min Act: -0.0001

Data Index: 278916 (C4 (Web Text))

Max Activating Token Index: 222

Text #2

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.2334. Min Act: -0.0000

Data Index: 1371889 (Python Code)

Max Activating Token Index: 845

Text #3

Max Range: 4.1220. Min Range: -4.1220

Max Act: 3.0430. Min Act: -0.0000

Data Index: 37182 (C4 (Web Text))

Max Activating Token Index: 413

Text #4

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.7209. Min Act: -0.0000

Data Index: 794115 (C4 (Web Text))

Max Activating Token Index: 738

Text #5

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.4825. Min Act: -0.0001

Data Index: 755791 (C4 (Web Text))

Max Activating Token Index: 980

Text #6

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.7078. Min Act: -0.0001

Data Index: 1193594 (C4 (Web Text))

Max Activating Token Index: 841

Text #7

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.6804. Min Act: -0.0000

Data Index: 1142780 (C4 (Web Text))

Max Activating Token Index: 250

Text #8

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.3275. Min Act: -0.0001

Data Index: 254822 (C4 (Web Text))

Max Activating Token Index: 803

Text #9

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.5147. Min Act: -0.0000

Data Index: 120291 (C4 (Web Text))

Max Activating Token Index: 583

Text #10

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.2319. Min Act: -0.0001

Data Index: 491981 (C4 (Web Text))

Max Activating Token Index: 1008

Text #11

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.3036. Min Act: -0.0001

Data Index: 832036 (C4 (Web Text))

Max Activating Token Index: 657

Text #12

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.5292. Min Act: -0.0000

Data Index: 398256 (C4 (Web Text))

Max Activating Token Index: 304

Text #13

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.4944. Min Act: -0.0000

Data Index: 925858 (C4 (Web Text))

Max Activating Token Index: 104

Text #14

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.5956. Min Act: -0.0001

Data Index: 493797 (C4 (Web Text))

Max Activating Token Index: 343

Text #15

Max Range: 4.1220. Min Range: -4.1220

Max Act: 1.9982. Min Act: -0.0000

Data Index: 951770 (C4 (Web Text))

Max Activating Token Index: 856

Text #16

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.0777. Min Act: -0.0000

Data Index: 1105040 (C4 (Web Text))

Max Activating Token Index: 712

Text #17

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.1088. Min Act: -0.0000

Data Index: 1231821 (C4 (Web Text))

Max Activating Token Index: 27

Text #18

Max Range: 4.1220. Min Range: -4.1220

Max Act: 2.0517. Min Act: -0.0001

Data Index: 1193408 (C4 (Web Text))

Max Activating Token Index: 119

Text #19

Max Range: 4.1220. Min Range: -4.1220

Max Act: 1.9314. Min Act: -0.0001

Data Index: 250489 (C4 (Web Text))

Max Activating Token Index: 891