Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 98 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.4029. Min Range: -3.4029

Max Act: 3.4029. Min Act: -0.0000

Data Index: 741569 (C4 (Web Text))

Max Activating Token Index: 534

Text #1

Max Range: 3.4029. Min Range: -3.4029

Max Act: 3.2708. Min Act: -0.0000

Data Index: 1060458 (C4 (Web Text))

Max Activating Token Index: 124

Text #2

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.7280. Min Act: -0.0000

Data Index: 374810 (C4 (Web Text))

Max Activating Token Index: 397

Text #3

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.7501. Min Act: -0.0000

Data Index: 869808 (C4 (Web Text))

Max Activating Token Index: 607

Text #4

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.7267. Min Act: -0.0000

Data Index: 288379 (C4 (Web Text))

Max Activating Token Index: 229

Text #5

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.7120. Min Act: -0.0000

Data Index: 807101 (C4 (Web Text))

Max Activating Token Index: 860

Text #6

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.4176. Min Act: -0.0000

Data Index: 239248 (C4 (Web Text))

Max Activating Token Index: 873

Text #7

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.5985. Min Act: -0.0000

Data Index: 59689 (C4 (Web Text))

Max Activating Token Index: 178

Text #8

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.5876. Min Act: -0.0000

Data Index: 1354170 (C4 (Web Text))

Max Activating Token Index: 469

Text #9

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.6945. Min Act: -0.0000

Data Index: 578819 (C4 (Web Text))

Max Activating Token Index: 275

Text #10

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.5397. Min Act: -0.0000

Data Index: 889289 (C4 (Web Text))

Max Activating Token Index: 243

Text #11

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.2337. Min Act: -0.0000

Data Index: 394685 (C4 (Web Text))

Max Activating Token Index: 535

Text #12

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.4240. Min Act: -0.0000

Data Index: 275080 (C4 (Web Text))

Max Activating Token Index: 326

Text #13

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.2788. Min Act: -0.0000

Data Index: 947128 (C4 (Web Text))

Max Activating Token Index: 402

Text #14

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.4004. Min Act: -0.0000

Data Index: 519795 (C4 (Web Text))

Max Activating Token Index: 642

Text #15

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.1474. Min Act: -0.0000

Data Index: 651217 (C4 (Web Text))

Max Activating Token Index: 575

Text #16

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.3821. Min Act: -0.0000

Data Index: 1223086 (C4 (Web Text))

Max Activating Token Index: 250

Text #17

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.4402. Min Act: -0.0000

Data Index: 1024263 (C4 (Web Text))

Max Activating Token Index: 327

Text #18

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.3716. Min Act: -0.0000

Data Index: 1005069 (C4 (Web Text))

Max Activating Token Index: 381

Text #19

Max Range: 3.4029. Min Range: -3.4029

Max Act: 2.2452. Min Act: -0.0000

Data Index: 275195 (C4 (Web Text))

Max Activating Token Index: 661