Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2096 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.9190. Min Act: -0.0000

Data Index: 1195830 (C4 (Web Text))

Max Activating Token Index: 533

Text #1

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.9234. Min Act: -0.0000

Data Index: 333949 (C4 (Web Text))

Max Activating Token Index: 259

Text #2

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.6165. Min Act: -0.0000

Data Index: 1197376 (C4 (Web Text))

Max Activating Token Index: 449

Text #3

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.6152. Min Act: -0.0000

Data Index: 1041705 (C4 (Web Text))

Max Activating Token Index: 560

Text #4

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.6847. Min Act: -0.0000

Data Index: 670382 (C4 (Web Text))

Max Activating Token Index: 794

Text #5

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.6354. Min Act: -0.0000

Data Index: 590327 (C4 (Web Text))

Max Activating Token Index: 571

Text #6

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.6174. Min Act: -0.0000

Data Index: 665001 (C4 (Web Text))

Max Activating Token Index: 516

Text #7

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.6698. Min Act: -0.0000

Data Index: 1050585 (C4 (Web Text))

Max Activating Token Index: 21

Text #8

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.5552. Min Act: -0.0000

Data Index: 1234726 (C4 (Web Text))

Max Activating Token Index: 258

Text #9

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.4333. Min Act: -0.0000

Data Index: 852140 (C4 (Web Text))

Max Activating Token Index: 465

Text #10

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.4837. Min Act: -0.0000

Data Index: 704170 (C4 (Web Text))

Max Activating Token Index: 192

Text #11

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.7357. Min Act: -0.0000

Data Index: 390087 (C4 (Web Text))

Max Activating Token Index: 159

Text #12

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.4265. Min Act: -0.0000

Data Index: 180609 (C4 (Web Text))

Max Activating Token Index: 303

Text #13

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.4451. Min Act: -0.0000

Data Index: 161028 (C4 (Web Text))

Max Activating Token Index: 360

Text #14

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.8012. Min Act: -0.0000

Data Index: 1207485 (C4 (Web Text))

Max Activating Token Index: 680

Text #15

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.4295. Min Act: -0.0000

Data Index: 1046115 (C4 (Web Text))

Max Activating Token Index: 357

Text #16

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.5170. Min Act: -0.0000

Data Index: 181402 (C4 (Web Text))

Max Activating Token Index: 479

Text #17

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.4873. Min Act: -0.0000

Data Index: 94955 (C4 (Web Text))

Max Activating Token Index: 614

Text #18

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.4812. Min Act: -0.0000

Data Index: 1286038 (C4 (Web Text))

Max Activating Token Index: 675

Text #19

Max Range: 1.9234. Min Range: -1.9234

Max Act: 1.4238. Min Act: -0.0000

Data Index: 751241 (C4 (Web Text))

Max Activating Token Index: 569