Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4365 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.9128. Min Act: -0.0000

Data Index: 120191 (C4 (Web Text))

Max Activating Token Index: 153

Text #1

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.5019. Min Act: -0.0000

Data Index: 741200 (C4 (Web Text))

Max Activating Token Index: 461

Text #2

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.3616. Min Act: -0.0000

Data Index: 229731 (C4 (Web Text))

Max Activating Token Index: 481

Text #3

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.3746. Min Act: -0.0000

Data Index: 1062414 (C4 (Web Text))

Max Activating Token Index: 170

Text #4

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.3528. Min Act: -0.0000

Data Index: 531192 (C4 (Web Text))

Max Activating Token Index: 32

Text #5

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.4125. Min Act: -0.0000

Data Index: 559780 (C4 (Web Text))

Max Activating Token Index: 367

Text #6

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.1795. Min Act: -0.0000

Data Index: 866737 (C4 (Web Text))

Max Activating Token Index: 467

Text #7

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.1651. Min Act: -0.0000

Data Index: 1316570 (C4 (Web Text))

Max Activating Token Index: 780

Text #8

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.1693. Min Act: -0.0000

Data Index: 491559 (C4 (Web Text))

Max Activating Token Index: 380

Text #9

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.2285. Min Act: -0.0000

Data Index: 987281 (C4 (Web Text))

Max Activating Token Index: 95

Text #10

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.2208. Min Act: -0.0000

Data Index: 427858 (C4 (Web Text))

Max Activating Token Index: 685

Text #11

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.1495. Min Act: -0.0000

Data Index: 729290 (C4 (Web Text))

Max Activating Token Index: 963

Text #12

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.1378. Min Act: -0.0000

Data Index: 583442 (C4 (Web Text))

Max Activating Token Index: 378

Text #13

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.1226. Min Act: -0.0000

Data Index: 859109 (C4 (Web Text))

Max Activating Token Index: 333

Text #14

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.2553. Min Act: -0.0000

Data Index: 1329133 (C4 (Web Text))

Max Activating Token Index: 53

Text #15

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.2426. Min Act: -0.0000

Data Index: 529313 (C4 (Web Text))

Max Activating Token Index: 206

Text #16

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.1752. Min Act: -0.0000

Data Index: 442187 (C4 (Web Text))

Max Activating Token Index: 640

Text #17

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.0794. Min Act: -0.0000

Data Index: 702090 (C4 (Web Text))

Max Activating Token Index: 185

Text #18

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.0757. Min Act: -0.0000

Data Index: 1357379 (C4 (Web Text))

Max Activating Token Index: 285

Text #19

Max Range: 1.9128. Min Range: -1.9128

Max Act: 1.0711. Min Act: -0.0000

Data Index: 82180 (C4 (Web Text))

Max Activating Token Index: 54