Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 457 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.9756. Min Act: -0.0002

Data Index: 1629080 (Python Code)

Max Activating Token Index: 76

Text #1

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8908. Min Act: -0.0002

Data Index: 1373779 (Python Code)

Max Activating Token Index: 267

Text #2

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8924. Min Act: -0.0002

Data Index: 1642960 (Python Code)

Max Activating Token Index: 534

Text #3

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8593. Min Act: -0.0002

Data Index: 1537306 (Python Code)

Max Activating Token Index: 1021

Text #4

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8168. Min Act: -0.0002

Data Index: 1544878 (Python Code)

Max Activating Token Index: 616

Text #5

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8379. Min Act: -0.0002

Data Index: 1515016 (Python Code)

Max Activating Token Index: 545

Text #6

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8412. Min Act: -0.0002

Data Index: 1596736 (Python Code)

Max Activating Token Index: 159

Text #7

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.7930. Min Act: -0.0002

Data Index: 1249365 (C4 (Web Text))

Max Activating Token Index: 687

Text #8

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8198. Min Act: -0.0002

Data Index: 1618209 (Python Code)

Max Activating Token Index: 937

Text #9

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.7869. Min Act: -0.0002

Data Index: 65411 (C4 (Web Text))

Max Activating Token Index: 723

Text #10

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8147. Min Act: -0.0002

Data Index: 1531226 (Python Code)

Max Activating Token Index: 976

Text #11

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8085. Min Act: -0.0002

Data Index: 1508780 (Python Code)

Max Activating Token Index: 258

Text #12

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8338. Min Act: -0.0002

Data Index: 1545852 (Python Code)

Max Activating Token Index: 131

Text #13

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8276. Min Act: -0.0002

Data Index: 1441081 (Python Code)

Max Activating Token Index: 291

Text #14

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8030. Min Act: -0.0002

Data Index: 1485099 (Python Code)

Max Activating Token Index: 727

Text #15

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.8475. Min Act: -0.0002

Data Index: 1565031 (Python Code)

Max Activating Token Index: 109

Text #16

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.7931. Min Act: -0.0002

Data Index: 1470338 (Python Code)

Max Activating Token Index: 318

Text #17

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.7404. Min Act: -0.0002

Data Index: 1445458 (Python Code)

Max Activating Token Index: 414

Text #18

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.7659. Min Act: -0.0002

Data Index: 1407857 (Python Code)

Max Activating Token Index: 43

Text #19

Max Range: 0.9756. Min Range: -0.9756

Max Act: 0.7519. Min Act: -0.0002

Data Index: 1587113 (Python Code)

Max Activating Token Index: 958