Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5688 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.2864. Min Act: -0.0002

Data Index: 411377 (C4 (Web Text))

Max Activating Token Index: 82

Text #1

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.1829. Min Act: -0.0002

Data Index: 566117 (C4 (Web Text))

Max Activating Token Index: 121

Text #2

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.1425. Min Act: -0.0002

Data Index: 944593 (C4 (Web Text))

Max Activating Token Index: 101

Text #3

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.1899. Min Act: -0.0002

Data Index: 668852 (C4 (Web Text))

Max Activating Token Index: 325

Text #4

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.2095. Min Act: -0.0002

Data Index: 917550 (C4 (Web Text))

Max Activating Token Index: 191

Text #5

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.1505. Min Act: -0.0002

Data Index: 669418 (C4 (Web Text))

Max Activating Token Index: 131

Text #6

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.1749. Min Act: -0.0002

Data Index: 715852 (C4 (Web Text))

Max Activating Token Index: 848

Text #7

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.1691. Min Act: -0.0002

Data Index: 994363 (C4 (Web Text))

Max Activating Token Index: 354

Text #8

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.1175. Min Act: -0.0002

Data Index: 843093 (C4 (Web Text))

Max Activating Token Index: 498

Text #9

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.0875. Min Act: -0.0002

Data Index: 139871 (C4 (Web Text))

Max Activating Token Index: 797

Text #10

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.0508. Min Act: -0.0002

Data Index: 480190 (C4 (Web Text))

Max Activating Token Index: 69

Text #11

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.0440. Min Act: -0.0002

Data Index: 895306 (C4 (Web Text))

Max Activating Token Index: 94

Text #12

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.0235. Min Act: -0.0002

Data Index: 149675 (C4 (Web Text))

Max Activating Token Index: 86

Text #13

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.0128. Min Act: -0.0002

Data Index: 154789 (C4 (Web Text))

Max Activating Token Index: 712

Text #14

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.0138. Min Act: -0.0002

Data Index: 68853 (C4 (Web Text))

Max Activating Token Index: 406

Text #15

Max Range: 1.2864. Min Range: -1.2864

Max Act: 0.9960. Min Act: -0.0002

Data Index: 355850 (C4 (Web Text))

Max Activating Token Index: 251

Text #16

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.0259. Min Act: -0.0002

Data Index: 1001084 (C4 (Web Text))

Max Activating Token Index: 984

Text #17

Max Range: 1.2864. Min Range: -1.2864

Max Act: 0.9908. Min Act: -0.0002

Data Index: 925730 (C4 (Web Text))

Max Activating Token Index: 166

Text #18

Max Range: 1.2864. Min Range: -1.2864

Max Act: 1.0196. Min Act: -0.0002

Data Index: 719388 (C4 (Web Text))

Max Activating Token Index: 403

Text #19

Max Range: 1.2864. Min Range: -1.2864

Max Act: 0.9862. Min Act: -0.0002

Data Index: 1189677 (C4 (Web Text))

Max Activating Token Index: 520