Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2167 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.5482. Min Act: -0.0000

Data Index: 575472 (C4 (Web Text))

Max Activating Token Index: 818

Text #1

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.3774. Min Act: -0.0000

Data Index: 686823 (C4 (Web Text))

Max Activating Token Index: 391

Text #2

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.1351. Min Act: -0.0000

Data Index: 249478 (C4 (Web Text))

Max Activating Token Index: 684

Text #3

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.2120. Min Act: -0.0000

Data Index: 298204 (C4 (Web Text))

Max Activating Token Index: 527

Text #4

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.1893. Min Act: -0.0000

Data Index: 1217279 (C4 (Web Text))

Max Activating Token Index: 456

Text #5

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.1899. Min Act: -0.0000

Data Index: 698144 (C4 (Web Text))

Max Activating Token Index: 890

Text #6

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.1951. Min Act: -0.0000

Data Index: 1211553 (C4 (Web Text))

Max Activating Token Index: 181

Text #7

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.1299. Min Act: -0.0000

Data Index: 505913 (C4 (Web Text))

Max Activating Token Index: 190

Text #8

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.1033. Min Act: -0.0000

Data Index: 769950 (C4 (Web Text))

Max Activating Token Index: 82

Text #9

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.0962. Min Act: -0.0000

Data Index: 1203917 (C4 (Web Text))

Max Activating Token Index: 110

Text #10

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.0991. Min Act: -0.0000

Data Index: 181081 (C4 (Web Text))

Max Activating Token Index: 26

Text #11

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.0659. Min Act: -0.0000

Data Index: 427358 (C4 (Web Text))

Max Activating Token Index: 326

Text #12

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.0643. Min Act: -0.0000

Data Index: 862531 (C4 (Web Text))

Max Activating Token Index: 113

Text #13

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.0859. Min Act: -0.0000

Data Index: 1028925 (C4 (Web Text))

Max Activating Token Index: 137

Text #14

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.0686. Min Act: -0.0000

Data Index: 91873 (C4 (Web Text))

Max Activating Token Index: 116

Text #15

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.0610. Min Act: -0.0000

Data Index: 515086 (C4 (Web Text))

Max Activating Token Index: 871

Text #16

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.0317. Min Act: -0.0000

Data Index: 1270071 (C4 (Web Text))

Max Activating Token Index: 406

Text #17

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.0067. Min Act: -0.0000

Data Index: 1041378 (C4 (Web Text))

Max Activating Token Index: 660

Text #18

Max Range: 3.5482. Min Range: -3.5482

Max Act: 3.1298. Min Act: -0.0000

Data Index: 831996 (C4 (Web Text))

Max Activating Token Index: 232

Text #19

Max Range: 3.5482. Min Range: -3.5482

Max Act: 2.8615. Min Act: -0.0000

Data Index: 280054 (C4 (Web Text))

Max Activating Token Index: 992