Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4753 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.3107. Min Range: -2.3107

Max Act: 2.3107. Min Act: -0.0000

Data Index: 1201005 (C4 (Web Text))

Max Activating Token Index: 617

Text #1

Max Range: 2.3107. Min Range: -2.3107

Max Act: 2.2120. Min Act: -0.0000

Data Index: 302603 (C4 (Web Text))

Max Activating Token Index: 439

Text #2

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.8822. Min Act: -0.0000

Data Index: 508225 (C4 (Web Text))

Max Activating Token Index: 167

Text #3

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.6812. Min Act: -0.0000

Data Index: 879212 (C4 (Web Text))

Max Activating Token Index: 450

Text #4

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.6982. Min Act: -0.0000

Data Index: 1431211 (Python Code)

Max Activating Token Index: 28

Text #5

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.6584. Min Act: -0.0000

Data Index: 198439 (C4 (Web Text))

Max Activating Token Index: 844

Text #6

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.2822. Min Act: -0.0000

Data Index: 839430 (C4 (Web Text))

Max Activating Token Index: 251

Text #7

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.4059. Min Act: -0.0000

Data Index: 12016 (C4 (Web Text))

Max Activating Token Index: 501

Text #8

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.3322. Min Act: -0.0000

Data Index: 89021 (C4 (Web Text))

Max Activating Token Index: 613

Text #9

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.2357. Min Act: -0.0000

Data Index: 410699 (C4 (Web Text))

Max Activating Token Index: 51

Text #10

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.2160. Min Act: -0.0000

Data Index: 520198 (C4 (Web Text))

Max Activating Token Index: 192

Text #11

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.2781. Min Act: -0.0000

Data Index: 870013 (C4 (Web Text))

Max Activating Token Index: 409

Text #12

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.2019. Min Act: -0.0000

Data Index: 1285030 (C4 (Web Text))

Max Activating Token Index: 464

Text #13

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.1740. Min Act: -0.0000

Data Index: 94289 (C4 (Web Text))

Max Activating Token Index: 260

Text #14

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.1215. Min Act: -0.0000

Data Index: 742957 (C4 (Web Text))

Max Activating Token Index: 273

Text #15

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.1514. Min Act: 0.0000

Data Index: 479909 (C4 (Web Text))

Max Activating Token Index: 191

Text #16

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.2029. Min Act: -0.0000

Data Index: 84431 (C4 (Web Text))

Max Activating Token Index: 279

Text #17

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.0704. Min Act: -0.0000

Data Index: 1277412 (C4 (Web Text))

Max Activating Token Index: 624

Text #18

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.0521. Min Act: -0.0000

Data Index: 83937 (C4 (Web Text))

Max Activating Token Index: 909

Text #19

Max Range: 2.3107. Min Range: -2.3107

Max Act: 1.0476. Min Act: 0.0000

Data Index: 876004 (C4 (Web Text))

Max Activating Token Index: 100