Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1481 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.7717. Min Act: -0.0001

Data Index: 9054 (C4 (Web Text))

Max Activating Token Index: 78

Text #1

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5821. Min Act: -0.0000

Data Index: 692813 (C4 (Web Text))

Max Activating Token Index: 901

Text #2

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5888. Min Act: -0.0001

Data Index: 227447 (C4 (Web Text))

Max Activating Token Index: 495

Text #3

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5756. Min Act: -0.0001

Data Index: 70677 (C4 (Web Text))

Max Activating Token Index: 567

Text #4

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.4998. Min Act: -0.0001

Data Index: 760432 (C4 (Web Text))

Max Activating Token Index: 368

Text #5

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5880. Min Act: -0.0000

Data Index: 756573 (C4 (Web Text))

Max Activating Token Index: 149

Text #6

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5164. Min Act: -0.0001

Data Index: 613663 (C4 (Web Text))

Max Activating Token Index: 492

Text #7

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5238. Min Act: -0.0001

Data Index: 327029 (C4 (Web Text))

Max Activating Token Index: 73

Text #8

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5110. Min Act: -0.0001

Data Index: 396834 (C4 (Web Text))

Max Activating Token Index: 689

Text #9

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5446. Min Act: -0.0001

Data Index: 315935 (C4 (Web Text))

Max Activating Token Index: 559

Text #10

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5212. Min Act: -0.0001

Data Index: 811564 (C4 (Web Text))

Max Activating Token Index: 401

Text #11

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5139. Min Act: -0.0001

Data Index: 704228 (C4 (Web Text))

Max Activating Token Index: 562

Text #12

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5101. Min Act: -0.0001

Data Index: 486926 (C4 (Web Text))

Max Activating Token Index: 965

Text #13

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5021. Min Act: -0.0001

Data Index: 960097 (C4 (Web Text))

Max Activating Token Index: 196

Text #14

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5140. Min Act: -0.0001

Data Index: 1075578 (C4 (Web Text))

Max Activating Token Index: 1009

Text #15

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5157. Min Act: -0.0001

Data Index: 548052 (C4 (Web Text))

Max Activating Token Index: 298

Text #16

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.5245. Min Act: -0.0001

Data Index: 610335 (C4 (Web Text))

Max Activating Token Index: 408

Text #17

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.4950. Min Act: -0.0001

Data Index: 321811 (C4 (Web Text))

Max Activating Token Index: 488

Text #18

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.4731. Min Act: -0.0000

Data Index: 751172 (C4 (Web Text))

Max Activating Token Index: 345

Text #19

Max Range: 0.7717. Min Range: -0.7717

Max Act: 0.4865. Min Act: -0.0001

Data Index: 515120 (C4 (Web Text))

Max Activating Token Index: 538