Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4505 in Layer 11

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.6726. Min Act: -0.0000

Data Index: 1157179 (C4 (Web Text))

Max Activating Token Index: 589

Text #1

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.5829. Min Act: -0.0000

Data Index: 1291664 (C4 (Web Text))

Max Activating Token Index: 650

Text #2

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.6067. Min Act: -0.0000

Data Index: 42041 (C4 (Web Text))

Max Activating Token Index: 190

Text #3

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.6175. Min Act: -0.0000

Data Index: 766142 (C4 (Web Text))

Max Activating Token Index: 513

Text #4

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.5131. Min Act: -0.0000

Data Index: 1129372 (C4 (Web Text))

Max Activating Token Index: 481

Text #5

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.6838. Min Act: -0.0000

Data Index: 1000734 (C4 (Web Text))

Max Activating Token Index: 190

Text #6

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.5082. Min Act: -0.0000

Data Index: 25853 (C4 (Web Text))

Max Activating Token Index: 340

Text #7

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.5548. Min Act: -0.0000

Data Index: 782178 (C4 (Web Text))

Max Activating Token Index: 943

Text #8

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.5293. Min Act: -0.0000

Data Index: 1147227 (C4 (Web Text))

Max Activating Token Index: 651

Text #9

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.4006. Min Act: -0.0000

Data Index: 1185503 (C4 (Web Text))

Max Activating Token Index: 690

Text #10

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.6066. Min Act: -0.0000

Data Index: 555401 (C4 (Web Text))

Max Activating Token Index: 180

Text #11

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.6056. Min Act: -0.0000

Data Index: 1064531 (C4 (Web Text))

Max Activating Token Index: 184

Text #12

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.5881. Min Act: -0.0000

Data Index: 195029 (C4 (Web Text))

Max Activating Token Index: 819

Text #13

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.4031. Min Act: -0.0000

Data Index: 763195 (C4 (Web Text))

Max Activating Token Index: 625

Text #14

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.7171. Min Act: -0.0000

Data Index: 689538 (C4 (Web Text))

Max Activating Token Index: 392

Text #15

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.6717. Min Act: -0.0000

Data Index: 106580 (C4 (Web Text))

Max Activating Token Index: 828

Text #16

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.5878. Min Act: -0.0000

Data Index: 775134 (C4 (Web Text))

Max Activating Token Index: 350

Text #17

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.5056. Min Act: -0.0000

Data Index: 231319 (C4 (Web Text))

Max Activating Token Index: 452

Text #18

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.4929. Min Act: -0.0000

Data Index: 1256377 (C4 (Web Text))

Max Activating Token Index: 521

Text #19

Max Range: 1.7171. Min Range: -1.7171

Max Act: 1.3842. Min Act: -0.0000

Data Index: 530287 (C4 (Web Text))

Max Activating Token Index: 435