Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 858 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.1375. Min Range: -1.1375

Max Act: 1.0978. Min Act: -0.0000

Data Index: 1027479 (C4 (Web Text))

Max Activating Token Index: 852

Text #1

Max Range: 1.1375. Min Range: -1.1375

Max Act: 1.1375. Min Act: -0.0000

Data Index: 1037910 (C4 (Web Text))

Max Activating Token Index: 314

Text #2

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.9971. Min Act: 0.0000

Data Index: 1343571 (C4 (Web Text))

Max Activating Token Index: 309

Text #3

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.9441. Min Act: -0.0000

Data Index: 130461 (C4 (Web Text))

Max Activating Token Index: 280

Text #4

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.8779. Min Act: -0.0000

Data Index: 275984 (C4 (Web Text))

Max Activating Token Index: 577

Text #5

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.8240. Min Act: -0.0000

Data Index: 735264 (C4 (Web Text))

Max Activating Token Index: 396

Text #6

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.8541. Min Act: -0.0000

Data Index: 235311 (C4 (Web Text))

Max Activating Token Index: 498

Text #7

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.8088. Min Act: -0.0000

Data Index: 1163714 (C4 (Web Text))

Max Activating Token Index: 771

Text #8

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.8732. Min Act: -0.0000

Data Index: 9593 (C4 (Web Text))

Max Activating Token Index: 570

Text #9

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.8397. Min Act: -0.0000

Data Index: 1036089 (C4 (Web Text))

Max Activating Token Index: 15

Text #10

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.8655. Min Act: -0.0000

Data Index: 978154 (C4 (Web Text))

Max Activating Token Index: 548

Text #11

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.7851. Min Act: -0.0000

Data Index: 23561 (C4 (Web Text))

Max Activating Token Index: 528

Text #12

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.8099. Min Act: -0.0000

Data Index: 1037145 (C4 (Web Text))

Max Activating Token Index: 757

Text #13

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.7998. Min Act: -0.0000

Data Index: 1102651 (C4 (Web Text))

Max Activating Token Index: 490

Text #14

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.7916. Min Act: -0.0000

Data Index: 517476 (C4 (Web Text))

Max Activating Token Index: 570

Text #15

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.7848. Min Act: -0.0000

Data Index: 327417 (C4 (Web Text))

Max Activating Token Index: 635

Text #16

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.7738. Min Act: -0.0000

Data Index: 1170751 (C4 (Web Text))

Max Activating Token Index: 436

Text #17

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.7707. Min Act: -0.0000

Data Index: 1196859 (C4 (Web Text))

Max Activating Token Index: 731

Text #18

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.8124. Min Act: -0.0000

Data Index: 1053181 (C4 (Web Text))

Max Activating Token Index: 35

Text #19

Max Range: 1.1375. Min Range: -1.1375

Max Act: 0.7608. Min Act: -0.0000

Data Index: 865913 (C4 (Web Text))

Max Activating Token Index: 618