Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2886 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.9759. Min Act: -0.0000

Data Index: 128575 (C4 (Web Text))

Max Activating Token Index: 259

Text #1

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.9140. Min Act: -0.0000

Data Index: 127131 (C4 (Web Text))

Max Activating Token Index: 746

Text #2

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.9392. Min Act: -0.0000

Data Index: 12810 (C4 (Web Text))

Max Activating Token Index: 786

Text #3

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.9336. Min Act: -0.0000

Data Index: 98632 (C4 (Web Text))

Max Activating Token Index: 390

Text #4

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.8547. Min Act: -0.0000

Data Index: 791607 (C4 (Web Text))

Max Activating Token Index: 672

Text #5

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.7906. Min Act: -0.0000

Data Index: 926441 (C4 (Web Text))

Max Activating Token Index: 339

Text #6

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.8187. Min Act: -0.0000

Data Index: 700566 (C4 (Web Text))

Max Activating Token Index: 997

Text #7

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.8420. Min Act: -0.0000

Data Index: 1095212 (C4 (Web Text))

Max Activating Token Index: 68

Text #8

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.7701. Min Act: -0.0000

Data Index: 411547 (C4 (Web Text))

Max Activating Token Index: 475

Text #9

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.7598. Min Act: -0.0000

Data Index: 1081405 (C4 (Web Text))

Max Activating Token Index: 375

Text #10

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.7680. Min Act: -0.0000

Data Index: 340672 (C4 (Web Text))

Max Activating Token Index: 342

Text #11

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.7899. Min Act: -0.0000

Data Index: 1198985 (C4 (Web Text))

Max Activating Token Index: 91

Text #12

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.7579. Min Act: -0.0000

Data Index: 575401 (C4 (Web Text))

Max Activating Token Index: 520

Text #13

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.7088. Min Act: -0.0000

Data Index: 782020 (C4 (Web Text))

Max Activating Token Index: 689

Text #14

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.7541. Min Act: -0.0000

Data Index: 1264668 (C4 (Web Text))

Max Activating Token Index: 213

Text #15

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.7280. Min Act: -0.0000

Data Index: 758374 (C4 (Web Text))

Max Activating Token Index: 790

Text #16

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.7727. Min Act: -0.0000

Data Index: 432613 (C4 (Web Text))

Max Activating Token Index: 448

Text #17

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.7447. Min Act: -0.0000

Data Index: 1109110 (C4 (Web Text))

Max Activating Token Index: 380

Text #18

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.6981. Min Act: -0.0000

Data Index: 262918 (C4 (Web Text))

Max Activating Token Index: 140

Text #19

Max Range: 0.9759. Min Range: -0.9759

Max Act: 0.7555. Min Act: -0.0000

Data Index: 857141 (C4 (Web Text))

Max Activating Token Index: 802