Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 82 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.8236. Min Act: -0.0000

Data Index: 654336 (C4 (Web Text))

Max Activating Token Index: 148

Text #1

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.9448. Min Act: -0.0000

Data Index: 517766 (C4 (Web Text))

Max Activating Token Index: 948

Text #2

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.8097. Min Act: -0.0000

Data Index: 797923 (C4 (Web Text))

Max Activating Token Index: 209

Text #3

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7730. Min Act: -0.0000

Data Index: 304091 (C4 (Web Text))

Max Activating Token Index: 679

Text #4

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7831. Min Act: -0.0000

Data Index: 899921 (C4 (Web Text))

Max Activating Token Index: 424

Text #5

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.6995. Min Act: -0.0000

Data Index: 1320440 (C4 (Web Text))

Max Activating Token Index: 303

Text #6

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7689. Min Act: -0.0000

Data Index: 1125349 (C4 (Web Text))

Max Activating Token Index: 622

Text #7

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7613. Min Act: -0.0000

Data Index: 729582 (C4 (Web Text))

Max Activating Token Index: 758

Text #8

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7870. Min Act: -0.0000

Data Index: 1017998 (C4 (Web Text))

Max Activating Token Index: 372

Text #9

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7528. Min Act: -0.0000

Data Index: 1145670 (C4 (Web Text))

Max Activating Token Index: 391

Text #10

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.6553. Min Act: -0.0000

Data Index: 115696 (C4 (Web Text))

Max Activating Token Index: 846

Text #11

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7187. Min Act: -0.0000

Data Index: 1162960 (C4 (Web Text))

Max Activating Token Index: 228

Text #12

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7158. Min Act: -0.0000

Data Index: 866279 (C4 (Web Text))

Max Activating Token Index: 19

Text #13

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.6893. Min Act: -0.0000

Data Index: 717173 (C4 (Web Text))

Max Activating Token Index: 751

Text #14

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7116. Min Act: -0.0000

Data Index: 601158 (C4 (Web Text))

Max Activating Token Index: 265

Text #15

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.6107. Min Act: -0.0000

Data Index: 1125920 (C4 (Web Text))

Max Activating Token Index: 723

Text #16

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7560. Min Act: -0.0000

Data Index: 396468 (C4 (Web Text))

Max Activating Token Index: 379

Text #17

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7734. Min Act: -0.0000

Data Index: 577665 (C4 (Web Text))

Max Activating Token Index: 806

Text #18

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7076. Min Act: -0.0000

Data Index: 1342606 (C4 (Web Text))

Max Activating Token Index: 132

Text #19

Max Range: 0.9448. Min Range: -0.9448

Max Act: 0.7246. Min Act: -0.0000

Data Index: 145192 (C4 (Web Text))

Max Activating Token Index: 642