Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4358 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.9526. Min Act: -0.0000

Data Index: 388350 (C4 (Web Text))

Max Activating Token Index: 734

Text #1

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.8829. Min Act: -0.0000

Data Index: 991513 (C4 (Web Text))

Max Activating Token Index: 597

Text #2

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.7350. Min Act: -0.0000

Data Index: 920751 (C4 (Web Text))

Max Activating Token Index: 459

Text #3

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.7304. Min Act: -0.0000

Data Index: 972865 (C4 (Web Text))

Max Activating Token Index: 23

Text #4

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.7213. Min Act: -0.0000

Data Index: 366088 (C4 (Web Text))

Max Activating Token Index: 139

Text #5

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.7062. Min Act: -0.0000

Data Index: 41522 (C4 (Web Text))

Max Activating Token Index: 438

Text #6

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.7253. Min Act: -0.0000

Data Index: 280699 (C4 (Web Text))

Max Activating Token Index: 853

Text #7

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.6651. Min Act: -0.0000

Data Index: 1158524 (C4 (Web Text))

Max Activating Token Index: 885

Text #8

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.7299. Min Act: -0.0000

Data Index: 786818 (C4 (Web Text))

Max Activating Token Index: 283

Text #9

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.6825. Min Act: -0.0000

Data Index: 1008931 (C4 (Web Text))

Max Activating Token Index: 116

Text #10

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.6839. Min Act: -0.0000

Data Index: 998821 (C4 (Web Text))

Max Activating Token Index: 519

Text #11

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.6828. Min Act: -0.0000

Data Index: 287479 (C4 (Web Text))

Max Activating Token Index: 22

Text #12

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.6750. Min Act: -0.0000

Data Index: 368150 (C4 (Web Text))

Max Activating Token Index: 434

Text #13

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.6509. Min Act: -0.0000

Data Index: 467876 (C4 (Web Text))

Max Activating Token Index: 598

Text #14

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.6679. Min Act: -0.0000

Data Index: 484137 (C4 (Web Text))

Max Activating Token Index: 39

Text #15

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.6575. Min Act: -0.0000

Data Index: 917427 (C4 (Web Text))

Max Activating Token Index: 416

Text #16

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.6574. Min Act: -0.0000

Data Index: 1132799 (C4 (Web Text))

Max Activating Token Index: 99

Text #17

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.6535. Min Act: -0.0000

Data Index: 1077954 (C4 (Web Text))

Max Activating Token Index: 106

Text #18

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.6540. Min Act: -0.0000

Data Index: 764331 (C4 (Web Text))

Max Activating Token Index: 1020

Text #19

Max Range: 0.9526. Min Range: -0.9526

Max Act: 0.6300. Min Act: -0.0000

Data Index: 652721 (C4 (Web Text))

Max Activating Token Index: 493