Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 948 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1567. Min Range: -1.1567

Max Act: 1.1567. Min Act: -0.0001

Data Index: 23388 (C4 (Web Text))

Max Activating Token Index: 379

Text #1

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.7911. Min Act: -0.0001

Data Index: 587133 (C4 (Web Text))

Max Activating Token Index: 139

Text #2

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.8848. Min Act: -0.0001

Data Index: 1269558 (C4 (Web Text))

Max Activating Token Index: 319

Text #3

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.8099. Min Act: -0.0001

Data Index: 564737 (C4 (Web Text))

Max Activating Token Index: 600

Text #4

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.6706. Min Act: -0.0001

Data Index: 1079959 (C4 (Web Text))

Max Activating Token Index: 279

Text #5

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.7390. Min Act: -0.0001

Data Index: 609272 (C4 (Web Text))

Max Activating Token Index: 571

Text #6

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.6356. Min Act: -0.0001

Data Index: 1344853 (C4 (Web Text))

Max Activating Token Index: 227

Text #7

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.7143. Min Act: -0.0001

Data Index: 1059752 (C4 (Web Text))

Max Activating Token Index: 721

Text #8

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.6361. Min Act: -0.0001

Data Index: 411993 (C4 (Web Text))

Max Activating Token Index: 852

Text #9

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.6904. Min Act: -0.0001

Data Index: 346762 (C4 (Web Text))

Max Activating Token Index: 726

Text #10

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.6208. Min Act: -0.0001

Data Index: 241424 (C4 (Web Text))

Max Activating Token Index: 744

Text #11

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.5854. Min Act: -0.0001

Data Index: 392282 (C4 (Web Text))

Max Activating Token Index: 671

Text #12

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.6735. Min Act: -0.0001

Data Index: 1115737 (C4 (Web Text))

Max Activating Token Index: 869

Text #13

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.7270. Min Act: -0.0001

Data Index: 1314623 (C4 (Web Text))

Max Activating Token Index: 677

Text #14

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.5916. Min Act: -0.0001

Data Index: 863514 (C4 (Web Text))

Max Activating Token Index: 595

Text #15

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.5995. Min Act: -0.0001

Data Index: 971473 (C4 (Web Text))

Max Activating Token Index: 761

Text #16

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.5414. Min Act: -0.0001

Data Index: 257353 (C4 (Web Text))

Max Activating Token Index: 826

Text #17

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.6557. Min Act: -0.0001

Data Index: 1160045 (C4 (Web Text))

Max Activating Token Index: 208

Text #18

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.5378. Min Act: -0.0001

Data Index: 597064 (C4 (Web Text))

Max Activating Token Index: 168

Text #19

Max Range: 1.1567. Min Range: -1.1567

Max Act: 0.6267. Min Act: -0.0001

Data Index: 1189485 (C4 (Web Text))

Max Activating Token Index: 173