Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1250 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.5567. Min Act: -0.0000

Data Index: 365519 (C4 (Web Text))

Max Activating Token Index: 34

Text #1

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.4905. Min Act: -0.0000

Data Index: 916340 (C4 (Web Text))

Max Activating Token Index: 510

Text #2

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.4731. Min Act: -0.0000

Data Index: 338424 (C4 (Web Text))

Max Activating Token Index: 212

Text #3

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.4494. Min Act: -0.0000

Data Index: 178919 (C4 (Web Text))

Max Activating Token Index: 306

Text #4

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.4540. Min Act: -0.0000

Data Index: 998944 (C4 (Web Text))

Max Activating Token Index: 285

Text #5

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.4225. Min Act: -0.0000

Data Index: 981592 (C4 (Web Text))

Max Activating Token Index: 326

Text #6

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.4407. Min Act: -0.0000

Data Index: 1161464 (C4 (Web Text))

Max Activating Token Index: 584

Text #7

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.4070. Min Act: -0.0000

Data Index: 1190392 (C4 (Web Text))

Max Activating Token Index: 541

Text #8

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.4249. Min Act: -0.0000

Data Index: 738231 (C4 (Web Text))

Max Activating Token Index: 542

Text #9

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.4103. Min Act: -0.0000

Data Index: 954910 (C4 (Web Text))

Max Activating Token Index: 92

Text #10

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.4125. Min Act: -0.0000

Data Index: 391671 (C4 (Web Text))

Max Activating Token Index: 896

Text #11

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.3953. Min Act: -0.0000

Data Index: 291412 (C4 (Web Text))

Max Activating Token Index: 492

Text #12

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.3718. Min Act: -0.0000

Data Index: 635334 (C4 (Web Text))

Max Activating Token Index: 568

Text #13

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.3530. Min Act: -0.0000

Data Index: 1108809 (C4 (Web Text))

Max Activating Token Index: 993

Text #14

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.3404. Min Act: -0.0000

Data Index: 681165 (C4 (Web Text))

Max Activating Token Index: 377

Text #15

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.3778. Min Act: -0.0000

Data Index: 1355375 (C4 (Web Text))

Max Activating Token Index: 650

Text #16

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.3574. Min Act: -0.0000

Data Index: 382698 (C4 (Web Text))

Max Activating Token Index: 637

Text #17

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.3807. Min Act: -0.0000

Data Index: 136983 (C4 (Web Text))

Max Activating Token Index: 96

Text #18

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.3374. Min Act: -0.0000

Data Index: 263037 (C4 (Web Text))

Max Activating Token Index: 143

Text #19

Max Range: 0.5567. Min Range: -0.5567

Max Act: 0.3575. Min Act: -0.0000

Data Index: 396781 (C4 (Web Text))

Max Activating Token Index: 612