Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4817 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 3.5953. Min Range: -3.5953

Max Act: 3.4784. Min Act: -0.0000

Data Index: 220011 (C4 (Web Text))

Max Activating Token Index: 288

Text #1

Max Range: 3.5953. Min Range: -3.5953

Max Act: 3.5953. Min Act: -0.0000

Data Index: 437713 (C4 (Web Text))

Max Activating Token Index: 80

Text #2

Max Range: 3.5953. Min Range: -3.5953

Max Act: 3.5244. Min Act: -0.0000

Data Index: 958985 (C4 (Web Text))

Max Activating Token Index: 728

Text #3

Max Range: 3.5953. Min Range: -3.5953

Max Act: 2.7567. Min Act: -0.0000

Data Index: 726968 (C4 (Web Text))

Max Activating Token Index: 461

Text #4

Max Range: 3.5953. Min Range: -3.5953

Max Act: 2.4330. Min Act: -0.0000

Data Index: 103337 (C4 (Web Text))

Max Activating Token Index: 1020

Text #5

Max Range: 3.5953. Min Range: -3.5953

Max Act: 2.3485. Min Act: -0.0000

Data Index: 864602 (C4 (Web Text))

Max Activating Token Index: 452

Text #6

Max Range: 3.5953. Min Range: -3.5953

Max Act: 2.3713. Min Act: -0.0000

Data Index: 869054 (C4 (Web Text))

Max Activating Token Index: 558

Text #7

Max Range: 3.5953. Min Range: -3.5953

Max Act: 2.1035. Min Act: -0.0000

Data Index: 384832 (C4 (Web Text))

Max Activating Token Index: 457

Text #8

Max Range: 3.5953. Min Range: -3.5953

Max Act: 1.9383. Min Act: -0.0000

Data Index: 29047 (C4 (Web Text))

Max Activating Token Index: 142

Text #9

Max Range: 3.5953. Min Range: -3.5953

Max Act: 1.8497. Min Act: -0.0000

Data Index: 930369 (C4 (Web Text))

Max Activating Token Index: 51

Text #10

Max Range: 3.5953. Min Range: -3.5953

Max Act: 1.8250. Min Act: -0.0000

Data Index: 743130 (C4 (Web Text))

Max Activating Token Index: 479

Text #11

Max Range: 3.5953. Min Range: -3.5953

Max Act: 1.8071. Min Act: -0.0000

Data Index: 639451 (C4 (Web Text))

Max Activating Token Index: 629

Text #12

Max Range: 3.5953. Min Range: -3.5953

Max Act: 1.8173. Min Act: -0.0000

Data Index: 262982 (C4 (Web Text))

Max Activating Token Index: 386

Text #13

Max Range: 3.5953. Min Range: -3.5953

Max Act: 1.6669. Min Act: -0.0000

Data Index: 457904 (C4 (Web Text))

Max Activating Token Index: 1020

Text #14

Max Range: 3.5953. Min Range: -3.5953

Max Act: 1.7469. Min Act: -0.0000

Data Index: 1229801 (C4 (Web Text))

Max Activating Token Index: 441

Text #15

Max Range: 3.5953. Min Range: -3.5953

Max Act: 1.6533. Min Act: -0.0000

Data Index: 904069 (C4 (Web Text))

Max Activating Token Index: 851

Text #16

Max Range: 3.5953. Min Range: -3.5953

Max Act: 1.6591. Min Act: -0.0000

Data Index: 1350399 (C4 (Web Text))

Max Activating Token Index: 722

Text #17

Max Range: 3.5953. Min Range: -3.5953

Max Act: 1.6092. Min Act: -0.0000

Data Index: 130687 (C4 (Web Text))

Max Activating Token Index: 340

Text #18

Max Range: 3.5953. Min Range: -3.5953

Max Act: 1.5960. Min Act: -0.0000

Data Index: 268550 (C4 (Web Text))

Max Activating Token Index: 204

Text #19

Max Range: 3.5953. Min Range: -3.5953

Max Act: 1.5036. Min Act: -0.0000

Data Index: 258053 (C4 (Web Text))

Max Activating Token Index: 203