Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2174 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.4773. Min Act: -0.0001

Data Index: 1147551 (C4 (Web Text))

Max Activating Token Index: 821

Text #1

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.4353. Min Act: -0.0001

Data Index: 369376 (C4 (Web Text))

Max Activating Token Index: 511

Text #2

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.3153. Min Act: -0.0001

Data Index: 321046 (C4 (Web Text))

Max Activating Token Index: 424

Text #3

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.3891. Min Act: -0.0001

Data Index: 247040 (C4 (Web Text))

Max Activating Token Index: 983

Text #4

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.3446. Min Act: -0.0001

Data Index: 401572 (C4 (Web Text))

Max Activating Token Index: 196

Text #5

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.2662. Min Act: -0.0001

Data Index: 939209 (C4 (Web Text))

Max Activating Token Index: 34

Text #6

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.2829. Min Act: -0.0001

Data Index: 1299591 (C4 (Web Text))

Max Activating Token Index: 276

Text #7

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.1464. Min Act: -0.0001

Data Index: 610606 (C4 (Web Text))

Max Activating Token Index: 987

Text #8

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.2019. Min Act: -0.0001

Data Index: 992020 (C4 (Web Text))

Max Activating Token Index: 904

Text #9

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.0758. Min Act: -0.0001

Data Index: 1015711 (C4 (Web Text))

Max Activating Token Index: 89

Text #10

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.2589. Min Act: -0.0001

Data Index: 127073 (C4 (Web Text))

Max Activating Token Index: 967

Text #11

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.2209. Min Act: -0.0001

Data Index: 471463 (C4 (Web Text))

Max Activating Token Index: 406

Text #12

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.0260. Min Act: -0.0001

Data Index: 386066 (C4 (Web Text))

Max Activating Token Index: 849

Text #13

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.1772. Min Act: -0.0001

Data Index: 727524 (C4 (Web Text))

Max Activating Token Index: 940

Text #14

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.1099. Min Act: -0.0001

Data Index: 814001 (C4 (Web Text))

Max Activating Token Index: 277

Text #15

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.2262. Min Act: -0.0001

Data Index: 219706 (C4 (Web Text))

Max Activating Token Index: 201

Text #16

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.0774. Min Act: -0.0001

Data Index: 870684 (C4 (Web Text))

Max Activating Token Index: 207

Text #17

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.0830. Min Act: -0.0001

Data Index: 607810 (C4 (Web Text))

Max Activating Token Index: 674

Text #18

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.1443. Min Act: -0.0001

Data Index: 461909 (C4 (Web Text))

Max Activating Token Index: 679

Text #19

Max Range: 1.4773. Min Range: -1.4773

Max Act: 1.1361. Min Act: -0.0001

Data Index: 52459 (C4 (Web Text))

Max Activating Token Index: 452