Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3714 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.4198. Min Range: -1.4198

Max Act: 1.4198. Min Act: -0.0000

Data Index: 74853 (C4 (Web Text))

Max Activating Token Index: 1011

Text #1

Max Range: 1.4198. Min Range: -1.4198

Max Act: 1.2404. Min Act: -0.0000

Data Index: 1089178 (C4 (Web Text))

Max Activating Token Index: 977

Text #2

Max Range: 1.4198. Min Range: -1.4198

Max Act: 1.1647. Min Act: -0.0000

Data Index: 653380 (C4 (Web Text))

Max Activating Token Index: 573

Text #3

Max Range: 1.4198. Min Range: -1.4198

Max Act: 1.0512. Min Act: -0.0000

Data Index: 199071 (C4 (Web Text))

Max Activating Token Index: 721

Text #4

Max Range: 1.4198. Min Range: -1.4198

Max Act: 1.0059. Min Act: -0.0000

Data Index: 768392 (C4 (Web Text))

Max Activating Token Index: 595

Text #5

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.9673. Min Act: -0.0000

Data Index: 1199587 (C4 (Web Text))

Max Activating Token Index: 364

Text #6

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.9630. Min Act: -0.0000

Data Index: 1160855 (C4 (Web Text))

Max Activating Token Index: 651

Text #7

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.8720. Min Act: -0.0000

Data Index: 1166110 (C4 (Web Text))

Max Activating Token Index: 829

Text #8

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.8741. Min Act: -0.0000

Data Index: 1123236 (C4 (Web Text))

Max Activating Token Index: 813

Text #9

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.7958. Min Act: -0.0000

Data Index: 1636691 (Python Code)

Max Activating Token Index: 705

Text #10

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.8411. Min Act: -0.0000

Data Index: 500706 (C4 (Web Text))

Max Activating Token Index: 33

Text #11

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.8286. Min Act: -0.0000

Data Index: 206366 (C4 (Web Text))

Max Activating Token Index: 50

Text #12

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.8045. Min Act: -0.0000

Data Index: 626020 (C4 (Web Text))

Max Activating Token Index: 860

Text #13

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.8454. Min Act: -0.0000

Data Index: 1414689 (Python Code)

Max Activating Token Index: 502

Text #14

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.8041. Min Act: -0.0000

Data Index: 861537 (C4 (Web Text))

Max Activating Token Index: 311

Text #15

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.8035. Min Act: -0.0000

Data Index: 1237910 (C4 (Web Text))

Max Activating Token Index: 26

Text #16

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.7982. Min Act: -0.0000

Data Index: 11445 (C4 (Web Text))

Max Activating Token Index: 823

Text #17

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.8806. Min Act: -0.0000

Data Index: 603224 (C4 (Web Text))

Max Activating Token Index: 1020

Text #18

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.7735. Min Act: -0.0000

Data Index: 1276933 (C4 (Web Text))

Max Activating Token Index: 809

Text #19

Max Range: 1.4198. Min Range: -1.4198

Max Act: 0.7950. Min Act: -0.0000

Data Index: 400551 (C4 (Web Text))

Max Activating Token Index: 924