Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2054 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.7088. Min Act: -0.0000

Data Index: 980893 (C4 (Web Text))

Max Activating Token Index: 84

Text #1

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.6861. Min Act: -0.0000

Data Index: 732675 (C4 (Web Text))

Max Activating Token Index: 520

Text #2

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.7153. Min Act: -0.0000

Data Index: 557291 (C4 (Web Text))

Max Activating Token Index: 909

Text #3

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.6420. Min Act: -0.0000

Data Index: 1106325 (C4 (Web Text))

Max Activating Token Index: 917

Text #4

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.6349. Min Act: -0.0000

Data Index: 1305323 (C4 (Web Text))

Max Activating Token Index: 18

Text #5

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.6735. Min Act: -0.0000

Data Index: 1591554 (Python Code)

Max Activating Token Index: 981

Text #6

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.6301. Min Act: -0.0000

Data Index: 1612394 (Python Code)

Max Activating Token Index: 306

Text #7

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.5686. Min Act: -0.0000

Data Index: 1487629 (Python Code)

Max Activating Token Index: 444

Text #8

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.6229. Min Act: -0.0000

Data Index: 43750 (C4 (Web Text))

Max Activating Token Index: 1010

Text #9

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.5726. Min Act: -0.0000

Data Index: 822016 (C4 (Web Text))

Max Activating Token Index: 688

Text #10

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.5625. Min Act: -0.0000

Data Index: 1327057 (C4 (Web Text))

Max Activating Token Index: 356

Text #11

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.5555. Min Act: -0.0000

Data Index: 337344 (C4 (Web Text))

Max Activating Token Index: 58

Text #12

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.5869. Min Act: -0.0000

Data Index: 421695 (C4 (Web Text))

Max Activating Token Index: 774

Text #13

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.5781. Min Act: -0.0000

Data Index: 1558786 (Python Code)

Max Activating Token Index: 631

Text #14

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.5557. Min Act: -0.0000

Data Index: 1315581 (C4 (Web Text))

Max Activating Token Index: 446

Text #15

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.5581. Min Act: -0.0000

Data Index: 520488 (C4 (Web Text))

Max Activating Token Index: 496

Text #16

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.5667. Min Act: -0.0000

Data Index: 843746 (C4 (Web Text))

Max Activating Token Index: 617

Text #17

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.5696. Min Act: -0.0000

Data Index: 1475715 (Python Code)

Max Activating Token Index: 389

Text #18

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.5420. Min Act: -0.0000

Data Index: 1166443 (C4 (Web Text))

Max Activating Token Index: 111

Text #19

Max Range: 0.7153. Min Range: -0.7153

Max Act: 0.5082. Min Act: -0.0000

Data Index: 54530 (C4 (Web Text))

Max Activating Token Index: 440