Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3035 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.3005. Min Range: -2.3005

Max Act: 2.3005. Min Act: -0.0000

Data Index: 461081 (C4 (Web Text))

Max Activating Token Index: 880

Text #1

Max Range: 2.3005. Min Range: -2.3005

Max Act: 2.0779. Min Act: -0.0000

Data Index: 1002599 (C4 (Web Text))

Max Activating Token Index: 962

Text #2

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.9832. Min Act: -0.0000

Data Index: 937449 (C4 (Web Text))

Max Activating Token Index: 617

Text #3

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.9698. Min Act: -0.0000

Data Index: 1120628 (C4 (Web Text))

Max Activating Token Index: 972

Text #4

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.8543. Min Act: -0.0000

Data Index: 574053 (C4 (Web Text))

Max Activating Token Index: 83

Text #5

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.9266. Min Act: -0.0000

Data Index: 1099733 (C4 (Web Text))

Max Activating Token Index: 884

Text #6

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.8018. Min Act: -0.0000

Data Index: 7833 (C4 (Web Text))

Max Activating Token Index: 115

Text #7

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.8201. Min Act: -0.0000

Data Index: 870206 (C4 (Web Text))

Max Activating Token Index: 168

Text #8

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.6847. Min Act: -0.0000

Data Index: 951697 (C4 (Web Text))

Max Activating Token Index: 532

Text #9

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.8053. Min Act: -0.0000

Data Index: 1231219 (C4 (Web Text))

Max Activating Token Index: 977

Text #10

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.7663. Min Act: -0.0000

Data Index: 1358550 (C4 (Web Text))

Max Activating Token Index: 32

Text #11

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.6508. Min Act: -0.0000

Data Index: 274801 (C4 (Web Text))

Max Activating Token Index: 593

Text #12

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.7310. Min Act: -0.0000

Data Index: 1043316 (C4 (Web Text))

Max Activating Token Index: 854

Text #13

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.7135. Min Act: -0.0000

Data Index: 939096 (C4 (Web Text))

Max Activating Token Index: 116

Text #14

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.7024. Min Act: -0.0000

Data Index: 771963 (C4 (Web Text))

Max Activating Token Index: 971

Text #15

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.6961. Min Act: -0.0000

Data Index: 574744 (C4 (Web Text))

Max Activating Token Index: 153

Text #16

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.7694. Min Act: -0.0000

Data Index: 1076391 (C4 (Web Text))

Max Activating Token Index: 499

Text #17

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.7842. Min Act: -0.0000

Data Index: 135553 (C4 (Web Text))

Max Activating Token Index: 833

Text #18

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.6704. Min Act: -0.0000

Data Index: 1135792 (C4 (Web Text))

Max Activating Token Index: 217

Text #19

Max Range: 2.3005. Min Range: -2.3005

Max Act: 1.6343. Min Act: -0.0000

Data Index: 704728 (C4 (Web Text))

Max Activating Token Index: 528