Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5923 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 4.0147. Min Range: -4.0147

Max Act: 4.0147. Min Act: -0.0000

Data Index: 1089907 (C4 (Web Text))

Max Activating Token Index: 377

Text #1

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.5497. Min Act: -0.0000

Data Index: 437146 (C4 (Web Text))

Max Activating Token Index: 154

Text #2

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.7172. Min Act: -0.0000

Data Index: 506107 (C4 (Web Text))

Max Activating Token Index: 138

Text #3

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.5984. Min Act: -0.0000

Data Index: 344127 (C4 (Web Text))

Max Activating Token Index: 92

Text #4

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.7378. Min Act: -0.0000

Data Index: 801184 (C4 (Web Text))

Max Activating Token Index: 117

Text #5

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.3889. Min Act: -0.0000

Data Index: 423705 (C4 (Web Text))

Max Activating Token Index: 186

Text #6

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.5292. Min Act: -0.0000

Data Index: 1056803 (C4 (Web Text))

Max Activating Token Index: 673

Text #7

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.5624. Min Act: -0.0000

Data Index: 240818 (C4 (Web Text))

Max Activating Token Index: 286

Text #8

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.6561. Min Act: -0.0000

Data Index: 696995 (C4 (Web Text))

Max Activating Token Index: 175

Text #9

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.4225. Min Act: -0.0000

Data Index: 499037 (C4 (Web Text))

Max Activating Token Index: 153

Text #10

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.3286. Min Act: -0.0000

Data Index: 392206 (C4 (Web Text))

Max Activating Token Index: 559

Text #11

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.4533. Min Act: -0.0000

Data Index: 139375 (C4 (Web Text))

Max Activating Token Index: 118

Text #12

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.2707. Min Act: -0.0000

Data Index: 1252940 (C4 (Web Text))

Max Activating Token Index: 483

Text #13

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.2510. Min Act: -0.0000

Data Index: 781500 (C4 (Web Text))

Max Activating Token Index: 312

Text #14

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.3791. Min Act: -0.0000

Data Index: 196179 (C4 (Web Text))

Max Activating Token Index: 889

Text #15

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.1909. Min Act: -0.0000

Data Index: 66094 (C4 (Web Text))

Max Activating Token Index: 905

Text #16

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.3757. Min Act: -0.0000

Data Index: 122532 (C4 (Web Text))

Max Activating Token Index: 901

Text #17

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.2052. Min Act: -0.0000

Data Index: 1329686 (C4 (Web Text))

Max Activating Token Index: 65

Text #18

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.1996. Min Act: -0.0000

Data Index: 712865 (C4 (Web Text))

Max Activating Token Index: 593

Text #19

Max Range: 4.0147. Min Range: -4.0147

Max Act: 3.1790. Min Act: -0.0000

Data Index: 191596 (C4 (Web Text))

Max Activating Token Index: 610