Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2996 in Layer 10

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.3151. Min Range: -2.3151

Max Act: 2.3151. Min Act: -0.0000

Data Index: 1199832 (C4 (Web Text))

Max Activating Token Index: 998

Text #1

Max Range: 2.3151. Min Range: -2.3151

Max Act: 2.1729. Min Act: -0.0000

Data Index: 1168250 (C4 (Web Text))

Max Activating Token Index: 925

Text #2

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.8267. Min Act: -0.0000

Data Index: 714058 (C4 (Web Text))

Max Activating Token Index: 589

Text #3

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.6263. Min Act: -0.0000

Data Index: 1485999 (Python Code)

Max Activating Token Index: 215

Text #4

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.7110. Min Act: -0.0000

Data Index: 1358991 (C4 (Web Text))

Max Activating Token Index: 279

Text #5

Max Range: 2.3151. Min Range: -2.3151

Max Act: 2.0274. Min Act: -0.0000

Data Index: 1616486 (Python Code)

Max Activating Token Index: 761

Text #6

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.4247. Min Act: -0.0000

Data Index: 1069982 (C4 (Web Text))

Max Activating Token Index: 671

Text #7

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.5813. Min Act: -0.0000

Data Index: 370680 (C4 (Web Text))

Max Activating Token Index: 892

Text #8

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.5270. Min Act: -0.0000

Data Index: 716624 (C4 (Web Text))

Max Activating Token Index: 1012

Text #9

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.4051. Min Act: -0.0000

Data Index: 1288285 (C4 (Web Text))

Max Activating Token Index: 229

Text #10

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.4253. Min Act: -0.0000

Data Index: 276610 (C4 (Web Text))

Max Activating Token Index: 165

Text #11

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.3410. Min Act: -0.0000

Data Index: 351469 (C4 (Web Text))

Max Activating Token Index: 939

Text #12

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.4134. Min Act: -0.0000

Data Index: 77746 (C4 (Web Text))

Max Activating Token Index: 1013

Text #13

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.4600. Min Act: -0.0000

Data Index: 115163 (C4 (Web Text))

Max Activating Token Index: 969

Text #14

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.3997. Min Act: -0.0000

Data Index: 1093863 (C4 (Web Text))

Max Activating Token Index: 631

Text #15

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.3898. Min Act: -0.0000

Data Index: 337867 (C4 (Web Text))

Max Activating Token Index: 640

Text #16

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.3850. Min Act: -0.0000

Data Index: 22893 (C4 (Web Text))

Max Activating Token Index: 400

Text #17

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.3767. Min Act: -0.0000

Data Index: 361072 (C4 (Web Text))

Max Activating Token Index: 250

Text #18

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.4486. Min Act: -0.0000

Data Index: 810967 (C4 (Web Text))

Max Activating Token Index: 328

Text #19

Max Range: 2.3151. Min Range: -2.3151

Max Act: 1.3001. Min Act: -0.0000

Data Index: 1097298 (C4 (Web Text))

Max Activating Token Index: 872