Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1732 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.7719. Min Act: -0.0000

Data Index: 469579 (C4 (Web Text))

Max Activating Token Index: 858

Text #1

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.8140. Min Act: -0.0000

Data Index: 728855 (C4 (Web Text))

Max Activating Token Index: 216

Text #2

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.8187. Min Act: -0.0000

Data Index: 613088 (C4 (Web Text))

Max Activating Token Index: 338

Text #3

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.6541. Min Act: -0.0000

Data Index: 1306331 (C4 (Web Text))

Max Activating Token Index: 412

Text #4

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.6151. Min Act: -0.0000

Data Index: 797351 (C4 (Web Text))

Max Activating Token Index: 211

Text #5

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.6302. Min Act: -0.0000

Data Index: 420841 (C4 (Web Text))

Max Activating Token Index: 889

Text #6

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.5705. Min Act: -0.0000

Data Index: 443760 (C4 (Web Text))

Max Activating Token Index: 949

Text #7

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.5663. Min Act: -0.0000

Data Index: 1356610 (C4 (Web Text))

Max Activating Token Index: 561

Text #8

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.5600. Min Act: -0.0000

Data Index: 837994 (C4 (Web Text))

Max Activating Token Index: 23

Text #9

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.5226. Min Act: -0.0000

Data Index: 178810 (C4 (Web Text))

Max Activating Token Index: 20

Text #10

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.5361. Min Act: -0.0000

Data Index: 40575 (C4 (Web Text))

Max Activating Token Index: 308

Text #11

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.4740. Min Act: -0.0000

Data Index: 760555 (C4 (Web Text))

Max Activating Token Index: 211

Text #12

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.5276. Min Act: -0.0000

Data Index: 1140757 (C4 (Web Text))

Max Activating Token Index: 330

Text #13

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.5092. Min Act: -0.0000

Data Index: 360147 (C4 (Web Text))

Max Activating Token Index: 331

Text #14

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.4855. Min Act: -0.0000

Data Index: 1228927 (C4 (Web Text))

Max Activating Token Index: 304

Text #15

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.5353. Min Act: -0.0000

Data Index: 664508 (C4 (Web Text))

Max Activating Token Index: 360

Text #16

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.5042. Min Act: -0.0000

Data Index: 1068678 (C4 (Web Text))

Max Activating Token Index: 486

Text #17

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.5003. Min Act: -0.0000

Data Index: 212670 (C4 (Web Text))

Max Activating Token Index: 561

Text #18

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.4998. Min Act: -0.0000

Data Index: 588752 (C4 (Web Text))

Max Activating Token Index: 720

Text #19

Max Range: 0.8187. Min Range: -0.8187

Max Act: 0.4780. Min Act: -0.0000

Data Index: 913411 (C4 (Web Text))

Max Activating Token Index: 772