Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 5094 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.7132. Min Act: -0.0000

Data Index: 1137289 (C4 (Web Text))

Max Activating Token Index: 401

Text #1

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.3784. Min Act: -0.0000

Data Index: 1080810 (C4 (Web Text))

Max Activating Token Index: 57

Text #2

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.2448. Min Act: -0.0000

Data Index: 432798 (C4 (Web Text))

Max Activating Token Index: 297

Text #3

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.3152. Min Act: -0.0000

Data Index: 1098264 (C4 (Web Text))

Max Activating Token Index: 918

Text #4

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.2395. Min Act: -0.0000

Data Index: 242208 (C4 (Web Text))

Max Activating Token Index: 267

Text #5

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.3095. Min Act: -0.0000

Data Index: 1038234 (C4 (Web Text))

Max Activating Token Index: 134

Text #6

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.1636. Min Act: -0.0000

Data Index: 278275 (C4 (Web Text))

Max Activating Token Index: 66

Text #7

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.1120. Min Act: -0.0000

Data Index: 750462 (C4 (Web Text))

Max Activating Token Index: 792

Text #8

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.0818. Min Act: -0.0000

Data Index: 558577 (C4 (Web Text))

Max Activating Token Index: 387

Text #9

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.0994. Min Act: -0.0000

Data Index: 970356 (C4 (Web Text))

Max Activating Token Index: 872

Text #10

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.0837. Min Act: -0.0000

Data Index: 1022008 (C4 (Web Text))

Max Activating Token Index: 32

Text #11

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.0127. Min Act: -0.0000

Data Index: 346267 (C4 (Web Text))

Max Activating Token Index: 999

Text #12

Max Range: 1.7132. Min Range: -1.7132

Max Act: 0.9913. Min Act: -0.0000

Data Index: 375148 (C4 (Web Text))

Max Activating Token Index: 328

Text #13

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.0676. Min Act: -0.0000

Data Index: 934753 (C4 (Web Text))

Max Activating Token Index: 245

Text #14

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.0610. Min Act: -0.0000

Data Index: 655011 (C4 (Web Text))

Max Activating Token Index: 867

Text #15

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.0638. Min Act: -0.0000

Data Index: 158476 (C4 (Web Text))

Max Activating Token Index: 952

Text #16

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.0635. Min Act: -0.0000

Data Index: 101081 (C4 (Web Text))

Max Activating Token Index: 26

Text #17

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.0461. Min Act: -0.0000

Data Index: 256303 (C4 (Web Text))

Max Activating Token Index: 636

Text #18

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.1013. Min Act: -0.0000

Data Index: 380727 (C4 (Web Text))

Max Activating Token Index: 562

Text #19

Max Range: 1.7132. Min Range: -1.7132

Max Act: 1.0308. Min Act: -0.0000

Data Index: 951162 (C4 (Web Text))

Max Activating Token Index: 365