Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 53 in Layer 2

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5457. Min Act: -0.0002

Data Index: 1462072 (Python Code)

Max Activating Token Index: 108

Text #1

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5325. Min Act: -0.0002

Data Index: 1389208 (Python Code)

Max Activating Token Index: 365

Text #2

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5434. Min Act: -0.0002

Data Index: 1509441 (Python Code)

Max Activating Token Index: 293

Text #3

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5255. Min Act: -0.0002

Data Index: 1478832 (Python Code)

Max Activating Token Index: 12

Text #4

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.4931. Min Act: -0.0002

Data Index: 1373084 (Python Code)

Max Activating Token Index: 545

Text #5

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.6281. Min Act: -0.0002

Data Index: 1508406 (Python Code)

Max Activating Token Index: 173

Text #6

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5646. Min Act: -0.0002

Data Index: 1431975 (Python Code)

Max Activating Token Index: 172

Text #7

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5389. Min Act: -0.0002

Data Index: 1416053 (Python Code)

Max Activating Token Index: 109

Text #8

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5795. Min Act: -0.0002

Data Index: 1607938 (Python Code)

Max Activating Token Index: 104

Text #9

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.4524. Min Act: -0.0002

Data Index: 1364428 (Python Code)

Max Activating Token Index: 182

Text #10

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.4921. Min Act: -0.0002

Data Index: 1463718 (Python Code)

Max Activating Token Index: 178

Text #11

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.4973. Min Act: -0.0002

Data Index: 1602861 (Python Code)

Max Activating Token Index: 129

Text #12

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5311. Min Act: -0.0002

Data Index: 1579969 (Python Code)

Max Activating Token Index: 348

Text #13

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5854. Min Act: -0.0002

Data Index: 1553442 (Python Code)

Max Activating Token Index: 102

Text #14

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5179. Min Act: -0.0002

Data Index: 1417544 (Python Code)

Max Activating Token Index: 237

Text #15

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5413. Min Act: -0.0002

Data Index: 1633632 (Python Code)

Max Activating Token Index: 251

Text #16

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5624. Min Act: -0.0002

Data Index: 1456692 (Python Code)

Max Activating Token Index: 210

Text #17

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.4915. Min Act: -0.0002

Data Index: 1588594 (Python Code)

Max Activating Token Index: 274

Text #18

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5791. Min Act: -0.0002

Data Index: 1600676 (Python Code)

Max Activating Token Index: 102

Text #19

Max Range: 0.6281. Min Range: -0.6281

Max Act: 0.5176. Min Act: -0.0002

Data Index: 1574502 (Python Code)

Max Activating Token Index: 172