Model: SoLU Model: 3 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 333 in Layer 0

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-3l')

Text #0

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0252. Min Act: -0.0006

Data Index: 1498531 (Python Code)

Max Activating Token Index: 53

Text #1

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0252. Min Act: -0.0005

Data Index: 1485251 (Python Code)

Max Activating Token Index: 664

Text #2

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0253. Min Act: -0.0005

Data Index: 1618789 (Python Code)

Max Activating Token Index: 366

Text #3

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0252. Min Act: -0.0006

Data Index: 1647765 (Python Code)

Max Activating Token Index: 357

Text #4

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0256. Min Act: -0.0005

Data Index: 1390687 (Python Code)

Max Activating Token Index: 475

Text #5

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0249. Min Act: -0.0005

Data Index: 1394183 (Python Code)

Max Activating Token Index: 156

Text #6

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0243. Min Act: -0.0005

Data Index: 1421642 (Python Code)

Max Activating Token Index: 963

Text #7

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0248. Min Act: -0.0005

Data Index: 1362903 (Python Code)

Max Activating Token Index: 723

Text #8

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0248. Min Act: -0.0006

Data Index: 1576473 (Python Code)

Max Activating Token Index: 669

Text #9

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0254. Min Act: -0.0005

Data Index: 1392101 (Python Code)

Max Activating Token Index: 163

Text #10

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0248. Min Act: -0.0005

Data Index: 1646018 (Python Code)

Max Activating Token Index: 437

Text #11

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0247. Min Act: -0.0006

Data Index: 1633985 (Python Code)

Max Activating Token Index: 782

Text #12

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0252. Min Act: -0.0006

Data Index: 1576143 (Python Code)

Max Activating Token Index: 344

Text #13

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0241. Min Act: -0.0005

Data Index: 1441616 (Python Code)

Max Activating Token Index: 568

Text #14

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0246. Min Act: -0.0005

Data Index: 1651567 (Python Code)

Max Activating Token Index: 161

Text #15

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0246. Min Act: -0.0005

Data Index: 1436719 (Python Code)

Max Activating Token Index: 181

Text #16

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0245. Min Act: -0.0006

Data Index: 1381410 (Python Code)

Max Activating Token Index: 1005

Text #17

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0245. Min Act: -0.0005

Data Index: 1406295 (Python Code)

Max Activating Token Index: 39

Text #18

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0245. Min Act: -0.0005

Data Index: 1645008 (Python Code)

Max Activating Token Index: 192

Text #19

Max Range: 0.0256. Min Range: -0.0256

Max Act: 0.0244. Min Act: -0.0006

Data Index: 1488357 (Python Code)

Max Activating Token Index: 118