Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 421 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.8230. Min Act: -0.0000

Data Index: 366287 (C4 (Web Text))

Max Activating Token Index: 846

Text #1

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.7059. Min Act: -0.0000

Data Index: 1390018 (Python Code)

Max Activating Token Index: 243

Text #2

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.6818. Min Act: -0.0000

Data Index: 1556869 (Python Code)

Max Activating Token Index: 351

Text #3

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.6615. Min Act: -0.0000

Data Index: 1614731 (Python Code)

Max Activating Token Index: 97

Text #4

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.6262. Min Act: -0.0000

Data Index: 1442003 (Python Code)

Max Activating Token Index: 264

Text #5

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.6354. Min Act: -0.0000

Data Index: 1596469 (Python Code)

Max Activating Token Index: 218

Text #6

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.5041. Min Act: -0.0000

Data Index: 1618910 (Python Code)

Max Activating Token Index: 203

Text #7

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.5778. Min Act: -0.0000

Data Index: 1648167 (Python Code)

Max Activating Token Index: 283

Text #8

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.6655. Min Act: -0.0000

Data Index: 1572997 (Python Code)

Max Activating Token Index: 102

Text #9

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.6667. Min Act: -0.0000

Data Index: 1031627 (C4 (Web Text))

Max Activating Token Index: 342

Text #10

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.5489. Min Act: -0.0000

Data Index: 1426575 (Python Code)

Max Activating Token Index: 216

Text #11

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.5009. Min Act: -0.0000

Data Index: 1646297 (Python Code)

Max Activating Token Index: 167

Text #12

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.5166. Min Act: -0.0000

Data Index: 1571135 (Python Code)

Max Activating Token Index: 54

Text #13

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.4385. Min Act: -0.0000

Data Index: 1518124 (Python Code)

Max Activating Token Index: 205

Text #14

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.5055. Min Act: -0.0000

Data Index: 1443329 (Python Code)

Max Activating Token Index: 187

Text #15

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.5090. Min Act: -0.0000

Data Index: 1632273 (Python Code)

Max Activating Token Index: 328

Text #16

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.5371. Min Act: -0.0000

Data Index: 1144023 (C4 (Web Text))

Max Activating Token Index: 871

Text #17

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.4963. Min Act: -0.0000

Data Index: 1420857 (Python Code)

Max Activating Token Index: 157

Text #18

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.5367. Min Act: -0.0000

Data Index: 1649478 (Python Code)

Max Activating Token Index: 124

Text #19

Max Range: 1.8230. Min Range: -1.8230

Max Act: 1.4864. Min Act: -0.0000

Data Index: 1585179 (Python Code)

Max Activating Token Index: 361