Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4970 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4332. Min Range: -1.4332

Max Act: 1.4332. Min Act: -0.0001

Data Index: 388156 (C4 (Web Text))

Max Activating Token Index: 590

Text #1

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.9291. Min Act: -0.0001

Data Index: 1263115 (C4 (Web Text))

Max Activating Token Index: 144

Text #2

Max Range: 1.4332. Min Range: -1.4332

Max Act: 1.0859. Min Act: -0.0001

Data Index: 820463 (C4 (Web Text))

Max Activating Token Index: 775

Text #3

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.9176. Min Act: -0.0001

Data Index: 1150701 (C4 (Web Text))

Max Activating Token Index: 762

Text #4

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.8254. Min Act: -0.0001

Data Index: 387994 (C4 (Web Text))

Max Activating Token Index: 108

Text #5

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.8415. Min Act: -0.0001

Data Index: 1053709 (C4 (Web Text))

Max Activating Token Index: 475

Text #6

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.7821. Min Act: -0.0001

Data Index: 1301180 (C4 (Web Text))

Max Activating Token Index: 229

Text #7

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.8422. Min Act: -0.0001

Data Index: 78364 (C4 (Web Text))

Max Activating Token Index: 292

Text #8

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.8400. Min Act: -0.0001

Data Index: 737710 (C4 (Web Text))

Max Activating Token Index: 206

Text #9

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.8157. Min Act: -0.0001

Data Index: 1231355 (C4 (Web Text))

Max Activating Token Index: 307

Text #10

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.8730. Min Act: -0.0001

Data Index: 631171 (C4 (Web Text))

Max Activating Token Index: 753

Text #11

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.8045. Min Act: -0.0001

Data Index: 305168 (C4 (Web Text))

Max Activating Token Index: 508

Text #12

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.8674. Min Act: -0.0001

Data Index: 420127 (C4 (Web Text))

Max Activating Token Index: 835

Text #13

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.7231. Min Act: -0.0001

Data Index: 1235288 (C4 (Web Text))

Max Activating Token Index: 274

Text #14

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.7938. Min Act: -0.0001

Data Index: 82245 (C4 (Web Text))

Max Activating Token Index: 524

Text #15

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.8161. Min Act: -0.0001

Data Index: 1162249 (C4 (Web Text))

Max Activating Token Index: 451

Text #16

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.8415. Min Act: -0.0001

Data Index: 1080215 (C4 (Web Text))

Max Activating Token Index: 81

Text #17

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.7546. Min Act: -0.0001

Data Index: 610788 (C4 (Web Text))

Max Activating Token Index: 806

Text #18

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.7717. Min Act: -0.0001

Data Index: 286829 (C4 (Web Text))

Max Activating Token Index: 306

Text #19

Max Range: 1.4332. Min Range: -1.4332

Max Act: 0.7970. Min Act: -0.0001

Data Index: 799024 (C4 (Web Text))

Max Activating Token Index: 472