Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1433 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 1.0533. Min Range: -1.0533

Max Act: 1.0533. Min Act: -0.0000

Data Index: 1565182 (Python Code)

Max Activating Token Index: 77

Text #1

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.9446. Min Act: -0.0000

Data Index: 1528706 (Python Code)

Max Activating Token Index: 754

Text #2

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.9190. Min Act: -0.0000

Data Index: 1558484 (Python Code)

Max Activating Token Index: 28

Text #3

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.8644. Min Act: -0.0000

Data Index: 1518508 (Python Code)

Max Activating Token Index: 284

Text #4

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.7822. Min Act: -0.0000

Data Index: 1563158 (Python Code)

Max Activating Token Index: 833

Text #5

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.7545. Min Act: -0.0000

Data Index: 1379030 (Python Code)

Max Activating Token Index: 22

Text #6

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.6853. Min Act: -0.0000

Data Index: 1445578 (Python Code)

Max Activating Token Index: 1012

Text #7

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.6514. Min Act: -0.0000

Data Index: 1454782 (Python Code)

Max Activating Token Index: 858

Text #8

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.6947. Min Act: -0.0000

Data Index: 1399053 (Python Code)

Max Activating Token Index: 243

Text #9

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.6929. Min Act: -0.0000

Data Index: 1581297 (Python Code)

Max Activating Token Index: 289

Text #10

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.6916. Min Act: -0.0000

Data Index: 1444348 (Python Code)

Max Activating Token Index: 253

Text #11

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.6086. Min Act: -0.0000

Data Index: 1627110 (Python Code)

Max Activating Token Index: 737

Text #12

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.6430. Min Act: -0.0000

Data Index: 1360399 (Python Code)

Max Activating Token Index: 170

Text #13

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.7017. Min Act: -0.0000

Data Index: 1401983 (Python Code)

Max Activating Token Index: 459

Text #14

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.6576. Min Act: -0.0000

Data Index: 1535043 (Python Code)

Max Activating Token Index: 856

Text #15

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.6449. Min Act: -0.0000

Data Index: 1588260 (Python Code)

Max Activating Token Index: 873

Text #16

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.6251. Min Act: -0.0000

Data Index: 1645762 (Python Code)

Max Activating Token Index: 276

Text #17

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.6629. Min Act: -0.0000

Data Index: 1411635 (Python Code)

Max Activating Token Index: 843

Text #18

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.6362. Min Act: -0.0000

Data Index: 1545382 (Python Code)

Max Activating Token Index: 458

Text #19

Max Range: 1.0533. Min Range: -1.0533

Max Act: 0.5959. Min Act: -0.0000

Data Index: 1595957 (Python Code)

Max Activating Token Index: 299