Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2154 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.7371. Min Act: -0.0001

Data Index: 1423169 (Python Code)

Max Activating Token Index: 309

Text #1

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.6717. Min Act: -0.0001

Data Index: 1613140 (Python Code)

Max Activating Token Index: 270

Text #2

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.7285. Min Act: -0.0001

Data Index: 1654203 (Python Code)

Max Activating Token Index: 74

Text #3

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.7788. Min Act: -0.0001

Data Index: 330430 (C4 (Web Text))

Max Activating Token Index: 685

Text #4

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.7438. Min Act: -0.0001

Data Index: 1573324 (Python Code)

Max Activating Token Index: 320

Text #5

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.7821. Min Act: -0.0001

Data Index: 1575502 (Python Code)

Max Activating Token Index: 188

Text #6

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.7175. Min Act: -0.0001

Data Index: 1481125 (Python Code)

Max Activating Token Index: 6

Text #7

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.6920. Min Act: -0.0001

Data Index: 1414252 (Python Code)

Max Activating Token Index: 615

Text #8

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.7080. Min Act: -0.0001

Data Index: 1506034 (Python Code)

Max Activating Token Index: 982

Text #9

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.6421. Min Act: -0.0001

Data Index: 1608500 (Python Code)

Max Activating Token Index: 112

Text #10

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.6452. Min Act: -0.0001

Data Index: 1545552 (Python Code)

Max Activating Token Index: 431

Text #11

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.6432. Min Act: -0.0001

Data Index: 1522484 (Python Code)

Max Activating Token Index: 450

Text #12

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.6792. Min Act: -0.0001

Data Index: 1624123 (Python Code)

Max Activating Token Index: 22

Text #13

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.6004. Min Act: -0.0001

Data Index: 1626497 (Python Code)

Max Activating Token Index: 314

Text #14

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.6146. Min Act: -0.0001

Data Index: 1503479 (Python Code)

Max Activating Token Index: 18

Text #15

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.6330. Min Act: -0.0001

Data Index: 1563603 (Python Code)

Max Activating Token Index: 150

Text #16

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.4954. Min Act: -0.0001

Data Index: 1371101 (Python Code)

Max Activating Token Index: 261

Text #17

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.5576. Min Act: -0.0001

Data Index: 1500920 (Python Code)

Max Activating Token Index: 583

Text #18

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.6275. Min Act: -0.0001

Data Index: 1374276 (Python Code)

Max Activating Token Index: 146

Text #19

Max Range: 0.7821. Min Range: -0.7821

Max Act: 0.6194. Min Act: -0.0001

Data Index: 1636451 (Python Code)

Max Activating Token Index: 263