Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4818 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.1657. Min Act: -0.0001

Data Index: 1656873 (Python Code)

Max Activating Token Index: 612

Text #1

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.1440. Min Act: -0.0001

Data Index: 1590417 (Python Code)

Max Activating Token Index: 293

Text #2

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0083. Min Act: -0.0001

Data Index: 1493906 (Python Code)

Max Activating Token Index: 275

Text #3

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0718. Min Act: -0.0001

Data Index: 1653694 (Python Code)

Max Activating Token Index: 317

Text #4

Max Range: 1.1657. Min Range: -1.1657

Max Act: 0.9905. Min Act: -0.0001

Data Index: 1571728 (Python Code)

Max Activating Token Index: 191

Text #5

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0837. Min Act: -0.0001

Data Index: 1503481 (Python Code)

Max Activating Token Index: 120

Text #6

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0451. Min Act: -0.0001

Data Index: 1476777 (Python Code)

Max Activating Token Index: 475

Text #7

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0108. Min Act: -0.0001

Data Index: 1655950 (Python Code)

Max Activating Token Index: 612

Text #8

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0020. Min Act: -0.0001

Data Index: 1541739 (Python Code)

Max Activating Token Index: 153

Text #9

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0328. Min Act: -0.0001

Data Index: 1652012 (Python Code)

Max Activating Token Index: 160

Text #10

Max Range: 1.1657. Min Range: -1.1657

Max Act: 0.9612. Min Act: -0.0001

Data Index: 1372971 (Python Code)

Max Activating Token Index: 216

Text #11

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0851. Min Act: -0.0001

Data Index: 1620088 (Python Code)

Max Activating Token Index: 101

Text #12

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0141. Min Act: -0.0001

Data Index: 1519117 (Python Code)

Max Activating Token Index: 270

Text #13

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0168. Min Act: -0.0001

Data Index: 1399917 (Python Code)

Max Activating Token Index: 137

Text #14

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0362. Min Act: -0.0001

Data Index: 1378684 (Python Code)

Max Activating Token Index: 120

Text #15

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0284. Min Act: -0.0001

Data Index: 1538137 (Python Code)

Max Activating Token Index: 67

Text #16

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0074. Min Act: -0.0001

Data Index: 1401666 (Python Code)

Max Activating Token Index: 177

Text #17

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0204. Min Act: -0.0001

Data Index: 1467428 (Python Code)

Max Activating Token Index: 90

Text #18

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0572. Min Act: -0.0001

Data Index: 1597805 (Python Code)

Max Activating Token Index: 155

Text #19

Max Range: 1.1657. Min Range: -1.1657

Max Act: 1.0194. Min Act: -0.0001

Data Index: 1444695 (Python Code)

Max Activating Token Index: 99