Model: SoLU Model: 4 Layers, 2048 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2036 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-4l')

Text #0

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.1150. Min Act: -0.0002

Data Index: 1628734 (Python Code)

Max Activating Token Index: 674

Text #1

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.1144. Min Act: -0.0002

Data Index: 1442037 (Python Code)

Max Activating Token Index: 97

Text #2

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.1116. Min Act: -0.0002

Data Index: 1433422 (Python Code)

Max Activating Token Index: 16

Text #3

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.1038. Min Act: -0.0002

Data Index: 1652257 (Python Code)

Max Activating Token Index: 249

Text #4

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.1076. Min Act: -0.0002

Data Index: 1391246 (Python Code)

Max Activating Token Index: 18

Text #5

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.1044. Min Act: -0.0002

Data Index: 1619748 (Python Code)

Max Activating Token Index: 246

Text #6

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.1070. Min Act: -0.0002

Data Index: 1436692 (Python Code)

Max Activating Token Index: 235

Text #7

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.1050. Min Act: -0.0002

Data Index: 1569397 (Python Code)

Max Activating Token Index: 66

Text #8

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.1029. Min Act: -0.0002

Data Index: 1646835 (Python Code)

Max Activating Token Index: 414

Text #9

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.1001. Min Act: -0.0002

Data Index: 1517948 (Python Code)

Max Activating Token Index: 245

Text #10

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.0966. Min Act: -0.0002

Data Index: 1561166 (Python Code)

Max Activating Token Index: 819

Text #11

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.0921. Min Act: -0.0002

Data Index: 1544719 (Python Code)

Max Activating Token Index: 997

Text #12

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.0936. Min Act: -0.0002

Data Index: 1364455 (Python Code)

Max Activating Token Index: 555

Text #13

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.0956. Min Act: -0.0002

Data Index: 1529612 (Python Code)

Max Activating Token Index: 411

Text #14

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.0937. Min Act: -0.0002

Data Index: 1614989 (Python Code)

Max Activating Token Index: 82

Text #15

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.0904. Min Act: -0.0002

Data Index: 1564970 (Python Code)

Max Activating Token Index: 200

Text #16

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.0929. Min Act: -0.0002

Data Index: 1477317 (Python Code)

Max Activating Token Index: 129

Text #17

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.0911. Min Act: -0.0002

Data Index: 1409325 (Python Code)

Max Activating Token Index: 351

Text #18

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.0918. Min Act: -0.0002

Data Index: 1547169 (Python Code)

Max Activating Token Index: 36

Text #19

Max Range: 0.1150. Min Range: -0.1150

Max Act: 0.0915. Min Act: -0.0002

Data Index: 1590731 (Python Code)

Max Activating Token Index: 70