Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 723 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1536. Min Act: -0.0001

Data Index: 351034 (C4 (Web Text))

Max Activating Token Index: 54

Text #1

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1338. Min Act: -0.0001

Data Index: 478631 (C4 (Web Text))

Max Activating Token Index: 556

Text #2

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1321. Min Act: -0.0001

Data Index: 246503 (C4 (Web Text))

Max Activating Token Index: 412

Text #3

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1259. Min Act: -0.0001

Data Index: 593559 (C4 (Web Text))

Max Activating Token Index: 958

Text #4

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1351. Min Act: -0.0001

Data Index: 1191113 (C4 (Web Text))

Max Activating Token Index: 147

Text #5

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1288. Min Act: -0.0001

Data Index: 131711 (C4 (Web Text))

Max Activating Token Index: 215

Text #6

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1321. Min Act: -0.0001

Data Index: 1273331 (C4 (Web Text))

Max Activating Token Index: 631

Text #7

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1102. Min Act: -0.0001

Data Index: 1106070 (C4 (Web Text))

Max Activating Token Index: 625

Text #8

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1190. Min Act: -0.0001

Data Index: 237203 (C4 (Web Text))

Max Activating Token Index: 922

Text #9

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1049. Min Act: -0.0001

Data Index: 745934 (C4 (Web Text))

Max Activating Token Index: 909

Text #10

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1113. Min Act: -0.0001

Data Index: 809981 (C4 (Web Text))

Max Activating Token Index: 260

Text #11

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1056. Min Act: -0.0001

Data Index: 1229289 (C4 (Web Text))

Max Activating Token Index: 220

Text #12

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1119. Min Act: -0.0001

Data Index: 945154 (C4 (Web Text))

Max Activating Token Index: 957

Text #13

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.0989. Min Act: -0.0001

Data Index: 2889 (C4 (Web Text))

Max Activating Token Index: 143

Text #14

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.1068. Min Act: -0.0001

Data Index: 1198558 (C4 (Web Text))

Max Activating Token Index: 512

Text #15

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.0949. Min Act: -0.0001

Data Index: 177292 (C4 (Web Text))

Max Activating Token Index: 153

Text #16

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.0952. Min Act: -0.0001

Data Index: 1256132 (C4 (Web Text))

Max Activating Token Index: 93

Text #17

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.0984. Min Act: -0.0001

Data Index: 653616 (C4 (Web Text))

Max Activating Token Index: 309

Text #18

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.0993. Min Act: -0.0001

Data Index: 660287 (C4 (Web Text))

Max Activating Token Index: 218

Text #19

Max Range: 0.1536. Min Range: -0.1536

Max Act: 0.0989. Min Act: -0.0001

Data Index: 545645 (C4 (Web Text))

Max Activating Token Index: 442