Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 4967 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1469. Min Act: -0.0000

Data Index: 916779 (C4 (Web Text))

Max Activating Token Index: 980

Text #1

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1457. Min Act: -0.0000

Data Index: 1212860 (C4 (Web Text))

Max Activating Token Index: 215

Text #2

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1320. Min Act: 0.0000

Data Index: 103072 (C4 (Web Text))

Max Activating Token Index: 298

Text #3

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1320. Min Act: -0.0000

Data Index: 699258 (C4 (Web Text))

Max Activating Token Index: 711

Text #4

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1323. Min Act: -0.0000

Data Index: 802021 (C4 (Web Text))

Max Activating Token Index: 611

Text #5

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1263. Min Act: -0.0000

Data Index: 141256 (C4 (Web Text))

Max Activating Token Index: 935

Text #6

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1289. Min Act: -0.0000

Data Index: 1146983 (C4 (Web Text))

Max Activating Token Index: 943

Text #7

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1119. Min Act: -0.0000

Data Index: 1306874 (C4 (Web Text))

Max Activating Token Index: 893

Text #8

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1210. Min Act: -0.0000

Data Index: 158964 (C4 (Web Text))

Max Activating Token Index: 569

Text #9

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1139. Min Act: -0.0000

Data Index: 805080 (C4 (Web Text))

Max Activating Token Index: 942

Text #10

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1135. Min Act: -0.0000

Data Index: 1348424 (C4 (Web Text))

Max Activating Token Index: 900

Text #11

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1136. Min Act: -0.0000

Data Index: 369610 (C4 (Web Text))

Max Activating Token Index: 685

Text #12

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1107. Min Act: -0.0000

Data Index: 1119556 (C4 (Web Text))

Max Activating Token Index: 470

Text #13

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1070. Min Act: -0.0000

Data Index: 1186945 (C4 (Web Text))

Max Activating Token Index: 159

Text #14

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1013. Min Act: -0.0000

Data Index: 31503 (C4 (Web Text))

Max Activating Token Index: 238

Text #15

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1035. Min Act: -0.0000

Data Index: 248011 (C4 (Web Text))

Max Activating Token Index: 354

Text #16

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.1109. Min Act: -0.0000

Data Index: 263582 (C4 (Web Text))

Max Activating Token Index: 144

Text #17

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.0897. Min Act: -0.0000

Data Index: 269524 (C4 (Web Text))

Max Activating Token Index: 492

Text #18

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.0967. Min Act: -0.0000

Data Index: 611447 (C4 (Web Text))

Max Activating Token Index: 149

Text #19

Max Range: 0.1469. Min Range: -0.1469

Max Act: 0.0937. Min Act: -0.0000

Data Index: 1149969 (C4 (Web Text))

Max Activating Token Index: 548