Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3014 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.3956. Min Act: -0.0001

Data Index: 1308820 (C4 (Web Text))

Max Activating Token Index: 667

Text #1

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.1563. Min Act: -0.0001

Data Index: 217129 (C4 (Web Text))

Max Activating Token Index: 747

Text #2

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.2081. Min Act: -0.0001

Data Index: 1028518 (C4 (Web Text))

Max Activating Token Index: 128

Text #3

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.4104. Min Act: -0.0001

Data Index: 263355 (C4 (Web Text))

Max Activating Token Index: 434

Text #4

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.4174. Min Act: -0.0001

Data Index: 323983 (C4 (Web Text))

Max Activating Token Index: 591

Text #5

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.2476. Min Act: -0.0001

Data Index: 508801 (C4 (Web Text))

Max Activating Token Index: 253

Text #6

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.2118. Min Act: -0.0001

Data Index: 456252 (C4 (Web Text))

Max Activating Token Index: 917

Text #7

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.2834. Min Act: -0.0001

Data Index: 816272 (C4 (Web Text))

Max Activating Token Index: 520

Text #8

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.1770. Min Act: -0.0001

Data Index: 1201025 (C4 (Web Text))

Max Activating Token Index: 753

Text #9

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.2359. Min Act: -0.0001

Data Index: 507748 (C4 (Web Text))

Max Activating Token Index: 94

Text #10

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.1567. Min Act: -0.0001

Data Index: 1206851 (C4 (Web Text))

Max Activating Token Index: 439

Text #11

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.1467. Min Act: -0.0001

Data Index: 1350904 (C4 (Web Text))

Max Activating Token Index: 201

Text #12

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.1243. Min Act: -0.0001

Data Index: 721549 (C4 (Web Text))

Max Activating Token Index: 838

Text #13

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.0640. Min Act: -0.0001

Data Index: 89728 (C4 (Web Text))

Max Activating Token Index: 640

Text #14

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.0156. Min Act: -0.0001

Data Index: 887848 (C4 (Web Text))

Max Activating Token Index: 35

Text #15

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.2452. Min Act: -0.0001

Data Index: 598177 (C4 (Web Text))

Max Activating Token Index: 885

Text #16

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.1735. Min Act: -0.0001

Data Index: 133502 (C4 (Web Text))

Max Activating Token Index: 348

Text #17

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.0468. Min Act: -0.0001

Data Index: 331915 (C4 (Web Text))

Max Activating Token Index: 355

Text #18

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.0083. Min Act: -0.0001

Data Index: 1141261 (C4 (Web Text))

Max Activating Token Index: 601

Text #19

Max Range: 1.4174. Min Range: -1.4174

Max Act: 1.1698. Min Act: -0.0001

Data Index: 654531 (C4 (Web Text))

Max Activating Token Index: 885