Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 577 in Layer 8

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.5587. Min Act: -0.0000

Data Index: 617008 (C4 (Web Text))

Max Activating Token Index: 461

Text #1

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.5512. Min Act: -0.0000

Data Index: 944672 (C4 (Web Text))

Max Activating Token Index: 526

Text #2

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.5402. Min Act: 0.0000

Data Index: 202223 (C4 (Web Text))

Max Activating Token Index: 656

Text #3

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.5310. Min Act: -0.0000

Data Index: 359103 (C4 (Web Text))

Max Activating Token Index: 104

Text #4

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.5107. Min Act: -0.0000

Data Index: 1292166 (C4 (Web Text))

Max Activating Token Index: 696

Text #5

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.5164. Min Act: -0.0000

Data Index: 1159585 (C4 (Web Text))

Max Activating Token Index: 206

Text #6

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.5263. Min Act: -0.0000

Data Index: 1091909 (C4 (Web Text))

Max Activating Token Index: 238

Text #7

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.5101. Min Act: -0.0000

Data Index: 1270754 (C4 (Web Text))

Max Activating Token Index: 60

Text #8

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.4874. Min Act: -0.0000

Data Index: 786048 (C4 (Web Text))

Max Activating Token Index: 203

Text #9

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.4876. Min Act: -0.0000

Data Index: 256870 (C4 (Web Text))

Max Activating Token Index: 156

Text #10

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.4691. Min Act: -0.0000

Data Index: 199003 (C4 (Web Text))

Max Activating Token Index: 104

Text #11

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.4834. Min Act: -0.0000

Data Index: 734802 (C4 (Web Text))

Max Activating Token Index: 870

Text #12

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.4802. Min Act: -0.0000

Data Index: 977467 (C4 (Web Text))

Max Activating Token Index: 151

Text #13

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.4730. Min Act: -0.0000

Data Index: 1259659 (C4 (Web Text))

Max Activating Token Index: 613

Text #14

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.4624. Min Act: -0.0000

Data Index: 144607 (C4 (Web Text))

Max Activating Token Index: 760

Text #15

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.4660. Min Act: -0.0000

Data Index: 1168632 (C4 (Web Text))

Max Activating Token Index: 298

Text #16

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.4620. Min Act: -0.0000

Data Index: 63473 (C4 (Web Text))

Max Activating Token Index: 150

Text #17

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.4603. Min Act: 0.0000

Data Index: 845284 (C4 (Web Text))

Max Activating Token Index: 999

Text #18

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.4557. Min Act: -0.0000

Data Index: 372231 (C4 (Web Text))

Max Activating Token Index: 947

Text #19

Max Range: 0.5587. Min Range: -0.5587

Max Act: 0.4691. Min Act: -0.0000

Data Index: 991458 (C4 (Web Text))

Max Activating Token Index: 882