Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2870 in Layer 4

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.2216. Min Act: -0.0001

Data Index: 926727 (C4 (Web Text))

Max Activating Token Index: 508

Text #1

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.2515. Min Act: -0.0001

Data Index: 1271574 (C4 (Web Text))

Max Activating Token Index: 573

Text #2

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.3246. Min Act: -0.0001

Data Index: 1234005 (C4 (Web Text))

Max Activating Token Index: 428

Text #3

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.1890. Min Act: -0.0001

Data Index: 132455 (C4 (Web Text))

Max Activating Token Index: 409

Text #4

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.3018. Min Act: -0.0001

Data Index: 641246 (C4 (Web Text))

Max Activating Token Index: 84

Text #5

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.2074. Min Act: -0.0001

Data Index: 357107 (C4 (Web Text))

Max Activating Token Index: 152

Text #6

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.1095. Min Act: -0.0001

Data Index: 368093 (C4 (Web Text))

Max Activating Token Index: 744

Text #7

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.0227. Min Act: -0.0001

Data Index: 667917 (C4 (Web Text))

Max Activating Token Index: 847

Text #8

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.1041. Min Act: -0.0001

Data Index: 1112947 (C4 (Web Text))

Max Activating Token Index: 242

Text #9

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.1984. Min Act: -0.0001

Data Index: 1212999 (C4 (Web Text))

Max Activating Token Index: 873

Text #10

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.1802. Min Act: -0.0001

Data Index: 191619 (C4 (Web Text))

Max Activating Token Index: 895

Text #11

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.2577. Min Act: -0.0001

Data Index: 877296 (C4 (Web Text))

Max Activating Token Index: 802

Text #12

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.0484. Min Act: -0.0001

Data Index: 514486 (C4 (Web Text))

Max Activating Token Index: 359

Text #13

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.0769. Min Act: -0.0001

Data Index: 774282 (C4 (Web Text))

Max Activating Token Index: 767

Text #14

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.1371. Min Act: -0.0001

Data Index: 614810 (C4 (Web Text))

Max Activating Token Index: 351

Text #15

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.1464. Min Act: -0.0001

Data Index: 324887 (C4 (Web Text))

Max Activating Token Index: 172

Text #16

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.0347. Min Act: -0.0001

Data Index: 909570 (C4 (Web Text))

Max Activating Token Index: 440

Text #17

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.0355. Min Act: -0.0001

Data Index: 1293624 (C4 (Web Text))

Max Activating Token Index: 161

Text #18

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.0078. Min Act: -0.0001

Data Index: 586577 (C4 (Web Text))

Max Activating Token Index: 905

Text #19

Max Range: 1.3246. Min Range: -1.3246

Max Act: 1.0309. Min Act: -0.0001

Data Index: 745529 (C4 (Web Text))

Max Activating Token Index: 162