Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 33 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.4264. Min Act: -0.0000

Data Index: 1523775 (Python Code)

Max Activating Token Index: 470

Text #1

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.3344. Min Act: -0.0000

Data Index: 518899 (C4 (Web Text))

Max Activating Token Index: 260

Text #2

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.3456. Min Act: -0.0000

Data Index: 652986 (C4 (Web Text))

Max Activating Token Index: 1023

Text #3

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.3161. Min Act: -0.0000

Data Index: 624012 (C4 (Web Text))

Max Activating Token Index: 1002

Text #4

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2572. Min Act: -0.0000

Data Index: 149965 (C4 (Web Text))

Max Activating Token Index: 954

Text #5

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2530. Min Act: -0.0000

Data Index: 1442863 (Python Code)

Max Activating Token Index: 841

Text #6

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2609. Min Act: -0.0000

Data Index: 1334554 (C4 (Web Text))

Max Activating Token Index: 323

Text #7

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2403. Min Act: -0.0000

Data Index: 633621 (C4 (Web Text))

Max Activating Token Index: 154

Text #8

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2269. Min Act: -0.0000

Data Index: 677629 (C4 (Web Text))

Max Activating Token Index: 321

Text #9

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2220. Min Act: -0.0000

Data Index: 786790 (C4 (Web Text))

Max Activating Token Index: 716

Text #10

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2342. Min Act: -0.0000

Data Index: 264923 (C4 (Web Text))

Max Activating Token Index: 93

Text #11

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2305. Min Act: -0.0000

Data Index: 89659 (C4 (Web Text))

Max Activating Token Index: 569

Text #12

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2370. Min Act: -0.0000

Data Index: 539381 (C4 (Web Text))

Max Activating Token Index: 162

Text #13

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2081. Min Act: -0.0000

Data Index: 581590 (C4 (Web Text))

Max Activating Token Index: 661

Text #14

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2092. Min Act: -0.0000

Data Index: 1346936 (C4 (Web Text))

Max Activating Token Index: 196

Text #15

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2186. Min Act: -0.0000

Data Index: 1025557 (C4 (Web Text))

Max Activating Token Index: 506

Text #16

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2107. Min Act: -0.0000

Data Index: 638283 (C4 (Web Text))

Max Activating Token Index: 471

Text #17

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.1808. Min Act: -0.0000

Data Index: 399707 (C4 (Web Text))

Max Activating Token Index: 630

Text #18

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2042. Min Act: -0.0000

Data Index: 201296 (C4 (Web Text))

Max Activating Token Index: 987

Text #19

Max Range: 0.4264. Min Range: -0.4264

Max Act: 0.2054. Min Act: -0.0000

Data Index: 1061600 (C4 (Web Text))

Max Activating Token Index: 157