Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1763 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.7694. Min Act: -0.0000

Data Index: 164805 (C4 (Web Text))

Max Activating Token Index: 482

Text #1

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.7764. Min Act: -0.0000

Data Index: 1353219 (C4 (Web Text))

Max Activating Token Index: 395

Text #2

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.7280. Min Act: -0.0000

Data Index: 772095 (C4 (Web Text))

Max Activating Token Index: 954

Text #3

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.6561. Min Act: -0.0000

Data Index: 783567 (C4 (Web Text))

Max Activating Token Index: 147

Text #4

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.6884. Min Act: -0.0000

Data Index: 253207 (C4 (Web Text))

Max Activating Token Index: 721

Text #5

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5330. Min Act: -0.0000

Data Index: 343541 (C4 (Web Text))

Max Activating Token Index: 235

Text #6

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.5442. Min Act: -0.0000

Data Index: 1235328 (C4 (Web Text))

Max Activating Token Index: 420

Text #7

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.4593. Min Act: -0.0000

Data Index: 419478 (C4 (Web Text))

Max Activating Token Index: 477

Text #8

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.4693. Min Act: -0.0000

Data Index: 105970 (C4 (Web Text))

Max Activating Token Index: 605

Text #9

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.4607. Min Act: -0.0000

Data Index: 1275171 (C4 (Web Text))

Max Activating Token Index: 201

Text #10

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.4445. Min Act: -0.0000

Data Index: 1260872 (C4 (Web Text))

Max Activating Token Index: 858

Text #11

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.4060. Min Act: -0.0000

Data Index: 46743 (C4 (Web Text))

Max Activating Token Index: 127

Text #12

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.4260. Min Act: -0.0000

Data Index: 1022365 (C4 (Web Text))

Max Activating Token Index: 1012

Text #13

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.3719. Min Act: -0.0000

Data Index: 1128388 (C4 (Web Text))

Max Activating Token Index: 102

Text #14

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.3939. Min Act: -0.0000

Data Index: 459104 (C4 (Web Text))

Max Activating Token Index: 292

Text #15

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.3726. Min Act: -0.0000

Data Index: 791224 (C4 (Web Text))

Max Activating Token Index: 493

Text #16

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.3904. Min Act: -0.0000

Data Index: 1242725 (C4 (Web Text))

Max Activating Token Index: 431

Text #17

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.3919. Min Act: -0.0000

Data Index: 1001040 (C4 (Web Text))

Max Activating Token Index: 973

Text #18

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.3800. Min Act: -0.0000

Data Index: 799149 (C4 (Web Text))

Max Activating Token Index: 692

Text #19

Max Range: 0.7764. Min Range: -0.7764

Max Act: 0.3418. Min Act: -0.0000

Data Index: 720593 (C4 (Web Text))

Max Activating Token Index: 1011